Compare commits

..
Author SHA1 Message Date
jschoubben 1c20b9fb9b Stop committing mesh-sdk's Go dependencies: resolve them by go.sum (issue 1304)
mesh/merge-gate checking again: Issue 1304 step 2. Rejected in review round 1 only because main's go/merge-check.sh still required go/vendor. Step 1, mesh-…
mesh/repo-check checking again: Issue 1304 step 2. Rejected in review round 1 only because main's go/merge-check.sh still required go/vendor. Step 1, mesh-…
mesh/delivery stopped: the pull request closed unmerged
2026-10-19 21:57:52 +02:00
mesh-tools 91763dd70e Merge pull request 'Stop the stdio channel holding its lock across a pipe write (issue 1286)' (#41) from fix/1286-refused-subscribe-deadlock into main 2026-10-19 19:55:53 +00:00
jschoubben eaec209a37 Stop the stdio channel holding its lock across a pipe write, so a refused subscribe cannot deadlock it (issue 1286)
mesh/merge-gate head eaec209a371b merged into trunk 9ebfb87b5d0c — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head eaec209a371b merged into trunk 9ebfb87b5d0c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-19 21:51:13 +02:00
mesh-tools 9ebfb87b5d Merge pull request 'Say a build's directory and a pull request's module directories in the SDK (issue 1233)' (#39) from fix/1233-path-and-dirs into main 2026-10-19 13:17:09 +00:00
jschoubben 9ade199740 Give the build fixture the path the controller writes for build-agent (issue 1233)
mesh/merge-gate head 9ade199740e3 merged into trunk d79264611071 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 9ade199740e3 merged into trunk d79264611071 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-19 14:42:04 +02:00
jschoubben 0596f44414 Say a build's directory and a pull request's module directories in the SDK (issue 1233) 2026-10-19 14:40:19 +02:00
mesh-tools d792646110 Merge pull request 'Give the build record's layout to readers outside the controller (issue 1223)' (#38) from fix/1223-build-record-layout into main 2026-10-19 12:07:24 +00:00
jschoubben 732238cb6a Cite the controller's lines rightly, and say the newest key holds an id (issue 1223, review round 1)
mesh/merge-gate head 732238cb6a72 merged into trunk aa68b67389c9 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 732238cb6a72 merged into trunk aa68b67389c9 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-19 13:46:36 +02:00
jschoubben 845c8b0e56 Give the build record's layout to readers outside the controller (issue 1223) 2026-10-19 13:44:09 +02:00
mesh-tools aa68b67389 Merge pull request 'Delivery's release 3 keys and the source fingerprint, in one place the parts read (design 47 R3a: issues 1200-1203)' (#37) from feat/delivery-source-keys into main 2026-10-19 09:10:11 +00:00
jschoubben a39661aa36 Prove the source fingerprint against three builds the live builder recorded (issue 1203)
mesh/merge-gate head a39661aa36f3 merged into trunk 34392e74916c — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head a39661aa36f3 merged into trunk 34392e74916c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-19 11:03:46 +02:00
jschoubben 662fb0a57c Map every byte the router's id refuses in an ask id, so a group with = gives a usable ask (issue 1201) 2026-10-19 10:37:03 +02:00
jschoubben 0bc5659076 Make every ask id end in a digest of its key so two keys never share an id (issue 1201) 2026-10-19 10:34:59 +02:00
jschoubben c23833d3e5 Move the source fingerprint into one function the builder and the indexer both call (issue 1203) 2026-10-19 10:34:19 +02:00
jschoubben 0ae600dc97 Give a module one subject and one layout to say a condition to the operator (issue 1202) 2026-10-19 10:33:53 +02:00
jschoubben 15af148520 Give the gate's merges, asks and owed acts and the teller's answers one layout the parts read (issue 1201) 2026-10-19 10:33:17 +02:00
jschoubben 6d620f24b7 Give the source revision, candidates, modules and forge messages one layout the parts read (issue 1200) 2026-10-19 10:31:59 +02:00
mesh-tools 34392e7491 Merge pull request 'A module's process connects to the bus on an account of its own (ADR 0436 clause 4, issues 1171, 1187-1191)' (#36) from feat/mail-sender-sdk into main 2026-10-19 06:20:50 +00:00
jochen cfa20f05ad Reconnect without limit, as the tool runner does, so a process outlives a bus restart (issue 1192, review round 1 finding 5)
mesh/merge-gate head cfa20f05ad6d merged into trunk c8bd1370828a — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head cfa20f05ad6d merged into trunk c8bd1370828a — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-19 04:52:10 +02:00
jochen f1438dff6a Prove Conn offers no Serve by reflection, and build its AC3 test account through ReadAccount (issue 1171, review round 1 findings 3 and 4) 2026-10-19 04:51:14 +02:00
jochen 1ffbf9ea5b Make the Term test prove no redelivery instead of passing on a 30s AckWait it never waits out (issue 1190, review round 1 finding 2) 2026-10-19 04:50:07 +02:00
jochen e05e7130ae Catch a denied publish at once instead of waiting it out, so an ungranted bucket's refusal names it (issue 1188, review round 1 finding 1) 2026-10-19 04:48:58 +02:00
jochen 80d3975762 Let a process call a tool or a seat verb on its own account, with the tool runner's caller fallback (issue 1191) 2026-10-19 04:41:08 +02:00
jochen f73ec7457a Let a process consume its module's work queue, acknowledging, delaying or terminating each message (issue 1190) 2026-10-19 04:38:45 +02:00
jochen 94702bcd79 Let a process read the router's record of its module's own ask (issue 1189) 2026-10-19 04:35:33 +02:00
jochen de2d944c2a Let a process read and change its module's state by compare-and-set, and emit with a chosen message id (issue 1188) 2026-10-19 04:34:04 +02:00
jochen 0796341c8d Let an SDK Emit choose its event's message id, so a repeated publish is kept to one (issue 1187) 2026-10-19 04:26:54 +02:00
jochen 96155dea23 Give mesh-sdk a process's own pinned bus connection, reading its delivered account (issue 1171) 2026-10-19 04:26:01 +02:00
jochen 1361c1b456 Vendor the bus library and give integration tests a bus of their own (issue 1171) 2026-10-19 03:43:19 +02:00
mesh-tools c8bd137082 Merge pull request 'Give the SDK's Go check an integration level and vendored dependencies, before a test needs them (issue 1175)' (#35) from fix/1175-sdk-integration-level into main 2026-10-19 00:06:01 +00:00
jochen 9908faaa02 Give the SDK's Go check an integration level and vendored dependencies, before a test needs them (issue 1175)
mesh/merge-gate head 9908faaa0295 merged into trunk 238440d6f8c4 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 9908faaa0295 merged into trunk 238440d6f8c4 — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh, go/merge-check.sh): main's versio…
mesh/delivery delivered
2026-10-19 02:00:12 +02:00
mesh-tools 238440d6f8 Merge pull request 'Give the watchman's stuck keys a layout the controller reads (issue 1153)' (#34) from feat/delivery-stuck into main 2026-10-18 18:45:57 +00:00
jschoubben 953446cc8f Give the watchman's stuck keys a layout the controller reads (issue 1153)
mesh/merge-gate out of date: judged against trunk f692fe7da89d, the trunk is now 238440d6f8c4; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f692fe7da89d, the trunk is now 238440d6f8c4; judged again on a recheck or when it is merged
mesh/delivery delivered
ADR 0433 §2, §3: the watchman writes what stopped moving and the controller's
self-check raises it, so the key and value live in the SDK (ADR 0429 §2).
2026-10-18 20:11:04 +02:00
mesh-tools f692fe7da8 Merge pull request 'Give delivery's keys and values one home the node-engine and the parts read (issue 1131)' (#33) from feat/delivery-keys into main 2026-10-18 17:06:04 +00:00
jschoubben 2b420603fe Give delivery's keys and values one home the node-engine and the parts read (issue 1131)
mesh/merge-gate head 2b420603fe83 merged into trunk f70a99dbf485 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 2b420603fe83 merged into trunk f70a99dbf485 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-18 18:49:24 +02:00
mesh-tools f70a99dbf4 Merge pull request 'Feature providers-after-switch-on, step 1: the provider helper's fixes and the moved-provider harness (issues 945, 825, 849)' (#31) from feat/providers-after-switch-on into main 2026-10-16 19:13:11 +00:00
fix-sdk-pas a4e7f480b3 Apply the plan's credential form only to an offer that declares a credential (issue 825)
mesh/merge-gate head a4e7f480b3b0 merged into trunk 219bedbdd2b3 beside mesh-catalog#337@d2992f81 +1 more — pass, with a note: builds mesh-sdk (+41 depende…
mesh/repo-check head a4e7f480b3b0 merged into trunk 219bedbdd2b3 beside mesh-catalog#337@d2992f81 +1 more — pass: its merge-check.sh passed, each part in i…
mesh/delivery delivered
mesh/delivery-group group feat/providers-after-switch-on merged: every member is merged
2026-10-16 19:44:26 +02:00
fix-sdk-pas d7cac79585 Carry the grant-file and keycloak-shape moved-path tests into harness (issue 849) 2026-10-16 19:37:02 +02:00
fix-sdk-pas 377fd6d67d Move the moved path's shared driving into harness.Moved, so postgres and keycloak keep only what differs (issue 849) 2026-10-16 19:34:02 +02:00
fix-sdk-pas bb7c82cac6 Move the moved-provider harness and its retirement into go/provider/harness (issue 849) 2026-10-16 19:29:27 +02:00
fix-sdk-pas 0db956dd83 Skip a bad assignment, honour the plan's credential form, delete only what was read, redact every credential form (issue 825) 2026-10-16 19:28:25 +02:00
fix-sdk-pas 40f3658557 Name the vault's credential under vault-name so the state guard accepts the bound entry (issue 945) 2026-10-16 19:26:58 +02:00
mesh-tools 219bedbdd2 Merge pull request 'Feature 57 (config and provisioning): release mesh-sdk' (#30) from release/57-config-and-provisioning into main 2026-10-16 09:32:31 +00:00
mesh-tools 1bc5719b8b Feature 57 (config and provisioning): release mesh-sdk, the reviewed release branch on top of main
mesh/merge-gate head 1bc5719b8b85 merged into trunk a2d2af00dc5d beside mesh-host#116@bff1ac14 — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ac…
mesh/repo-check head 1bc5719b8b85 merged into trunk a2d2af00dc5d beside mesh-host#116@bff1ac14 — pass: its merge-check.sh passed, each part in its toolchai…
mesh/delivery delivered
mesh/delivery-group group release/57-config-and-provisioning merged: every member is merged
2026-10-16 08:24:11 +00:00
mesh-admin a2d2af00dc Merge pull request 'go/asks: a message carries its note apart, and an explanation is held to a glance (issues 383 and 639)' (#25) from fix/639-an-ask-reads-at-a-glance into main 2026-10-13 19:02:17 +00:00
jochen 5933eb757d go/asks: whole details never digest as another ask's Whole and Details (the review of #25)
mesh/merge-gate head 5933eb757d44 merged into trunk a0fc82fdcd7c — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 5933eb757d44 merged into trunk a0fc82fdcd7c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
WholeDetails was written after a tag, so an ask with only whole details "+ x: 1" encoded as one whose Whole
was "whole-details" and Details "+ x: 1". Whole and Details are now written whenever whole details are
given, then the whole details, untagged: the number of trailing values tells the three shapes apart, and an
ask without whole details digests as before. The note's comment says what ADR 0347 §5 holds of cuts.
2026-10-13 20:29:24 +02:00
jochen f4745447ba go/asks: an ask may carry its Details whole, for a channel that proves who answers (hq issue 383)
mesh/merge-gate head f4745447ba6d merged into trunk a0fc82fdcd7c — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head f4745447ba6d merged into trunk a0fc82fdcd7c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery superseded: a newer head of the same pull request
The review of mesh-controller #304: a settings proposal with more than three keys showed only the names
of the rest, and its Details could carry no value under the content rule, so Approve bound values the
operator never saw. WholeDetails carries every value whole for the note on a proving channel, held to
the secret rule alone, as Whole is; it is digested only when given.
2026-10-13 16:56:22 +02:00
jochen 9588871d66 go/asks: a message carries its note apart, and an explanation is held to a glance (hq issues 639 and 383)
mesh/merge-gate head 9588871d6629 merged into trunk a0fc82fdcd7c — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 9588871d6629 merged into trunk a0fc82fdcd7c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery superseded: a newer head of the same pull request
The operator will not read a Telegram ask of thirty plain lines. The secondary words (who asked, what each
answer does, the expiry, the fingerprint, how to read it whole) travel as Message.Note, so a channel can
show them small or collapsed; an ask's Details go there. Glance names the bound of an explanation: a
summary of at most 300 characters, then at most three points of at most 160.
2026-10-13 16:15:20 +02:00
mesh-admin a0fc82fdcd Merge pull request '0.1.14: the provider loop says a consumer it keeps failing on the bus, and its recovery (hq ADR 0224, issue 272)' (#24) from fix/272-the-provider-loop-says-its-standing into main 2026-10-13 06:37:30 +00:00
jochen d49d3a14d8 0.1.14: the provider loop says a consumer it keeps failing on the bus, and its recovery (hq ADR 0224, issue 272)
mesh/merge-gate head d49d3a14d82a merged into trunk c1ae30b0a2c8 — pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change…
mesh/repo-check head d49d3a14d82a merged into trunk c1ae30b0a2c8 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
No provider on runProvisioner passed announce, and the loop had no standing per
consumer, so the object store, the SQL and document stores, the mail server and
the others failed their consumers in their journals alone. The loop now keeps the
Go copy's standing (failing after five minutes, again every fifteen, recovered on
success, when unasked or retired, and on the first success since start) and says
it through the SDK's emit unless given another announce.
2026-10-13 08:26:33 +02:00
mesh-admin c1ae30b0a2 Merge pull request 'go/outward: the content rule in one place, for the operator channel and node-reads (hq ADR 0310 §4, issue 418)' (#23) from fix/418-node-reads into main 2026-10-12 13:20:22 +00:00
jochen 67bfa22132 go/outward: refuse a password in an address, a value given to an env-style secret name, and base64 wherever its slashes fall (review 53, issue 418)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
mesh/delivery-group group fix/418-node-reads merged: every member is merged
node-reads applies CheckSecret to env, unit and configuration files, where
user:pass@ in a URL, DB_PASS=, MESH_TOKEN= and name = value with spaces passed,
and a random base64 value passed when its slashes or dashes cut it into short
pieces. A run is now judged in pieces only where it is a path, base64 padding
marks a value, and letters alone are random when the case changes as a random
string's does, so a CamelCase name still passes.
2026-10-12 14:27:49 +02:00
jochen 4dc5f29d3e go/outward: the content rule in one place, so the operator channel and node-reads apply the same one (hq ADR 0310 §4, issue 418)
mesh/delivery-group group fix/418-node-reads rejected: a member's own check failed
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery superseded: a newer head of the same pull request
The operator channel's rule (messenger's content.go, with the secret class
split out as in issue 383's branch) moves here as package outward: Check for
what may leave on a channel, CheckSecret for a secret's shape alone, which is
what a node-reads answer is held to. The messenger switches to it once
mesh-catalog #178, which changes that file, has merged.
2026-10-12 13:52:57 +02:00
mesh-admin 9747f3e1b1 Merge pull request 'go: a module reads the values its bucket still keeps for a key (ADR 0307, issue 502)' (#22) from adr-0307-state-history into main 2026-10-12 08:21:13 +00:00
jochen d41d255f28 go: a module reads the values its bucket still keeps for a key, so the tracker can carry them into its log (hq ADR 0307)
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery-group group adr-0307-state-history merged: every member is merged
2026-10-12 10:17:56 +02:00
mesh-admin 5794ccd569 Merge pull request 'go: a module appends to and reads its log through the runtime (ADR 0297, issues 501, 502)' (#21) from adr-0297-log into main 2026-10-11 18:08:52 +00:00
jochen eaed934629 go: a module appends to and reads its log through the runtime, so its record outlives a bucket's history (hq ADR 0297)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
2026-10-11 19:16:09 +02:00
mesh-admin 4cf45b5ca0 Merge pull request 'go: a tool reads its caller from the runtime (hq issue 365)' (#20) from fix/365-a-tool-call-names-its-caller into main 2026-10-11 09:19:42 +00:00
jschoubben d9e24e95a2 go: a tool reads its caller from the runtime, so a record can say who acted (hq issue 365)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
mesh/delivery-group group fix/365-a-tool-call-names-its-caller merged: every member is merged
The runtime names the caller it took from the subject a call arrived on in the
call's _meta; stdio.CallerOf(args) hands it to the tool. Never read from the
arguments, and Unproven when the runtime named none, so a bundle on an older
runtime still says, explicitly, that its caller was not proven.
2026-10-11 05:11:08 +02:00
mesh-admin ee1e1a8303 Merge pull request 'cli: a client reads the answer its failed write left waiting, so a refusal is heard (issue 444)' (#19) from fix/444-a-client-reads-the-answer-its-write-could-not-finish into main 2026-10-11 00:56:01 +00:00
jschoubben 24f5ee81bd cli: after a failed write take only a refusal, since nothing can have run, and pin that the read keeps the ask's deadline (hq issue 444)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
2026-10-11 02:49:17 +02:00
jschoubben b4900f52b8 cli: read the engine's answer after a failed request write, so a refusal is not lost as a broken pipe (hq issue 444)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 02:43:54 +02:00
mesh-admin 19cfe0ef03 Merge pull request 'go/health: a health tool answers healthy, unhealthy or waits (issue 386, ADR 0283)' (#18) from feat/386-a-wait-for-the-operator into main 2026-10-10 19:13:46 +00:00
jochen 33917f91ac Refuse every form of path or address in a wait's words, a contradicting healthy answer, and a malformed wait at Waiting
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
mesh/delivery-group group feat/386-a-wait-for-the-operator merged: every member is merged
From the review of mesh-sdk#18: quoted and bracketed paths, home and relative
paths, IPv6, host and port, UNC and drive paths passed the check.
2026-10-10 21:10:01 +02:00
jochen 622b6b4f55 Add go/health: a health tool answers healthy, unhealthy or waits (hq ADR 0283, issue 386)
mesh/delivery-group group feat/386-a-wait-for-the-operator checking: 1 of 2 member(s) ready
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery superseded: a newer head of the same pull request
A module waiting for the operator's secret had only 'not healthy' to say,
so its first-node gate failed and the operator was told nothing to do.
The answer gains a state and named waits, and Read refuses any wait it
cannot check, so a malformed wait never excuses a fault.
2026-10-10 21:06:44 +02:00
95 changed files with 13880 additions and 77 deletions
+1
View File
@@ -1,2 +1,3 @@
node_modules/
dist/
vendor/
+21 -5
View File
@@ -20,7 +20,13 @@ Not the manifest schema (the control plane owns and parses that, in Go). The sha
contains — the contract both a provider and a consumer conform to), and the tool and event types.
These change rarely and deliberately; when one does, a rebuild of everything is *correct*.
### `provisioner` — the reconcile harness every provider shares
### `provider-loop` — the reconcile harness every provider on the grant files shares
Called `provisioner` until design 57 (novox/hq issue 678); that word is mesh-provisioner's now. The old path and
names (`/provisioner`, `runProvisioner`, `provisionerTools`) stay aliases until the providers importing them move to
the Go helper that answers consumers over the bus (`go/provider`, design 57 B2 and B3) and this loop retires with the
grant files. Its events and tools keep their names (`provisioner.failing`, `provisioner_retirement`, …): the
controller and the operator read them.
The loop is identical for postgres, redis, minio and umami: watch the grants directory, for each
requested grant call the provider's adapter, write the sealed credential, handle withdrawal, and
keep converging (`watch`). That loop lives here. A **module provides only the adapter** — "create
@@ -39,11 +45,21 @@ never called to retire**; it is the delete of an adapter without `delete`, reach
person's `provisioner_delete`. A stable set of more than 3, or more than half of those held where more
than one is held, is not retired without a person: the provider waits for
`provisioner_retire_approve` or `provisioner_retire_reject`, and says `provisioner.retirement`
(`waiting`) through the `announce` option. Asked for again, a retired consumer is created — re-enabled
(`waiting`). Asked for again, a retired consumer is created — re-enabled
— at once. Given `module`, the four tools (`provisioner_retirement`, `provisioner_retire_approve`,
`provisioner_retire_reject`, `provisioner_delete`) are registered for it; `provisionerTools(handle)`
builds them otherwise. Additions and changes are never delayed.
**A provider that keeps failing a consumer says so on the bus** (novox/hq ADR 0224). A consumer whose
create, check (`holds`) or secret has failed for five minutes without one success between
(`failingAfterMs`) is announced `provisioner.failing` — the consumer, its machine, the class of error
(`credentials-rejected`, `unreachable`, `secret-unreadable`, `refused`; an adapter's `classify` is
believed over the words), the error with the password removed, since when and how many attempts — and
again every fifteen minutes while it lasts. The first success after it is `provisioner.recovered`, as is
a consumer no longer asked for or retired, and the first success for each consumer since the process
started. Every event goes through the `announce` option, which defaults to the SDK's `emit`: a provider
says them without asking, and a failed publish is said in its journal, never failing the act.
### `tools` — the tool-serving harness
`registerModuleTools`, the `ToolDefinition` type, and the registry/worker that loads a module's
tools and exposes them through the mesh's command surface. *How* a tool is declared and served is
@@ -136,12 +152,12 @@ import { UmamiClient } from "../client.js"; // the client lives in the module, n
registerModuleTools("umami", (env) => getUmamiTools(new UmamiClient(env.UMAMI_URL)));
```
A provider's provisioner:
A provider's loop:
```ts
import { runProvisioner, Grant, Credential } from "@novox/mesh-sdk/provisioner";
import { runProviderLoop, Grant, Credential } from "@novox/mesh-sdk/provider-loop";
runProvisioner("analytics", {
runProviderLoop("analytics", {
async create(grant: Grant): Promise<Credential> { /* create a umami site, return the grant */ },
async remove(grant: Grant): Promise<void> { /* delete the site */ },
});
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+3 -3
View File
@@ -2,8 +2,8 @@
`stdio.Serve("", tools)` speaks MCP over stdio to the node's tool runtime, as the module the runtime
names (novox/hq ADR 0188, ADR 0193). A `Tool` is a name, a description, an argument schema and a
function; calls run concurrently. A tool named `<seat>.<verb>` is the seat's implementation.
`stdio.Emit(type, body)` publishes an event as the module, `stdio.Subscribe(pattern, handler)` receives the module's events (taken when the handler returns nil, offered again when it returns an error — the one rule is in the repository README, "Taking an event") and `stdio.Ask(key, body)` calls a tool, and `stdio.State(name)` gets, puts, deletes, lists and watches the module's declared state — all through the runtime (novox/hq ADR 0193, 0198, 0201). Standard library only.
function; calls run concurrently. `stdio.CallerOf(args)` inside a tool is who called it: the bus user the runtime took from the subject the call arrived on, named in the call's `_meta`, never read from the arguments — or `stdio.Unproven` when the runtime named none (novox/hq issue 365). A tool named `<seat>.<verb>` is the seat's implementation.
`stdio.Emit(type, body)` publishes an event as the module, `stdio.Subscribe(pattern, handler)` receives the module's events (taken when the handler returns nil, offered again when it returns an error — the one rule is in the repository README, "Taking an event") and `stdio.Ask(key, body)` calls a tool, and `stdio.State(name)` gets, puts, deletes, lists, watches and reads the kept history of the module's declared state, and `stdio.Log(name)` appends to, reads, and measures a log of operations the module declared — all through the runtime (novox/hq ADR 0193, 0198, 0201, 0297, 0307). `stdio.Config(key)` and `stdio.Bound(provision, key)` read, and `stdio.WatchConfig` and `stdio.WatchBound` watch, the module's own config values and the non-secret values of what it is bound to, for what its manifest declares: the runtime asks its machine's provisioner client as the module it launched, never a module the code names. `stdio.Secret(name)` is refused while the module's code runs in the tool runner; its secrets reach it through its files (novox/hq ADR 0349 §14, ADR 0370). `provider` is a providing module's half of a binding (novox/hq ADR 0349 §12, issue 678): it sees each consumer that binds the provider in the plans of the assignments mirror, writes one entry per consumer to `<module>_bound` (`<node>.<module>.<provision>[.<local>]`: served values composed with the consumer's identity, the login, and the name of the vault's credential, never a secret), hands the provider's adapter the vault's value (`provision/<provider>/<provision>` shared, `pair/<provider>/<provision>/<node>/<module>[/<local>]` per consumer), refuses a derived value to a consumer bound under a local name, naming both ends, and removes the entry of one that unbinds; the bus and the vault are interfaces the provider wires, the vault as `provider.ConsumersFile`, the file its machine's provisioner client fills from `${consumers:<provision>}` with each bound consumer's credential (novox/hq ADR 0373), since a provider in the tool runner never asks the vault itself. Standard library only.
## Installing
@@ -16,4 +16,4 @@ at the same version as the TypeScript package.
What it holds, under ADR 0039's test: the stdio loop and the tool type, and `cli`, the local protocol
`mesh-cli` speaks to the node-engine on its machine (novox/hq ADR 0272): the socket's path and one request and one
answer. Nothing else — no bus, no client, nothing of any module's. `go build ./example` and pipe a `tools/list` request into it.
answer; and `seal`, the mesh's sealed box (novox/hq ADR 0206: X25519, HKDF-SHA256, AES-256-GCM, `{v, eph, iv, tag, ct}`), byte for byte the box the licence manager made before it lived here and the TypeScript `@novox/mesh-sdk/seal` makes. Nothing else — no bus, no client, nothing of any module's. `go build ./example` and pipe a `tools/list` request into it.
+69 -5
View File
@@ -170,10 +170,18 @@ func (a Ask) Digest() string {
canonical(&b, v)
}
}
if a.Whole != "" || a.Details != "" {
// Whole and Details are written whenever either or WholeDetails is given, and WholeDetails after them whenever
// it is given (novox/hq ADR 0347): the fields after the options are then always one of three fixed shapes —
// none, the two, or the three — and the count of values the sequence holds tells them apart, so no two asks
// encode the same (the review of mesh-sdk #25). An ask with none of them digests as before they existed, and
// one without WholeDetails as before it existed.
if a.Whole != "" || a.Details != "" || a.WholeDetails != "" {
canonical(&b, a.Whole)
canonical(&b, a.Details)
}
if a.WholeDetails != "" {
canonical(&b, a.WholeDetails)
}
sum := sha256.Sum256([]byte(b.String()))
return "sha256:" + hex.EncodeToString(sum[:])
}
@@ -202,9 +210,59 @@ type Ask struct {
// The asker withholds a value shaped like a secret in it, and the router refuses the ask when one is left;
// Explanation stays under the whole content rule everywhere else. Empty, Explanation is shown everywhere.
Whole string `json:"whole,omitempty"`
// Details is what the Details answer on the ask's message shows, line by line under the content rule: a
// fingerprint, how to read the proposal whole at the terminal. Empty, an ask's own message offers no Details.
// Details are the ask's secondary words, line by line under the content rule: who proposed it, a
// fingerprint, how to read it whole. The router shows them in the note of the ask's own message
// (Message.Note), after what the ask's options do and when it expires — never in its body (novox/hq
// issue 639).
Details string `json:"details,omitempty"`
// WholeDetails are the Details with their exact values whole, shown in place of Details where Whole is shown
// in place of the explanation (a kind that proves who answers, given the ask's answers): every value an
// approval sets, whole, so the person reads exactly what they approve (novox/hq issue 383). Held, like
// Whole, to the secret rule alone; the router refuses the ask when a secret's shape is left in it. Empty,
// Details are shown everywhere.
WholeDetails string `json:"whole-details,omitempty"`
}
// What an ask's explanation is held to, so it reads at a glance on a phone (novox/hq issue 639): a summary
// of two or three short sentences on its first line — what changes, why, what it costs or risks — then at
// most three points, a line each, only where they add something. Whole is held to the same. Who asked, when,
// the fingerprint and how to read it whole are Details, never the explanation.
const (
SummaryLength = 300
MostPoints = 3
PointLength = 160
)
// Glance says why an explanation does not read at a glance, or nil: its first line, the summary, longer than
// SummaryLength; more than MostPoints lines after it; a point longer than PointLength. Blank lines are not
// counted. An asker holds its asks to it in its own tests; the router says, without refusing, an ask that
// does not hold to it.
func Glance(explanation string) error {
var lines []string
for _, l := range strings.Split(explanation, "\n") {
if strings.TrimSpace(l) != "" {
lines = append(lines, strings.TrimSpace(l))
}
}
var problems []string
switch {
case len(lines) == 0:
problems = append(problems, "it has no summary")
case len([]rune(lines[0])) > SummaryLength:
problems = append(problems, fmt.Sprintf("its summary is %d characters, more than %d", len([]rune(lines[0])), SummaryLength))
}
if len(lines) > 1+MostPoints {
problems = append(problems, fmt.Sprintf("it has %d points after its summary, more than %d", len(lines)-1, MostPoints))
}
for i, l := range lines {
if i > 0 && len([]rune(l)) > PointLength {
problems = append(problems, fmt.Sprintf("its point %d is %d characters, more than %d", i, len([]rune(l)), PointLength))
}
}
if len(problems) > 0 {
return errors.New("the explanation does not read at a glance: " + strings.Join(problems, "; "))
}
return nil
}
// The bounds of an ask (novox/hq ADR 0234 §8, ADR 0259 §4).
@@ -435,8 +493,14 @@ type Message struct {
Title string `json:"title"`
Body string `json:"body"`
Buttons []Button `json:"buttons,omitempty"`
Urgent bool `json:"urgent,omitempty"`
Silent bool `json:"silent,omitempty"`
// Note is the message's secondary words, after its body and its buttons: who asked and when, what each
// answer does, when it expires, and the ask's Details. A channel shows it smaller, collapsed or set apart
// where it can, and after the body where it cannot (novox/hq issue 639). A channel that bounds a message's
// length cuts the body first and the note only when the title and the note alone do not fit, saying each
// cut (novox/hq ADR 0347 §5).
Note string `json:"note,omitempty"`
Urgent bool `json:"urgent,omitempty"`
Silent bool `json:"silent,omitempty"`
// Reply is the Choice or LinkAsked this answers, by its ID: the channel shows it where that was made.
Reply string `json:"reply,omitempty"`
// To is the account linked as the operator on this kind, for a channel that verifies its sender: where
+88 -2
View File
@@ -253,8 +253,8 @@ func TestDigestsAreCanonicalOverNamedFields(t *testing.T) {
t.Error("changing Whole beside Details left the digest the same")
}
// A field added to Ask or Option without being added to Digest fails here.
if n := reflect.TypeOf(Ask{}).NumField(); n != 11 {
t.Errorf("Ask has %d fields, and Digest covers 11: add the new one to Digest and to this test", n)
if n := reflect.TypeOf(Ask{}).NumField(); n != 12 {
t.Errorf("Ask has %d fields, and Digest covers 12: add the new one to Digest and to this test", n)
}
if n := reflect.TypeOf(Option{}).NumField(); n != 5 {
t.Errorf("Option has %d fields, and Digest covers 5: add the new one to Digest and to this test", n)
@@ -279,3 +279,89 @@ func TestAWarrantForAnAskWithNoExpiryIsRefused(t *testing.T) {
t.Error("a warrant for an ask with no expiry was taken")
}
}
// An explanation that reads at a glance passes; a wall of text says each way it is one (novox/hq issue 639).
func TestAnExplanationReadsAtAGlance(t *testing.T) {
if err := Glance("Your agents' memory follows you to every machine. A file changed on two keeps both.\n\n• Only the memory folders are carried\n• A conflict reaches you as a warning"); err != nil {
t.Fatalf("a summary and two points: %v", err)
}
var wall strings.Builder
wall.WriteString("ADR 0343 was proposed by fix-367 and approved by the knowledge base's own review.")
for i := 1; i <= 13; i++ {
wall.WriteString("\n" + strings.Repeat("A clause lead. ", 3))
}
err := Glance(wall.String())
if err == nil || !strings.Contains(err.Error(), "13 points") {
t.Fatalf("thirteen clause lines pass as a glance: %v", err)
}
if err := Glance(strings.Repeat("word ", 70)); err == nil || !strings.Contains(err.Error(), "summary is 349") {
t.Fatalf("a long summary passes: %v", err)
}
if err := Glance("short.\n" + strings.Repeat("p", PointLength+1)); err == nil || !strings.Contains(err.Error(), "point 1") {
t.Fatalf("a long point passes: %v", err)
}
if err := Glance(" \n"); err == nil {
t.Fatal("an empty explanation passes")
}
}
// A message's note travels beside its body, and is left out of the encoding when empty, so a channel older
// than it reads every message it read before.
func TestAMessageCarriesItsNoteApart(t *testing.T) {
raw, _ := json.Marshal(Message{Handle: "h", Title: "t", Body: "b", Note: "Asked by x."})
if !strings.Contains(string(raw), `"note":"Asked by x."`) {
t.Fatalf("the note is not encoded: %s", raw)
}
raw, _ = json.Marshal(Message{Handle: "h", Title: "t"})
if strings.Contains(string(raw), "note") {
t.Fatalf("an empty note is encoded: %s", raw)
}
}
// WholeDetails is digested when given, and an ask without it digests as before it existed: a router and an asker
// of different builds still agree on every such ask.
func TestWholeDetailsAreDigestedOnlyWhenGiven(t *testing.T) {
a := Ask{ID: "a", Headline: "h", Explanation: "e", Details: "d"}
before := a.Digest()
a.WholeDetails = "+ shares: media=/storage/media"
if a.Digest() == before {
t.Fatal("the whole details are not in the digest")
}
b := a
b.WholeDetails = "+ shares: media=/storage/other"
if a.Digest() == b.Digest() {
t.Fatal("two whole details digest the same")
}
a.WholeDetails = ""
if a.Digest() != before {
t.Fatal("an ask without whole details digests otherwise than before")
}
}
// No two asks encode the same (the review of mesh-sdk #25): whole details given beside nothing else are not read as
// a Whole and Details that happen to spell the same values, nor as any other split of the three fields.
func TestWholeDetailsCollideWithNoOtherAsk(t *testing.T) {
base := Ask{ID: "a", Headline: "h", Explanation: "e"}
shapes := []Ask{}
for _, f := range []struct{ whole, details, wholeDetails string }{
{"", "", "+ x: 1"},
{"whole-details", "+ x: 1", ""},
{"", "+ x: 1", ""},
{"+ x: 1", "", ""},
{"", "", "whole-details"},
{"", "+ x: 1", "+ x: 1"},
{"+ x: 1", "+ x: 1", ""},
{"", "", ""},
} {
a := base
a.Whole, a.Details, a.WholeDetails = f.whole, f.details, f.wholeDetails
shapes = append(shapes, a)
}
seen := map[string]int{}
for i, a := range shapes {
if j, ok := seen[a.Digest()]; ok {
t.Errorf("asks %d and %d digest the same: %+v / %+v", j, i, shapes[j], a)
}
seen[a.Digest()] = i
}
}
+149
View File
@@ -0,0 +1,149 @@
package asks
import (
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
)
// ConditionSubject is where a module says a condition to the operator: a by-caller accept of the operator-channel
// seat, `mesh.seat.operator-channel.accept.condition.<publisher>` (novox/hq ADR 0437 §9). The router reads the
// verb at the fifth part and the caller at the sixth, so the publisher is one token, the module's own name; it is
// refused when empty or when it holds a dot, a wildcard or whitespace.
func ConditionSubject(publisher string) (string, error) {
if publisher == "" {
return "", errors.New("asks: the condition's publisher is empty")
}
if strings.ContainsAny(publisher, ".*> \t\r\n") || strings.ContainsFunc(publisher, func(r rune) bool { return r <= ' ' }) {
return "", fmt.Errorf("asks: the condition's publisher %q is not one token: it holds a dot, a wildcard or whitespace", publisher)
}
return "mesh.seat." + Seat + ".accept.condition." + publisher, nil
}
// ConditionVersion is the version of the condition value this package writes and the only one it reads.
const ConditionVersion = 1
// ErrUnknownConditionVersion is what DecodeCondition returns for a value whose version is missing or not
// ConditionVersion. Test it with errors.Is.
var ErrUnknownConditionVersion = errors.New("asks: unknown condition version")
// The words a condition's explanation opens with (novox/hq ADR 0253 §1).
const (
NothingToDo = "Nothing for you to do."
NeedsYou = "Needs you: "
)
// The severities and states of a condition.
const (
SeverityUrgent = "urgent"
SeverityWarning = "warning"
StateRaised = "raised"
StateCleared = "cleared"
)
// Scope is what a condition is about, under the controller's names.
type Scope struct {
Scope string `json:"scope"`
ID string `json:"id"`
Machine string `json:"machine,omitempty"`
}
// Action is one answer a notification offers, under the controller's names.
type Action struct {
Label string `json:"label"`
Verb string `json:"verb"`
Machine string `json:"machine,omitempty"`
Arguments map[string]string `json:"arguments,omitempty"`
Level Level `json:"level,omitempty"`
}
// Condition is a condition a module says to the operator, under the names of the controller's conditions: its key,
// severity, subject, state (raised or cleared), summary and the plain words of ADR 0253. A cleared condition needs
// only its key, subject and state. The router alone chooses the channels.
type Condition struct {
V int `json:"v"`
Key string `json:"key"`
Severity string `json:"severity,omitempty"`
Subject Scope `json:"subject"`
State string `json:"state"`
Summary string `json:"summary,omitempty"`
Headline string `json:"headline,omitempty"`
Needs string `json:"needs,omitempty"`
Explanation string `json:"explanation,omitempty"`
Resolved string `json:"resolved,omitempty"`
Actions []Action `json:"actions,omitempty"`
}
func (c Condition) check() error {
if c.Key == "" {
return errors.New("asks: a condition has no key")
}
if c.Subject.Scope == "" || c.Subject.ID == "" {
return errors.New("asks: a condition's subject needs its scope and id")
}
switch c.State {
case StateCleared:
return nil
case StateRaised:
default:
return fmt.Errorf("asks: the condition's state %q is not raised or cleared", c.State)
}
switch c.Severity {
case SeverityUrgent, SeverityWarning:
default:
return fmt.Errorf("asks: the condition's severity %q is not urgent or warning", c.Severity)
}
if c.Headline == "" {
return errors.New("asks: a raised condition has no headline")
}
if n := len([]rune(c.Headline)); n > HeadlineLength {
return fmt.Errorf("asks: the condition's headline is %d characters, more than %d", n, HeadlineLength)
}
verdict := NothingToDo
if c.Needs != "" {
verdict = NeedsYou + c.Needs
}
if !strings.HasPrefix(c.Explanation, verdict) {
return fmt.Errorf("asks: the condition's explanation must open with %q", verdict)
}
return nil
}
// EncodeCondition is c as JSON at ConditionVersion; it refuses what ADR 0253 and ADR 0437 §9 refuse.
func EncodeCondition(c Condition) ([]byte, error) {
if err := c.check(); err != nil {
return nil, err
}
c.V = ConditionVersion
return json.Marshal(c)
}
// DecodeCondition reads a condition; it refuses an unknown version (ErrUnknownConditionVersion) and what
// EncodeCondition refuses.
func DecodeCondition(data []byte) (Condition, error) {
var probe struct {
V *int `json:"v"`
}
if err := json.Unmarshal(data, &probe); err != nil {
return Condition{}, fmt.Errorf("asks: the condition is not JSON: %w", err)
}
if probe.V == nil {
return Condition{}, fmt.Errorf("%w: no version in the value, this reader reads %d", ErrUnknownConditionVersion, ConditionVersion)
}
if *probe.V != ConditionVersion {
return Condition{}, fmt.Errorf("%w: the value is version %d, this reader reads %d", ErrUnknownConditionVersion, *probe.V, ConditionVersion)
}
var c Condition
if err := json.Unmarshal(data, &c); err != nil {
return Condition{}, fmt.Errorf("asks: the condition does not fit its shape: %w", err)
}
if err := c.check(); err != nil {
return Condition{}, err
}
return c, nil
}
// SameCondition says whether two conditions are the same value.
func SameCondition(a, b Condition) bool { return reflect.DeepEqual(a, b) }
+143
View File
@@ -0,0 +1,143 @@
package asks
import (
"encoding/json"
"errors"
"strings"
"testing"
)
func TestConditionSubject(t *testing.T) { // 1202/AC1
s, err := ConditionSubject("mesh-delivery-teller")
if err != nil || s != "mesh.seat.operator-channel.accept.condition.mesh-delivery-teller" {
t.Fatalf("subject = %q, %v", s, err)
}
// The router splits an accept subject in six parts and reads the verb at 4 and the caller at 5.
parts := strings.Split(s, ".")
if len(parts) != 6 || parts[4] != "condition" || parts[5] != "mesh-delivery-teller" {
t.Errorf("subject %q does not keep the router's shape: %v", s, parts)
}
for _, bad := range []string{"", "a.b", "a*", "a>", "a b", "a\tb", "a\nb"} {
if _, err := ConditionSubject(bad); err == nil {
t.Errorf("publisher %q accepted", bad)
}
}
}
func raisedCondition() Condition {
return Condition{
Key: "delivery.stuck.mesh-mail", Severity: "warning",
Subject: Scope{Scope: "module", ID: "mesh-mail", Machine: "ace"}, State: "raised",
Summary: "plan 12 stopped", Headline: "Delivery of mesh-mail is stuck",
Needs: "approve the merge.", Explanation: "Needs you: approve the merge. The plan waits.",
Resolved: "The delivery of mesh-mail moved on.",
Actions: []Action{{Label: "Approve", Verb: "mesh-controller.assign", Machine: "ace",
Arguments: map[string]string{"why": ""}, Level: Approve}},
}
}
func clearedCondition() Condition {
return Condition{Key: "delivery.stuck.mesh-mail", Subject: Scope{Scope: "module", ID: "mesh-mail"}, State: "cleared"}
}
func TestConditionRoundTrip(t *testing.T) { // 1202/AC2
for _, c := range []Condition{raisedCondition(), clearedCondition()} {
data, err := EncodeCondition(c)
if err != nil {
t.Fatalf("%s: %v", c.State, err)
}
var probe map[string]any
if err := json.Unmarshal(data, &probe); err != nil || probe["v"] != float64(1) {
t.Fatalf("%s: %s does not carry v:1", c.State, data)
}
c.V = ConditionVersion
got, err := DecodeCondition(data)
if err != nil || !SameCondition(got, c) {
t.Errorf("%s round trip: %+v %v", c.State, got, err)
}
}
data, _ := EncodeCondition(raisedCondition())
var probe map[string]any
_ = json.Unmarshal(data, &probe)
for _, name := range []string{"key", "severity", "subject", "summary", "headline", "needs", "explanation", "resolved", "actions", "state"} {
if _, ok := probe[name]; !ok {
t.Errorf("the JSON has no %q: %s", name, data)
}
}
subject := probe["subject"].(map[string]any)
for _, name := range []string{"scope", "id", "machine"} {
if _, ok := subject[name]; !ok {
t.Errorf("the subject has no %q: %s", name, data)
}
}
action := probe["actions"].([]any)[0].(map[string]any)
for _, name := range []string{"label", "verb", "machine", "arguments", "level"} {
if _, ok := action[name]; !ok {
t.Errorf("the action has no %q: %s", name, data)
}
}
for _, in := range []string{`{"key":"k"}`, `{"v":2,"key":"k"}`} {
if _, err := DecodeCondition([]byte(in)); !errors.Is(err, ErrUnknownConditionVersion) {
t.Errorf("decode of %s = %v, want ErrUnknownConditionVersion", in, err)
}
}
}
func TestConditionVerdict(t *testing.T) { // 1202/AC3
c := raisedCondition()
c.Needs, c.Explanation = "", "The plan waits."
_, err := EncodeCondition(c)
if err == nil || !strings.Contains(err.Error(), "Nothing for you to do.") {
t.Errorf("empty needs without its verdict: %v", err)
}
c.Explanation = "Nothing for you to do. The plan waits."
if _, err := EncodeCondition(c); err != nil {
t.Errorf("empty needs with its verdict refused: %v", err)
}
c = raisedCondition()
c.Explanation = "Nothing for you to do. The plan waits."
_, err = EncodeCondition(c)
if err == nil || !strings.Contains(err.Error(), "Needs you: approve the merge.") {
t.Errorf("needs set with the wrong verdict: %v", err)
}
c.Explanation = "Needs you: something else. The plan waits."
if _, err = EncodeCondition(c); err == nil {
t.Error("needs set with other needs in the verdict accepted")
}
}
func TestConditionRefused(t *testing.T) { // 1202/AC4
mutate := func(f func(*Condition)) error {
c := raisedCondition()
f(&c)
_, err := EncodeCondition(c)
return err
}
for name, f := range map[string]func(*Condition){
"severity": func(c *Condition) { c.Severity = "critical" },
"state": func(c *Condition) { c.State = "open" },
"key": func(c *Condition) { c.Key = "" },
"headline": func(c *Condition) { c.Headline = "" },
"long headline": func(c *Condition) { c.Headline = strings.Repeat("h", HeadlineLength+1) },
} {
if err := mutate(f); err == nil {
t.Errorf("a condition with a bad %s accepted", name)
}
}
if err := mutate(func(c *Condition) { c.Headline = strings.Repeat("h", HeadlineLength) }); err != nil {
t.Errorf("a headline of the greatest length refused: %v", err)
}
if _, err := EncodeCondition(clearedCondition()); err != nil {
t.Errorf("a cleared condition with only key, subject and state refused: %v", err)
}
c := clearedCondition()
c.Key = ""
if _, err := EncodeCondition(c); err == nil {
t.Error("a cleared condition without its key accepted")
}
c = clearedCondition()
c.State = "gone"
if _, err := EncodeCondition(c); err == nil {
t.Error("a condition with an unknown state accepted")
}
}
+59
View File
@@ -0,0 +1,59 @@
package stdio
import (
"reflect"
"regexp"
"sync"
"unsafe"
)
// Unproven is the caller of a call the runtime named no caller for: one that arrived on the subject that names
// none, or from a runtime that names none (novox/hq issue 365). Said as a word, never left empty, so a record
// shows it was not proven.
const Unproven = "unproven"
// MetaCaller is where in a `tools/call`'s `_meta` the runtime names the caller: the bus user that published the
// call, which the runtime takes from the subject the call arrived on — that subject's last token, which the
// bus lets each user publish with its own name alone (novox/hq issue 365, by ADR 0259 §3's precedent). The
// runtime writes `_meta`; a caller writes only `arguments`, so no caller can name another.
const MetaCaller = "mesh/caller"
// busUser is the shape of a bus user's name: `<node>.<module>`, `person.<name>`, `controller` — names whose
// parts are safe in a subject. Anything else the runtime named is not a caller and reads as Unproven.
var busUser = regexp.MustCompile(`^[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*$`)
// calls holds the caller of every call while its tool runs, by the arguments map that call was handed.
var calls sync.Map // unsafe.Pointer → string
func argsKey(args map[string]any) unsafe.Pointer {
return reflect.ValueOf(args).UnsafePointer()
}
// CallerOf is who called the tool that was handed args, as the runtime proved it: the bus user that
// published the call, or Unproven. Read from the runtime, never from the arguments' contents — a `caller`
// argument names nobody. Valid while the tool runs; arguments no call was handed have no caller.
func CallerOf(args map[string]any) string {
if args == nil {
return Unproven
}
if c, ok := calls.Load(argsKey(args)); ok {
return c.(string)
}
return Unproven
}
// callerIn is the caller a `tools/call`'s `_meta` names, or Unproven.
func callerIn(meta map[string]any) string {
if s, ok := meta[MetaCaller].(string); ok && busUser.MatchString(s) {
return s
}
return Unproven
}
// runAs runs a tool with the caller held for its arguments for as long as it runs.
func runAs(t Tool, caller string, args map[string]any) (any, error) {
key := argsKey(args)
calls.Store(key, caller)
defer calls.Delete(key)
return t.Run(args)
}
+105
View File
@@ -0,0 +1,105 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"testing"
"time"
)
// callWith serves one tool that answers the caller CallerOf reads from its arguments, sends it one tools/call
// with the params given, and returns the caller it answered.
func callWith(t *testing.T, params map[string]any) string {
t.Helper()
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() {
_ = ServeOn("tracker", []Tool{{Name: "who", Run: func(args map[string]any) (any, error) {
return map[string]any{"caller": CallerOf(args)}, nil
}}}, inR, outW)
}()
defer inW.Close()
b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": params})
_, _ = inW.Write(append(b, '\n'))
lines := bufio.NewScanner(outR)
done := make(chan string, 1)
go func() {
if lines.Scan() {
var m struct {
Result struct {
Content []struct {
Text string `json:"text"`
} `json:"content"`
} `json:"result"`
}
_ = json.Unmarshal(lines.Bytes(), &m)
var answer struct {
Caller string `json:"caller"`
}
if len(m.Result.Content) > 0 {
_ = json.Unmarshal([]byte(m.Result.Content[0].Text), &answer)
}
done <- answer.Caller
}
}()
select {
case got := <-done:
return got
case <-time.After(3 * time.Second):
t.Fatal("the bundle did not answer the call")
return ""
}
}
// **A tool reads its caller from the runtime, never from the call's arguments** (novox/hq issue 365): the
// runtime names the caller it took from the subject the call arrived on in the call's `_meta`, and CallerOf
// hands it to the tool by the arguments the tool was given.
func TestAToolReadsTheCallerTheRuntimeNamed(t *testing.T) {
got := callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"title": "x"},
"_meta": map[string]any{MetaCaller: "shanks.node-tools"}})
if got != "shanks.node-tools" {
t.Fatalf("the tool read caller %q, want the one the runtime named", got)
}
}
// A caller named in the arguments is not the caller: an argument is the caller's to write.
func TestACallerInTheArgumentsIsNeverTheCaller(t *testing.T) {
got := callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"caller": "novox.node-tools",
"_meta": map[string]any{MetaCaller: "novox.node-tools"}, MetaCaller: "novox.node-tools"}})
if got != Unproven {
t.Fatalf("a caller the arguments claimed was read as %q, want %q", got, Unproven)
}
got = callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"caller": "novox.node-tools"},
"_meta": map[string]any{MetaCaller: "shanks.node-tools"}})
if got != "shanks.node-tools" {
t.Fatalf("with a caller in the arguments the tool read %q, want the runtime's", got)
}
}
// A runtime that names no caller — one from before issue 365, or a call on the subject that names none — is
// said as Unproven, explicitly; a caller that is not a bus user's name is Unproven too.
func TestACallTheRuntimeNamedNoCallerForIsUnproven(t *testing.T) {
for _, params := range []map[string]any{
{"name": "who", "arguments": map[string]any{}},
{"name": "who"},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{}},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: ""}},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: "a b"}},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: 7}},
} {
if got := callWith(t, params); got != Unproven {
t.Errorf("%v: the tool read caller %q, want %q", params, got, Unproven)
}
}
}
// Arguments the runtime never handed a tool have no caller.
func TestArgumentsOfNoCallHaveNoCaller(t *testing.T) {
if got := CallerOf(map[string]any{"x": 1}); got != Unproven {
t.Fatalf("CallerOf of a map no call was given answered %q", got)
}
if got := CallerOf(nil); got != Unproven {
t.Fatalf("CallerOf(nil) answered %q", got)
}
}
+15 -1
View File
@@ -140,7 +140,21 @@ func AskWith(ctx context.Context, socket string, req Request) (Answer, error) {
_ = conn.SetDeadline(deadline)
}
if err := WriteRequest(conn, req); err != nil {
return Answer{}, fmt.Errorf("asking the node-engine: %w", err)
// The engine may have answered before it read the request — a refusal (busy, an unknown account) written
// and the connection closed — and the write then fails on the close. That answer is already waiting, and
// it says why where the write error says only that the pipe broke (novox/hq issue 444, found under issue
// 434). So read it, within the deadline the connection already has, and only when none came say the write.
// Only a refusal is taken: the request was never wholly written, so nothing can have run, and any other
// answer here is not one a correct engine gives — passing it on as success would hide the failure.
a, rerr := ReadAnswer(conn)
if rerr != nil {
return Answer{}, fmt.Errorf("asking the node-engine: %w; and no answer came (%v)", err, rerr)
}
if a.Refused == "" {
return Answer{}, fmt.Errorf("asking the node-engine: %w; and the answer that came is not a refusal, "+
"though the request was never wholly written", err)
}
return a, nil
}
a, err := ReadAnswer(conn)
if err != nil {
+123
View File
@@ -0,0 +1,123 @@
package cli
import (
"context"
"errors"
"net"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// anEngineThatClosesFirst listens like the node-engine before mesh-host#80: it writes its answer, if any, and
// closes without reading the request (novox/hq issue 434). It answers one connection.
func anEngineThatClosesFirst(t *testing.T, answer *Answer) string {
t.Helper()
socket := filepath.Join(t.TempDir(), "cli.sock")
l, err := net.Listen("unix", socket)
if err != nil {
t.Skipf("no unix socket here: %v", err)
}
t.Cleanup(func() { l.Close() })
go func() {
conn, err := l.Accept()
if err != nil {
return
}
if answer != nil {
_ = WriteAnswer(conn, *answer)
}
conn.Close()
}()
return socket
}
// aLineTheSocketCannotHold is a request larger than a unix socket buffers, so its write cannot finish before
// the engine reads it — and an engine that closes first makes it fail every time, not only when it loses a race.
func aLineTheSocketCannotHold() []string { return []string{"status", strings.Repeat("x", 4<<20)} }
// An engine that refuses and closes before reading the request is heard: its refusal reaches the caller, not
// the broken pipe its close left the client's write (novox/hq issue 444).
func TestARefusalWrittenBeforeTheRequestIsReadReachesTheCaller(t *testing.T) {
busy := Refusal("the node-engine is busy: too many lines at once")
socket := anEngineThatClosesFirst(t, &busy)
a, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
if err != nil {
t.Fatalf("the engine said %q and the caller heard: %v", busy.Refused, err)
}
if a.Refused != busy.Refused || a.Exit != 1 {
t.Fatalf("the engine said %q and the caller heard %+v", busy.Refused, a)
}
}
// When the write fails and no answer is waiting, the error says both: what failed, and that no answer came.
func TestAFailedWriteWithNoAnswerSaysBoth(t *testing.T) {
socket := anEngineThatClosesFirst(t, nil)
start := time.Now()
_, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
if err == nil {
t.Fatal("a request the engine never read, and never answered, came back without an error")
}
if !strings.Contains(err.Error(), "asking the node-engine") || !strings.Contains(err.Error(), "no answer came") {
t.Fatalf("the error does not say what failed and that no answer came: %v", err)
}
if waited := time.Since(start); waited > 30*time.Second {
t.Fatalf("waited %s on a connection the engine had closed", waited)
}
}
// After a failed write only a refusal is taken: nothing can have run, so an answer saying it ran and exited 0
// is an error, never a success (novox/hq issue 444).
func TestAnAnswerOtherThanARefusalAfterAFailedWriteIsAnError(t *testing.T) {
ran := Answer{Exit: 0, Stdout: []byte("ok")}
socket := anEngineThatClosesFirst(t, &ran)
a, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
if err == nil {
t.Fatalf("a request never wholly written came back as success: %+v", a)
}
if !strings.Contains(err.Error(), "not a refusal") {
t.Fatalf("the error does not say the answer was not a refusal: %v", err)
}
}
// Reading after a failed write never waits longer than the ask already allowed: a peer that never reads and
// never answers is given up on at the context's deadline, not later (novox/hq issue 444).
func TestAFailedWriteWaitsNoLongerThanTheDeadline(t *testing.T) {
socket := filepath.Join(t.TempDir(), "cli.sock")
l, err := net.Listen("unix", socket)
if err != nil {
t.Skipf("no unix socket here: %v", err)
}
t.Cleanup(func() { l.Close() })
held := make(chan net.Conn, 1)
go func() {
conn, err := l.Accept()
if err != nil {
return
}
held <- conn // neither read nor answered, nor closed, until the test ends
}()
t.Cleanup(func() {
select {
case c := <-held:
c.Close()
default:
}
})
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
start := time.Now()
_, err = Ask(ctx, socket, aLineTheSocketCannotHold())
waited := time.Since(start)
if err == nil {
t.Fatal("a peer that never read nor answered gave no error")
}
if !errors.Is(err, os.ErrDeadlineExceeded) {
t.Fatalf("the error is not the deadline: %v", err)
}
if waited > 3*time.Second {
t.Fatalf("waited %s with a deadline of 1s", waited)
}
}
+41
View File
@@ -0,0 +1,41 @@
package delivery
import "time"
// BuildRecord is the layout mesh-controller internal/chain/records.go Build writes (line 91), the fields a reader outside
// the controller reads (novox/hq ADR 0437 §4 §5, issue 1223). The controller's record carries more; a decode ignores
// what is not here. Path is the module's directory in its repository, as the controller writes it (Build.Path, records.go:100,
// set from the build request trimmed of slashes at internal/inventory/chain_requests.go:19); empty for a module whose
// manifest is the repository's root. SourceInputs has no json keys of its own: it is written under its Go field names, as the controller
// writes it.
type BuildRecord struct {
ID string `json:"id"`
State string `json:"state,omitempty"`
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
Module string `json:"module,omitempty"`
Path string `json:"path,omitempty"`
Commit string `json:"commit,omitempty"`
At time.Time `json:"at,omitzero"`
SourceFingerprint string `json:"source-fingerprint,omitempty"`
SourceInputs *SourceInputs `json:"source-inputs,omitempty"`
}
// BuildsBucket is the bucket the builds are kept in: mesh-controller internal/broker/controller_buckets.go:101,
// BucketName(ControllerSeat, ChainBuilds), with ControllerSeat "mesh-controller" (internal/broker/streams.go:359) and
// ChainBuilds "builds" (controller_buckets.go:113).
const BuildsBucket = "mesh-controller_builds"
// BuildKey is a build's record under its module, a build that knew none under the request's key: copied from
// mesh-controller internal/chain/keys.go:35-40 (BuildKey, with RequestKey at line 42).
func BuildKey(module, id string) string {
if module == "" {
return "req." + id
}
return module + ".build." + id
}
// NewestKey is the key of a module's newest build, of any state: copied from mesh-controller internal/chain/keys.go:44.
// The key holds no record: its value is the newest build's id as raw bytes (internal/inventory/chain_builds.go:210 and
// :546), and the record lives at BuildKey(module, id). The pointer is a hint, checked against that record.
func NewestKey(module string) string { return module + ".newest" }
+123
View File
@@ -0,0 +1,123 @@
package delivery
import (
"encoding/json"
"os"
"testing"
"time"
)
// The fixture's path is what the controller writes for a build request: strings.Trim(path, "/") at mesh-controller
// internal/inventory/chain_requests.go:19, the module's directory in its repository (Build.Path, internal/chain/records.go:100).
// The fixture is a record the controller's own encoder wrote (json.Marshal of a chain.Build, mesh-controller
// internal/chain, a throwaway test run once and deleted) with build-agent's build-1792395447088936520 values, plus a
// manifest-digest and one made artifact, which a BuildRecord does not carry.
func loadBuildFixture(t *testing.T) []byte {
t.Helper()
raw, err := os.ReadFile("testdata/build-record-build-agent.json")
if err != nil {
t.Fatal(err)
}
return raw
}
func TestBuildRecordDecodes(t *testing.T) { // 1223/AC1
var r BuildRecord
if err := json.Unmarshal(loadBuildFixture(t), &r); err != nil {
t.Fatal(err)
}
for _, c := range []struct{ name, got, want string }{
{"id", r.ID, "build-1792395447088936520"},
{"module", r.Module, "build-agent"},
{"repository", r.Repository, "novox/mesh-controller"},
{"ref", r.Ref, "main"},
{"commit", r.Commit, "7a74cee83f88e6585a5511c3cc53bdf0f5303d20"},
{"state", r.State, "built"},
{"source-fingerprint", r.SourceFingerprint, "src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7"},
} {
if c.got != c.want {
t.Errorf("%s = %q, want %q", c.name, c.got, c.want)
}
}
if want := time.Date(2026, 10, 19, 9, 38, 0, 0, time.UTC); !r.At.Equal(want) {
t.Errorf("at = %v, want %v", r.At, want)
}
if r.SourceInputs == nil {
t.Fatal("source-inputs missing")
}
fp, none := SourceFingerprint(*r.SourceInputs)
if none != "" || fp != r.SourceFingerprint {
t.Errorf("SourceFingerprint of the decoded inputs = %q (none %q), the record says %q", fp, none, r.SourceFingerprint)
}
}
func TestBuildRecordPath(t *testing.T) { // 1233/AC1
var r BuildRecord
if err := json.Unmarshal(loadBuildFixture(t), &r); err != nil {
t.Fatal(err)
}
if r.Path != "modules/build-agent" {
t.Errorf("path = %q, want modules/build-agent", r.Path)
}
// mesh-controller's manifest is its repository's root: the controller writes no path (omitempty), so the path is "".
var m map[string]json.RawMessage
if err := json.Unmarshal(loadBuildFixture(t), &m); err != nil {
t.Fatal(err)
}
delete(m, "path")
raw, _ := json.Marshal(m)
var root BuildRecord
if err := json.Unmarshal(raw, &root); err != nil {
t.Fatal(err)
}
if root.Path != "" {
t.Errorf("path of a root module = %q, want empty", root.Path)
}
// Every field 1223/AC1 checks is still equal in both.
if root.ID != r.ID || root.Module != r.Module || root.Repository != r.Repository || root.Ref != r.Ref ||
root.Commit != r.Commit || root.State != r.State || root.SourceFingerprint != r.SourceFingerprint || !root.At.Equal(r.At) {
t.Errorf("fields differ: %+v vs %+v", root, r)
}
// And the path survives a round trip through the SDK's own type.
back, _ := json.Marshal(r)
var again BuildRecord
if err := json.Unmarshal(back, &again); err != nil || again.Path != r.Path {
t.Errorf("round trip path = %q (%v)", again.Path, err)
}
}
func TestBuildRecordWithoutInputs(t *testing.T) { // 1223/AC2
var m map[string]json.RawMessage
if err := json.Unmarshal(loadBuildFixture(t), &m); err != nil {
t.Fatal(err)
}
delete(m, "source-inputs")
raw, _ := json.Marshal(m)
var r BuildRecord
if err := json.Unmarshal(raw, &r); err != nil {
t.Fatal(err)
}
if r.SourceInputs != nil {
t.Errorf("source-inputs = %+v, want nil", r.SourceInputs)
}
if r.Module != "build-agent" {
t.Errorf("module = %q, want build-agent", r.Module)
}
}
func TestBuildKeys(t *testing.T) { // 1223/AC3
// Literal values of mesh-controller internal/broker/controller_buckets.go:101 (BucketName(ControllerSeat, ChainBuilds)
// is "mesh-controller_builds") and internal/chain/keys.go:35-44.
if BuildsBucket != "mesh-controller_builds" {
t.Errorf("BuildsBucket = %q", BuildsBucket)
}
if got := BuildKey("build-agent", "build-1"); got != "build-agent.build.build-1" {
t.Errorf("BuildKey = %q", got)
}
if got := BuildKey("", "build-1"); got != "req.build-1" {
t.Errorf("BuildKey without a module = %q", got)
}
if got := NewestKey("build-agent"); got != "build-agent.newest" {
t.Errorf("NewestKey = %q", got)
}
}
+390
View File
@@ -0,0 +1,390 @@
// Package delivery is the one home of the keys and values the parts of delivery read and write on the bus
// (novox/hq ADR 0429 §2: the key layouts another repository reads live in mesh-sdk; research 063 section 01). The
// node-engine writes the have keys, the watchman of the repository mesh-delivery writes presence and pulse, the
// view reads all of them, and the controller grants them; every one imports this package, and nothing else defines
// these names.
//
// **Buckets** are named `<owner module>_<name>` as the broker names a module's state (provider.BucketOf): the
// node-engine's `have`, and the watchman's `presence` and `pulse`.
//
// **Keys.** A have key is `<node>.<module>`, and a node's own facts are `<node>.node`, so a module may not be named
// `node`. A presence key is `<node>` for a node and `part.<part>.<instance>` for a part's instance, so a node may
// not be named `part`. A pulse key is `<bucket>.<key>`: the full name of the bucket watched, then that bucket's key,
// which may hold dots, so it splits at the first dot. A name is refused when it is empty or holds a dot, a wildcard,
// whitespace or a character a bus key refuses, and the error names it.
//
// **Values** are JSON of under about 500 bytes, each carrying its version in `v`. A reader refuses a value whose
// version is missing or one it does not know, with an error that is ErrUnknownVersion (research 063 section 05
// question 9 item 7): a writer writes a new version only once every reader's have shows a version that reads it. A
// writer writes only when the new value differs from the one before (the Same functions).
//
// **Source, candidates, modules and the forge** (issue 1200; ADR 0437 §5, release 3): the gate's bucket `source` with
// its one key `main` (every repository's merged head and the group that moved it), the indexer's `candidates.<group>`
// and `modules.<module>` (a module's source fingerprint, or none and why), and the forge bridge's messages named
// `forge.<owner>.<name>.<kind>` (source.go).
//
// **Merges, asks, owed and answers** (issue 1201; ADR 0437 §6, §8): the gate's `merges.<group>`, its asks
// `merge.<group>.<heads hash>` (the router's ask whole), its owed keys `<repository>.<number>.<head>.<act>` that the
// forge bridge performs, and the teller's answer under the ask's own key (gate.go).
//
// **The source fingerprint** (issue 1203; ADR 0437 §5): one function of named inputs that the controller's builder and
// the indexer both call, so they agree byte for byte (fingerprint.go).
//
// Standard library only, and this module's own asks package.
package delivery
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"time"
)
// The buckets of delivery's parts. Each is written by one owner: every node's node-engine writes the have keys of
// its own node only, and the watchman writes presence and pulse.
const (
HaveBucket = "node-engine_have"
PresenceBucket = WatchmanModule + "_presence"
PulseBucket = WatchmanModule + "_pulse"
)
// HaveHistory and PresenceHistory are the revisions the have and presence buckets keep per key (research 063
// section 01).
const (
HaveHistory = 20
PresenceHistory = 20
)
// PulseMoveBound is how long a move may take: a pulse key lives for this time minus the move's age, and the bucket
// expires it on its own (the server writes a delete marker), so a move that overruns it is seen to stall.
const PulseMoveBound = 120 * time.Second
// Version is the version of every value this package writes and the only one it reads.
const Version = 1
// ErrUnknownVersion is what a decode returns for a value whose version is missing or not Version. Test it with
// errors.Is; the message names the version seen and the one read.
var ErrUnknownVersion = errors.New("delivery: unknown value version")
// Phase is how far a module's change has come on a node.
type Phase string
// The phases of a module's change on a node.
const (
PhaseBackingUp Phase = "backing-up"
PhaseApplying Phase = "applying"
PhaseChecking Phase = "checking"
PhaseRunning Phase = "running"
PhaseFailed Phase = "failed"
)
// Health is what the module's own check says.
type Health string
// The healths a have carries.
const (
HealthOK Health = "ok"
HealthFailed Health = "failed"
HealthWaits Health = "waits"
HealthUnknown Health = "unknown"
)
var nameOK = regexp.MustCompile(`^[-/_=a-zA-Z0-9]+$`)
// checkName refuses a name that is empty or holds a dot, a wildcard, whitespace or a character a bus key refuses.
func checkName(kind, name string) error {
if name == "" {
return fmt.Errorf("delivery: the %s name is empty", kind)
}
if !nameOK.MatchString(name) {
return fmt.Errorf("delivery: the %s name %q holds a dot, a wildcard, whitespace or a character a key refuses", kind, name)
}
return nil
}
// HaveKey is the key, in HaveBucket, of a module's have on a node: `<node>.<module>`. A module named `node` is
// refused, since `<node>.node` is the node's own facts.
func HaveKey(node, module string) (string, error) {
if err := checkName("node", node); err != nil {
return "", err
}
if err := checkName("module", module); err != nil {
return "", err
}
if module == "node" {
return "", fmt.Errorf(`delivery: the module name "node" is refused: %s.node is the node's own key`, node)
}
return node + "." + module, nil
}
// HaveNodeKey is the key, in HaveBucket, of a node's own facts: `<node>.node`.
func HaveNodeKey(node string) (string, error) {
if err := checkName("node", node); err != nil {
return "", err
}
return node + ".node", nil
}
// ParseHaveKey is the inverse of HaveKey and HaveNodeKey: isNode is true for `<node>.node`, with module empty.
func ParseHaveKey(key string) (node, module string, isNode bool, err error) {
node, module, ok := strings.Cut(key, ".")
if !ok {
return "", "", false, fmt.Errorf("delivery: %q is not a have key, <node>.<module>", key)
}
if err := checkName("node", node); err != nil {
return "", "", false, fmt.Errorf("delivery: %q is not a have key: %w", key, err)
}
if err := checkName("module", module); err != nil {
return "", "", false, fmt.Errorf("delivery: %q is not a have key: %w", key, err)
}
if module == "node" {
return node, "", true, nil
}
return node, module, false, nil
}
// PresenceNodeKey is the key, in PresenceBucket, of a node: `<node>`. A node named `part` is refused.
func PresenceNodeKey(node string) (string, error) {
if err := checkName("node", node); err != nil {
return "", err
}
if node == "part" {
return "", errors.New(`delivery: the node name "part" is refused: part.<part>.<instance> is a part's key`)
}
return node, nil
}
// PresencePartKey is the key, in PresenceBucket, of a part's instance: `part.<part>.<instance>`.
func PresencePartKey(part, instance string) (string, error) {
if err := checkName("part", part); err != nil {
return "", err
}
if err := checkName("instance", instance); err != nil {
return "", err
}
return "part." + part + "." + instance, nil
}
// ParsePresenceKey is the inverse of PresenceNodeKey and PresencePartKey: for a node's key part and instance are
// empty, for a part's key node is.
func ParsePresenceKey(key string) (node, part, instance string, err error) {
if !strings.Contains(key, ".") {
node, err = PresenceNodeKey(key)
if err != nil {
return "", "", "", fmt.Errorf("delivery: %q is not a presence key: %w", key, err)
}
return node, "", "", nil
}
fields := strings.Split(key, ".")
if len(fields) != 3 || fields[0] != "part" {
return "", "", "", fmt.Errorf("delivery: %q is not a presence key, <node> or part.<part>.<instance>", key)
}
if _, err := PresencePartKey(fields[1], fields[2]); err != nil {
return "", "", "", fmt.Errorf("delivery: %q is not a presence key: %w", key, err)
}
return "", fields[1], fields[2], nil
}
// PulseKey is the key, in PulseBucket, of the move watched at key of bucket: `<bucket>.<key>`. The bucket is the
// full bucket name and holds no dot; the key may.
func PulseKey(bucket, key string) (string, error) {
if err := checkName("bucket", bucket); err != nil {
return "", err
}
if key == "" || strings.HasPrefix(key, ".") || strings.HasSuffix(key, ".") || strings.Contains(key, "..") {
return "", fmt.Errorf("delivery: the pulse key %q is empty or has an empty part", key)
}
for _, part := range strings.Split(key, ".") {
if err := checkName("key", part); err != nil {
return "", err
}
}
return bucket + "." + key, nil
}
// ParsePulseKey is the inverse of PulseKey, splitting at the first dot.
func ParsePulseKey(pulseKey string) (bucket, key string, err error) {
bucket, key, ok := strings.Cut(pulseKey, ".")
if !ok {
return "", "", fmt.Errorf("delivery: %q is not a pulse key, <bucket>.<key>", pulseKey)
}
if _, err := PulseKey(bucket, key); err != nil {
return "", "", fmt.Errorf("delivery: %q is not a pulse key: %w", pulseKey, err)
}
return bucket, key, nil
}
// PulseTTL is the time to live of a pulse key for a move of the given age: PulseMoveBound minus the age. At or past
// the bound the move is stalled, with no time left and nothing to write: the watchman raises it at once.
func PulseTTL(moveAge time.Duration) (ttl time.Duration, stalled bool) {
if moveAge >= PulseMoveBound {
return 0, true
}
return PulseMoveBound - moveAge, false
}
// Have is a module's state on a node, written by that node's node-engine under HaveKey. FP is the module's
// fingerprint once the indexer exists (design 47 release 3); until then Build is the build the node was sent, and
// the view compares by build.
type Have struct {
V int `json:"v"`
FP string `json:"fp,omitempty"`
Build string `json:"build,omitempty"`
Phase Phase `json:"phase"`
Health Health `json:"health"`
Detail string `json:"detail,omitempty"`
}
// HaveNode is a node's own facts, written under HaveNodeKey. Each field is the node-engine's own JSON, carried whole:
// this package names the facts and does not type them.
type HaveNode struct {
V int `json:"v"`
Windows json.RawMessage `json:"windows,omitempty"`
Busy json.RawMessage `json:"busy,omitempty"`
Strays json.RawMessage `json:"strays,omitempty"`
Filters json.RawMessage `json:"filters,omitempty"`
Profile json.RawMessage `json:"profile,omitempty"`
}
// Presence is whether a node or a part's instance is online; the entry's own time says since when. The watchman
// writes it.
type Presence struct {
V int `json:"v"`
Online bool `json:"online"`
}
// Pulse is the value of a pulse key: the watched key mirrored, its revision, and who owes the next move (a node or
// a part instance). The watchman writes it with the time to live PulseTTL gives.
type Pulse struct {
V int `json:"v"`
Bucket string `json:"bucket"`
Key string `json:"key"`
Revision uint64 `json:"revision"`
Owes string `json:"owes,omitempty"`
}
func checkVersion(data []byte) error {
var probe struct {
V *int `json:"v"`
}
if err := json.Unmarshal(data, &probe); err != nil {
return fmt.Errorf("delivery: value is not JSON: %w", err)
}
if probe.V == nil {
return fmt.Errorf("%w: no version in the value, this reader reads %d", ErrUnknownVersion, Version)
}
if *probe.V != Version {
return fmt.Errorf("%w: the value is version %d, this reader reads %d", ErrUnknownVersion, *probe.V, Version)
}
return nil
}
func decode(data []byte, into any) error {
if err := checkVersion(data); err != nil {
return err
}
if err := json.Unmarshal(data, into); err != nil {
return fmt.Errorf("delivery: value does not fit its shape: %w", err)
}
return nil
}
func (h Have) check() error {
switch h.Phase {
case PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed:
default:
return fmt.Errorf("delivery: phase %q is not backing-up, applying, checking, running or failed", h.Phase)
}
switch h.Health {
case HealthOK, HealthFailed, HealthWaits, HealthUnknown:
default:
return fmt.Errorf("delivery: health %q is not ok, failed, waits or unknown", h.Health)
}
return nil
}
// EncodeHave is h as JSON at Version; it refuses a phase or health outside the lists.
func EncodeHave(h Have) ([]byte, error) {
if err := h.check(); err != nil {
return nil, err
}
h.V = Version
return json.Marshal(h)
}
// DecodeHave reads a have; it refuses an unknown version (ErrUnknownVersion) and a phase or health outside the lists.
func DecodeHave(data []byte) (Have, error) {
var h Have
if err := decode(data, &h); err != nil {
return Have{}, err
}
if err := h.check(); err != nil {
return Have{}, err
}
return h, nil
}
// EncodeHaveNode is n as JSON at Version.
func EncodeHaveNode(n HaveNode) ([]byte, error) {
n.V = Version
return json.Marshal(n)
}
// DecodeHaveNode reads a node's facts; it refuses an unknown version (ErrUnknownVersion).
func DecodeHaveNode(data []byte) (HaveNode, error) {
var n HaveNode
err := decode(data, &n)
return n, err
}
// EncodePresence is p as JSON at Version.
func EncodePresence(p Presence) ([]byte, error) {
p.V = Version
return json.Marshal(p)
}
// DecodePresence reads a presence; it refuses an unknown version (ErrUnknownVersion).
func DecodePresence(data []byte) (Presence, error) {
var p Presence
err := decode(data, &p)
return p, err
}
// EncodePulse is p as JSON at Version.
func EncodePulse(p Pulse) ([]byte, error) {
p.V = Version
return json.Marshal(p)
}
// DecodePulse reads a pulse; it refuses an unknown version (ErrUnknownVersion).
func DecodePulse(data []byte) (Pulse, error) {
var p Pulse
err := decode(data, &p)
return p, err
}
// SameHave says whether two haves are the same value, so a writer writes only when they differ.
func SameHave(a, b Have) bool { return a == b }
// SameHaveNode says whether two node facts are the same value, comparing each carried JSON without its spacing.
func SameHaveNode(a, b HaveNode) bool {
return a.V == b.V && sameJSON(a.Windows, b.Windows) && sameJSON(a.Busy, b.Busy) &&
sameJSON(a.Strays, b.Strays) && sameJSON(a.Filters, b.Filters) && sameJSON(a.Profile, b.Profile)
}
// SamePresence says whether two presences are the same value.
func SamePresence(a, b Presence) bool { return a == b }
// SamePulse says whether two pulses are the same value.
func SamePulse(a, b Pulse) bool { return a == b }
func sameJSON(a, b json.RawMessage) bool {
var ca, cb bytes.Buffer
if json.Compact(&ca, a) != nil || json.Compact(&cb, b) != nil {
return bytes.Equal(a, b)
}
return bytes.Equal(ca.Bytes(), cb.Bytes())
}
+294
View File
@@ -0,0 +1,294 @@
package delivery
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
)
// 1131/AC1: the have key forms and parses back; bad names are refused, naming them.
func TestHaveKey(t *testing.T) {
if HaveBucket != "node-engine_have" {
t.Fatalf("HaveBucket = %q", HaveBucket)
}
key, err := HaveKey("ace", "plex")
if err != nil || key != "ace.plex" {
t.Fatalf("HaveKey = %q, %v", key, err)
}
node, module, isNode, err := ParseHaveKey(key)
if err != nil || node != "ace" || module != "plex" || isNode {
t.Fatalf("ParseHaveKey = %q %q %v %v", node, module, isNode, err)
}
if got, err := HaveNodeKey("ace"); err != nil || got != "ace.node" {
t.Fatalf("HaveNodeKey = %q, %v", got, err)
}
if _, err := HaveNodeKey("a b"); err == nil {
t.Error("HaveNodeKey with a space accepted")
}
node, module, isNode, err = ParseHaveKey("ace.node")
if err != nil || node != "ace" || module != "" || !isNode {
t.Fatalf("ParseHaveKey node = %q %q %v %v", node, module, isNode, err)
}
for _, c := range []struct{ node, module, named string }{
{"ace", "node", "node"},
{"ace", "", "module"},
{"", "plex", "node"},
{"a.b", "plex", "a.b"},
{"ace", "p*", "p*"},
{"ace", "p>", "p>"},
{"ace", "p x", "p x"},
{"ace", "p\tx", "module"},
} {
_, err := HaveKey(c.node, c.module)
if err == nil || !strings.Contains(err.Error(), c.named) {
t.Errorf("HaveKey(%q, %q) = %v, want an error naming %q", c.node, c.module, err, c.named)
}
}
for _, bad := range []string{"", "ace", "ace.", ".plex", "ace.plex.x", "ace.p x"} {
if _, _, _, err := ParseHaveKey(bad); err == nil {
t.Errorf("ParseHaveKey(%q) accepted", bad)
}
}
}
// 1131/AC2: the presence keys form and parse back; a node named part is refused.
func TestPresenceKey(t *testing.T) {
if PresenceBucket != "mesh-delivery-watchman_presence" {
t.Fatalf("PresenceBucket = %q", PresenceBucket)
}
key, err := PresenceNodeKey("ace")
if err != nil || key != "ace" {
t.Fatalf("PresenceNodeKey = %q, %v", key, err)
}
node, part, instance, err := ParsePresenceKey(key)
if err != nil || node != "ace" || part != "" || instance != "" {
t.Fatalf("ParsePresenceKey node = %q %q %q %v", node, part, instance, err)
}
key, err = PresencePartKey("view", "novox")
if err != nil || key != "part.view.novox" {
t.Fatalf("PresencePartKey = %q, %v", key, err)
}
node, part, instance, err = ParsePresenceKey(key)
if err != nil || node != "" || part != "view" || instance != "novox" {
t.Fatalf("ParsePresenceKey part = %q %q %q %v", node, part, instance, err)
}
if _, err := PresenceNodeKey("part"); err == nil || !strings.Contains(err.Error(), "part") {
t.Errorf("PresenceNodeKey(part) = %v", err)
}
if _, err := PresenceNodeKey("a.b"); err == nil {
t.Error("PresenceNodeKey(a.b) accepted")
}
if _, err := PresencePartKey("", "x"); err == nil {
t.Error("PresencePartKey with empty part accepted")
}
if _, err := PresencePartKey("view", "x y"); err == nil {
t.Error("PresencePartKey with a space accepted")
}
for _, bad := range []string{"", "part", "part.view", "part.view.a.b", "a.b", "part..x"} {
if _, _, _, err := ParsePresenceKey(bad); err == nil {
t.Errorf("ParsePresenceKey(%q) accepted", bad)
}
}
}
// 1131/AC3: the pulse key splits at the first dot, the key holding dots.
func TestPulseKey(t *testing.T) {
if PulseBucket != "mesh-delivery-watchman_pulse" {
t.Fatalf("PulseBucket = %q", PulseBucket)
}
key, err := PulseKey("node-engine_have", "ace.plex")
if err != nil || key != "node-engine_have.ace.plex" {
t.Fatalf("PulseKey = %q, %v", key, err)
}
bucket, rest, err := ParsePulseKey(key)
if err != nil || bucket != "node-engine_have" || rest != "ace.plex" {
t.Fatalf("ParsePulseKey = %q %q %v", bucket, rest, err)
}
for _, c := range [][2]string{{"", "k"}, {"a.b", "k"}, {"b", ""}, {"b", "k x"}, {"b", ".k"}, {"b", "k."}, {"b", "k*"}} {
if _, err := PulseKey(c[0], c[1]); err == nil {
t.Errorf("PulseKey(%q, %q) accepted", c[0], c[1])
}
}
for _, bad := range []string{"", "nodot", ".k", "b."} {
if _, _, err := ParsePulseKey(bad); err == nil {
t.Errorf("ParsePulseKey(%q) accepted", bad)
}
}
}
// 1131/AC4: each value encodes, decodes equal, and carries "v":1.
func TestRoundTrip(t *testing.T) {
hasV1 := func(t *testing.T, b []byte) {
t.Helper()
var m map[string]any
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
if m["v"] != float64(1) {
t.Fatalf("v = %v in %s", m["v"], b)
}
if len(b) > 500 {
t.Fatalf("value is %d bytes", len(b))
}
}
t.Run("have", func(t *testing.T) {
in := Have{Build: "b1", FP: "f1", Phase: PhaseRunning, Health: HealthOK, Detail: "up"}
b, err := EncodeHave(in)
if err != nil {
t.Fatal(err)
}
hasV1(t, b)
out, err := DecodeHave(b)
if err != nil {
t.Fatal(err)
}
in.V = Version
if out != in {
t.Fatalf("got %+v want %+v", out, in)
}
})
t.Run("have node", func(t *testing.T) {
in := HaveNode{Windows: json.RawMessage(`{"a":1}`), Busy: json.RawMessage(`true`)}
b, err := EncodeHaveNode(in)
if err != nil {
t.Fatal(err)
}
hasV1(t, b)
out, err := DecodeHaveNode(b)
if err != nil {
t.Fatal(err)
}
if !SameHaveNode(out, HaveNode{V: Version, Windows: in.Windows, Busy: in.Busy}) {
t.Fatalf("got %+v", out)
}
})
t.Run("presence", func(t *testing.T) {
b, err := EncodePresence(Presence{Online: true})
if err != nil {
t.Fatal(err)
}
hasV1(t, b)
out, err := DecodePresence(b)
if err != nil || out != (Presence{V: Version, Online: true}) {
t.Fatalf("got %+v %v", out, err)
}
})
t.Run("pulse", func(t *testing.T) {
in := Pulse{Bucket: "node-engine_have", Key: "ace.plex", Revision: 7, Owes: "ace"}
b, err := EncodePulse(in)
if err != nil {
t.Fatal(err)
}
hasV1(t, b)
out, err := DecodePulse(b)
in.V = Version
if err != nil || out != in {
t.Fatalf("got %+v %v", out, err)
}
})
}
// 1131/AC5: a missing or unknown version is refused as ErrUnknownVersion, naming both versions.
func TestUnknownVersion(t *testing.T) {
decoders := map[string]func([]byte) error{
"have": func(b []byte) error { _, err := DecodeHave(b); return err },
"have node": func(b []byte) error { _, err := DecodeHaveNode(b); return err },
"presence": func(b []byte) error { _, err := DecodePresence(b); return err },
"pulse": func(b []byte) error { _, err := DecodePulse(b); return err },
}
for name, dec := range decoders {
t.Run(name, func(t *testing.T) {
err := dec([]byte(`{"v":2,"phase":"running","health":"ok"}`))
if !errors.Is(err, ErrUnknownVersion) || !strings.Contains(err.Error(), "2") || !strings.Contains(err.Error(), "1") {
t.Errorf("v 2: %v", err)
}
err = dec([]byte(`{"phase":"running","health":"ok"}`))
if !errors.Is(err, ErrUnknownVersion) {
t.Errorf("missing v: %v", err)
}
if err := dec([]byte(`not json`)); err == nil || errors.Is(err, ErrUnknownVersion) {
t.Errorf("garbage: %v", err)
}
})
}
}
// 1131/AC6: a phase or health outside the lists is refused on encode and decode.
func TestPhaseHealthRefused(t *testing.T) {
for _, p := range []Phase{PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed} {
for _, h := range []Health{HealthOK, HealthFailed, HealthWaits, HealthUnknown} {
if _, err := EncodeHave(Have{Phase: p, Health: h}); err != nil {
t.Errorf("%s/%s refused: %v", p, h, err)
}
}
}
if _, err := EncodeHave(Have{Phase: "sleeping", Health: HealthOK}); err == nil || !strings.Contains(err.Error(), "sleeping") {
t.Errorf("bad phase encoded: %v", err)
}
if _, err := EncodeHave(Have{Phase: PhaseRunning, Health: "fine"}); err == nil || !strings.Contains(err.Error(), "fine") {
t.Errorf("bad health encoded: %v", err)
}
if _, err := EncodeHave(Have{Phase: PhaseRunning}); err == nil {
t.Error("empty health encoded")
}
if _, err := DecodeHave([]byte(`{"v":1,"phase":"sleeping","health":"ok"}`)); err == nil {
t.Error("bad phase decoded")
}
if _, err := DecodeHave([]byte(`{"v":1,"phase":"running","health":"fine"}`)); err == nil {
t.Error("bad health decoded")
}
}
// 1131/AC7: the pulse's time to live is 120 seconds minus the move's age; at the bound the move is stalled.
func TestPulseTTL(t *testing.T) {
for _, c := range []struct {
age time.Duration
ttl time.Duration
stalled bool
}{
{0, 120 * time.Second, false},
{30 * time.Second, 90 * time.Second, false},
{119 * time.Second, time.Second, false},
{120 * time.Second, 0, true},
{200 * time.Second, 0, true},
} {
ttl, stalled := PulseTTL(c.age)
if ttl != c.ttl || stalled != c.stalled {
t.Errorf("PulseTTL(%v) = %v, %v; want %v, %v", c.age, ttl, stalled, c.ttl, c.stalled)
}
}
if PulseMoveBound != 120*time.Second || HaveHistory != 20 || PresenceHistory != 20 {
t.Error("constants")
}
}
// 1131/AC8: equal values say same, different ones do not.
func TestSame(t *testing.T) {
a := Have{V: 1, Build: "b", Phase: PhaseRunning, Health: HealthOK}
b := a
if !SameHave(a, b) {
t.Error("equal haves differ")
}
b.Detail = "x"
if SameHave(a, b) {
t.Error("different haves same")
}
n := HaveNode{V: 1, Busy: json.RawMessage(`true`)}
if !SameHaveNode(n, HaveNode{V: 1, Busy: json.RawMessage(`true`)}) || SameHaveNode(n, HaveNode{V: 1}) {
t.Error("have node")
}
if !SamePresence(Presence{V: 1, Online: true}, Presence{V: 1, Online: true}) || SamePresence(Presence{V: 1}, Presence{V: 1, Online: true}) {
t.Error("presence")
}
p := Pulse{V: 1, Bucket: "b", Key: "k", Revision: 1, Owes: "ace"}
q := p
if !SamePulse(p, q) {
t.Error("equal pulses differ")
}
q.Revision = 2
if SamePulse(p, q) {
t.Error("different pulses same")
}
}
+63
View File
@@ -0,0 +1,63 @@
package delivery
import (
"crypto/sha256"
"encoding/hex"
"sort"
"strings"
)
// SourceInputs are what a module's source fingerprint is of (novox/hq ADR 0437 §5): the module, the git tree of
// its directory, its bases (references, of which only the digest counts), the root tree of each repository its
// artifacts read (by artifact), each artifact's toolchain line, and the reason the source does not pin the build
// when it does not (a build that resolves packages when built). A context is the value the builder records for it:
// a repository's root tree, or for an image that compiles a Go program the `narrow:` hash of the build source it was
// handed (mesh-controller internal/builder/builder.go:1146-1156).
type SourceInputs struct {
Module string
Tree string
Bases []string
Contexts map[string]string
Toolchains map[string]string
Unpinned string
}
// SourceFingerprint is the module's source fingerprint, `src1:` and the hex sha256 of its inputs' lines, or none
// and the reason when the source does not pin the build: Unpinned when set, "no tree" when Tree is empty. It is the
// one function the controller's builder and the indexer both call, so they agree byte for byte. The lines are
// "module <m>" and "tree <t>", then "base <digest>" (once each), "context <artifact> <tree>" and "toolchain
// <artifact> <line>" sorted, all joined by a newline.
func SourceFingerprint(in SourceInputs) (fp, none string) {
if in.Unpinned != "" {
return "", in.Unpinned
}
if in.Tree == "" {
return "", "no tree"
}
lines := []string{"module " + in.Module, "tree " + in.Tree}
bases := map[string]bool{}
for _, b := range in.Bases {
bases[ReferenceDigest(b)] = true
}
for b := range bases {
lines = append(lines, "base "+b)
}
for a, t := range in.Contexts {
lines = append(lines, "context "+a+" "+t)
}
for a, t := range in.Toolchains {
lines = append(lines, "toolchain "+a+" "+t)
}
sort.Strings(lines[2:])
sum := sha256.Sum256([]byte(strings.Join(lines, "\n")))
return SourcePrefix + hex.EncodeToString(sum[:]), ""
}
// ReferenceDigest is a reference's digest, `sha256:...`, so the registry address a base was copied into does not
// enter the fingerprint; the reference itself when it carries none.
func ReferenceDigest(reference string) string {
if _, digest, pinned := strings.Cut(reference, "@"); pinned && digest != "" {
return digest
}
return reference
}
+178
View File
@@ -0,0 +1,178 @@
package delivery
import (
"strings"
"testing"
)
func inputs() SourceInputs {
return SourceInputs{
Module: "mesh-mail", Tree: "t-module",
Bases: []string{"registry.example/base@sha256:aaa", "registry.example/other@sha256:bbb"},
Contexts: map[string]string{"mail": "t-root", "tool": "t-root2"},
Toolchains: map[string]string{"mail": "go sha256:ccc 0123456789abcdef", "tool": "go sha256:ccc fedcba9876543210"},
}
}
func mustFP(t *testing.T, in SourceInputs) string {
t.Helper()
fp, none := SourceFingerprint(in)
if fp == "" || none != "" {
t.Fatalf("fingerprint = %q, none = %q", fp, none)
}
return fp
}
func TestFingerprintForm(t *testing.T) {
fp := mustFP(t, inputs())
hex := strings.TrimPrefix(fp, SourcePrefix)
if SourcePrefix != "src1:" || hex == fp || len(hex) != 64 || strings.Trim(hex, "0123456789abcdef") != "" {
t.Errorf("fingerprint %q is not src1: and 64 hex digits", fp)
}
}
func TestFingerprintOrderAndDuplicates(t *testing.T) { // 1203/AC2
want := mustFP(t, inputs())
in := inputs()
in.Bases = []string{in.Bases[1], in.Bases[0]}
if got := mustFP(t, in); got != want {
t.Errorf("the order of bases changes it: %s", got)
}
in = inputs()
in.Bases = append(in.Bases, in.Bases[0])
if got := mustFP(t, in); got != want {
t.Errorf("a base given twice changes it: %s", got)
}
in = inputs()
in.Bases = append(in.Bases, "other.registry.example/base@sha256:aaa")
if got := mustFP(t, in); got != want {
t.Errorf("two references to one digest change it: %s", got)
}
// Maps have no order, but build them in many insertion orders anyway.
for i := 0; i < 20; i++ {
in = inputs()
in.Contexts = map[string]string{}
in.Toolchains = map[string]string{}
for _, k := range []string{"tool", "mail"} {
in.Contexts[k] = inputs().Contexts[k]
in.Toolchains[k] = inputs().Toolchains[k]
}
if got := mustFP(t, in); got != want {
t.Fatalf("map order changes it: %s", got)
}
}
}
func TestFingerprintChanges(t *testing.T) { // 1203/AC3
want := mustFP(t, inputs())
for name, f := range map[string]func(*SourceInputs){
"module": func(in *SourceInputs) { in.Module = "other" },
"tree": func(in *SourceInputs) { in.Tree = "t-other" },
"base": func(in *SourceInputs) { in.Bases[0] = "registry.example/base@sha256:zzz" },
"a base added": func(in *SourceInputs) { in.Bases = append(in.Bases, "x@sha256:new") },
"context": func(in *SourceInputs) { in.Contexts["mail"] = "t-changed" },
"context artifact": func(in *SourceInputs) { in.Contexts["extra"] = "t-root" },
"toolchain": func(in *SourceInputs) { in.Toolchains["tool"] = "go sha256:ccc changed" },
} {
in := inputs()
f(&in)
if got := mustFP(t, in); got == want {
t.Errorf("a changed %s keeps the fingerprint", name)
}
}
}
func TestFingerprintNone(t *testing.T) { // 1203/AC4
in := inputs()
in.Unpinned = "the build resolves packages when built"
fp, none := SourceFingerprint(in)
if fp != "" || none != in.Unpinned {
t.Errorf("unpinned: fp = %q, none = %q", fp, none)
}
in = inputs()
in.Tree = ""
fp, none = SourceFingerprint(in)
if fp != "" || none != "no tree" {
t.Errorf("no tree: fp = %q, none = %q", fp, none)
}
in = inputs()
in.Tree, in.Unpinned = "", "the build resolves packages when built"
if _, none = SourceFingerprint(in); none != in.Unpinned {
t.Errorf("the reason for unpinned stands over no tree: %q", none)
}
}
func TestReferenceDigest(t *testing.T) {
for in, want := range map[string]string{
"registry.example/base@sha256:aaa": "sha256:aaa",
"registry.example/base": "registry.example/base",
"registry.example/base@": "registry.example/base@",
"": "",
} {
if got := ReferenceDigest(in); got != want {
t.Errorf("ReferenceDigest(%q) = %q, want %q", in, got, want)
}
}
}
// 1203/AC1: three builds the live builder recorded a source fingerprint for. Each value is captured, never written
// by hand: the recorded fingerprint was read with mesh-controller.builds (after issues 1204 and 1208 were
// delivered); each tree is `git rev-parse <commit>:<dir>` in the repository's clone; the bases are from the
// build's `[bases]` log lines; the toolchain lines are mesh-controller's toolchainOf over its toolchains values.
func TestFingerprintLiveBuilds(t *testing.T) {
const ts = "typescript sha256:4b81682df5665c1f461d3f720a3a0fd7593f4aa71e4fc272dbfdc4bbd235b489 20125bd590041690"
for _, c := range []struct {
name string
in SourceInputs
recorded string
}{
{
// mailu, build-1792391134620131370: mesh-catalog at b4527d9091e972f28cb365623d9280045e7741a3, dir modules/mailu.
name: "mailu",
in: SourceInputs{
Module: "mailu",
Tree: "cfc6fa8ff5fc1e220132e51ade653b9d3f387b3e",
Bases: []string{"sha256:25f3cfeaceca14921366af4d1240b56457ef46273bdb508c7b0e8f469f6fd228"},
Toolchains: map[string]string{"code": ts},
},
recorded: "src1:539f2e64000d436c572ef53b40e524012455945f8c5a1c6bb173ee0707e88ef4",
},
{
// radarr, build-1792391147149753355: mesh-media-catalog at 197abb061d9b77d67c7f11e9e567528ea7a85436,
// dir modules/radarr; no bases.
name: "radarr",
in: SourceInputs{
Module: "radarr",
Tree: "42b9c918b5af75cba2bbc75845012e7f64e16df9",
Toolchains: map[string]string{
"code": ts,
"tools-go": "go sha256:2054591ca034b68323d87ec887ba34f56819cb3364034360c0777da4b314dd54 0995d85edc9a6a23",
},
},
recorded: "src1:b3c150d39c395b99dd68974a48490e9a42467aa5d486d0c604d6ef1e8e9c910a",
},
{
// build-agent, build-1792395447088936520: mesh-controller at 7a74cee83f88e6585a5511c3cc53bdf0f5303d20,
// dir modules/build-agent; the context is narrowTree's hash of the 185 paths GoBuildSource gave for
// cmd/mesh-builder at 7a74cee8.
name: "build-agent",
in: SourceInputs{
Module: "build-agent",
Tree: "11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca",
Bases: []string{
"sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc",
"sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c",
},
Contexts: map[string]string{"server": "narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"},
},
recorded: "src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7",
},
} {
t.Run(c.name, func(t *testing.T) {
fp, none := SourceFingerprint(c.in)
if none != "" || fp != c.recorded {
t.Errorf("fingerprint = %q (none %q), the live builder recorded %q", fp, none, c.recorded)
}
})
}
}
+404
View File
@@ -0,0 +1,404 @@
package delivery
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"reflect"
"regexp"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/asks"
)
// The gate's buckets and the teller's (ADR 0437 §6, §8). The gate writes merges (each pull request's expected
// head, its state and each merge commit), its asks (`merge.<group>.<heads hash>`, the router's ask whole) and owed
// (one key per act it owes the forge, which the forge bridge performs once per revision). The teller writes its
// answer under the ask's own key in its bucket answers.
const (
MergesBucket = GateModule + "_merges"
AsksBucket = GateModule + "_asks"
OwedBucket = GateModule + "_owed"
AnswersBucket = TellerModule + "_answers"
MergesHistory = 10
AsksHistory = 5
OwedHistory = 5
AnswersHistory = 5
)
// AskKindMerge is the kind of the gate's ask for a group's merge, the first part of its key.
const AskKindMerge = "merge"
// The acts the gate owes the forge, the last part of an owed key.
const (
OwedStatus = "status"
OwedView = "view"
OwedNote = "note"
OwedComment = "comment"
)
// The states of a pull request in a merge.
type MergeState string
// The states a merge's pull request is in.
const (
MergeMerging MergeState = "merging"
MergeMerged MergeState = "merged"
MergeRefused MergeState = "refused"
)
// The states a status the gate owes the forge may carry.
var statusStates = []string{"pending", "success", "failure", "error", "warning"}
var hashOK = regexp.MustCompile(`^[0-9a-f]{12}$`)
// MergesKey is the key, in MergesBucket, of a group: `<group>`.
func MergesKey(group string) (string, error) {
if err := checkName("group", group); err != nil {
return "", err
}
return group, nil
}
// MergeAskKey is the key, in AsksBucket, of the ask for merging a group at these heads: `merge.<group>.<heads
// hash>`. The teller's answer has the same key in AnswersBucket (AnswerKey).
func MergeAskKey(group string, heads []Head) (string, error) {
if err := checkName("group", group); err != nil {
return "", err
}
hash, err := HeadsHash(heads)
if err != nil {
return "", err
}
return AskKindMerge + "." + group + "." + hash, nil
}
// ParseAskKey is the inverse of MergeAskKey: the kind, the group and the heads hash.
func ParseAskKey(key string) (kind, group, hash string, err error) {
parts := strings.Split(key, ".")
if len(parts) != 3 || parts[0] != AskKindMerge {
return "", "", "", fmt.Errorf("delivery: %q is not an ask key, merge.<group>.<heads hash>", key)
}
if err := checkName("group", parts[1]); err != nil {
return "", "", "", fmt.Errorf("delivery: %q is not an ask key: %w", key, err)
}
if !hashOK.MatchString(parts[2]) {
return "", "", "", fmt.Errorf("delivery: %q is not an ask key: %q is not 12 hex digits", key, parts[2])
}
return parts[0], parts[1], parts[2], nil
}
// AnswerKey is the key, in AnswersBucket, of the teller's answer to the ask at askKey: the same string.
func AnswerKey(askKey string) string { return askKey }
// AskID is the id the teller gives the router for the ask at askKey. The router's ids refuse dots (asks.UsableID),
// so in the readable part a dot becomes `-` and any other byte outside letters, digits, `_` and `-` becomes `_`; that part is cut to fit, and every id ends in `-` and
// the first 12 hex digits of the sha256 of the whole key, so two keys never give one id. It cannot be read back:
// the teller keeps the key beside it.
func AskID(askKey string) string {
const most = 64 // the router's bound on an id
const tail = 13 // "-" and 12 hex digits
readable := []byte(askKey)
for i, c := range readable {
switch {
case c >= 'A' && c <= 'Z', c >= 'a' && c <= 'z', c >= '0' && c <= '9':
case c == '.':
readable[i] = '-'
default: // a slash, an equals sign or any other byte the router's id refuses
readable[i] = '_'
}
}
if len(readable) > 0 && (readable[0] == '-' || readable[0] == '_') {
readable[0] = 'k' // an id starts with a letter or digit
}
if len(readable) > most-tail {
readable = readable[:most-tail]
}
sum := sha256.Sum256([]byte(askKey))
return string(readable) + "-" + hex.EncodeToString(sum[:])[:12]
}
func checkOwedAct(act string) (kind, context string, err error) {
kind, context, hasContext := strings.Cut(act, "=")
switch kind {
case OwedStatus:
if !hasContext {
return "", "", fmt.Errorf("delivery: the owed act %q is a status without its context, status=<context>", act)
}
if err := checkName("status context", context); err != nil {
return "", "", err
}
return kind, context, nil
case OwedView, OwedNote, OwedComment:
if hasContext {
return "", "", fmt.Errorf("delivery: the owed act %q takes no context", act)
}
return kind, "", nil
}
return "", "", fmt.Errorf("delivery: the owed act %q is not status=<context>, view, note or comment", act)
}
// ParseOwedAct splits an owed act into its kind and, for a status, its context.
func ParseOwedAct(act string) (kind, context string, err error) { return checkOwedAct(act) }
// OwedKey is the key, in OwedBucket, of an act the gate owes the forge on a pull request at a head:
// `<repository>.<number>.<head>.<act>`, the act being `status=<context>` (mesh/merge-gate), `view`, `note` or
// `comment`.
func OwedKey(repository string, number int, head, act string) (string, error) {
if err := checkRepository(repository); err != nil {
return "", err
}
if number <= 0 {
return "", fmt.Errorf("delivery: an owed key needs the pull request number, not %d", number)
}
if err := checkName("head", head); err != nil {
return "", err
}
if _, _, err := checkOwedAct(act); err != nil {
return "", err
}
return repository + "." + strconv.Itoa(number) + "." + head + "." + act, nil
}
// ParseOwedKey is the inverse of OwedKey; act is whole, as `status=<context>` for a status.
func ParseOwedKey(key string) (repository string, number int, head, act string, err error) {
parts := strings.Split(key, ".")
if len(parts) != 4 {
return "", 0, "", "", fmt.Errorf("delivery: %q is not an owed key, <repository>.<number>.<head>.<act>", key)
}
number, convErr := strconv.Atoi(parts[1])
if convErr != nil {
return "", 0, "", "", fmt.Errorf("delivery: %q is not an owed key: %q is not a pull request number", key, parts[1])
}
if _, err := OwedKey(parts[0], number, parts[2], parts[3]); err != nil {
return "", 0, "", "", fmt.Errorf("delivery: %q is not an owed key: %w", key, err)
}
return parts[0], number, parts[2], parts[3], nil
}
// Merge is the value of a merges key: the ask approved and, for each pull request, the head the approval expects
// and how far its merge has come.
type Merge struct {
V int `json:"v"`
Group string `json:"group"`
Ask string `json:"ask"`
Pulls []MergePull `json:"pulls"`
}
// MergePull is one pull request of a merge: its expected head, its state, the merge commit once merged, the
// forge's words when refused.
type MergePull struct {
Head
State MergeState `json:"state"`
Commit string `json:"commit,omitempty"`
Words string `json:"words,omitempty"`
}
func (m Merge) check() error {
if err := checkName("group", m.Group); err != nil {
return err
}
if m.Ask == "" {
return fmt.Errorf("delivery: a merge names no ask")
}
if len(m.Pulls) == 0 {
return fmt.Errorf("delivery: a merge names no pull request")
}
for _, p := range m.Pulls {
who := fmt.Sprintf("%s#%d", p.Repository, p.Number)
switch p.State {
case MergeMerging:
case MergeMerged:
if p.Commit == "" {
return fmt.Errorf("delivery: the merged pull request %s has no merge commit", who)
}
case MergeRefused:
if p.Words == "" {
return fmt.Errorf("delivery: the refused pull request %s has no words of the forge", who)
}
default:
return fmt.Errorf("delivery: the state %q of %s is not merging, merged or refused", p.State, who)
}
}
return nil
}
// Ask is the value of an asks key: the router's ask whole as the gate composes it, so the teller publishes it
// unchanged, and the group and heads it was composed for.
type Ask struct {
V int `json:"v"`
Group string `json:"group"`
Heads []Head `json:"heads"`
Ask asks.Ask `json:"ask"`
}
func (a Ask) check() error {
key, err := MergeAskKey(a.Group, a.Heads)
if err != nil {
return fmt.Errorf("delivery: an ask needs its group and at least one pull request's head: %w", err)
}
if want := AskID(key); a.Ask.ID != want {
return fmt.Errorf("delivery: the ask's id %q is not %q, the id its group and heads give", a.Ask.ID, want)
}
if len(a.Ask.Options) == 0 {
return fmt.Errorf("delivery: the ask has no option")
}
return nil
}
// Answer is the teller's answer to an ask: the option chosen, the warrant's id and its words.
type Answer struct {
V int `json:"v"`
Option string `json:"option"`
Warrant string `json:"warrant"`
Words string `json:"words"`
}
func (a Answer) check() error {
for _, f := range []struct{ name, value string }{{"option", a.Option}, {"warrant", a.Warrant}, {"words", a.Words}} {
if f.value == "" {
return fmt.Errorf("delivery: an answer has no %s", f.name)
}
}
return nil
}
// Owed is the value of an owed key: what the forge bridge performs. A status needs Context and State, a view and a
// comment need Text, a note needs Text and Ref (the notes ref, mesh-plan).
type Owed struct {
V int `json:"v"`
Act string `json:"act"`
Context string `json:"context,omitempty"`
State string `json:"state,omitempty"`
Text string `json:"text,omitempty"`
Ref string `json:"ref,omitempty"`
}
func (o Owed) check() error {
switch o.Act {
case OwedStatus:
if o.Context == "" {
return fmt.Errorf("delivery: an owed status has no context")
}
for _, s := range statusStates {
if o.State == s {
return nil
}
}
return fmt.Errorf("delivery: the owed status state %q is not %s", o.State, strings.Join(statusStates, ", "))
case OwedView, OwedComment:
if o.Text == "" {
return fmt.Errorf("delivery: an owed %s has no text", o.Act)
}
case OwedNote:
if o.Text == "" {
return fmt.Errorf("delivery: an owed note has no text")
}
if o.Ref == "" {
return fmt.Errorf("delivery: an owed note has no ref")
}
default:
return fmt.Errorf("delivery: the owed act %q is not status, view, note or comment", o.Act)
}
return nil
}
// EncodeMerge is m as JSON at Version; it refuses a merged pull request without its merge commit, a refused one
// without the forge's words and a state outside the list.
func EncodeMerge(m Merge) ([]byte, error) {
if err := m.check(); err != nil {
return nil, err
}
m.V = Version
return json.Marshal(m)
}
// DecodeMerge reads a merge; it refuses an unknown version (ErrUnknownVersion) and what EncodeMerge refuses.
func DecodeMerge(data []byte) (Merge, error) {
var m Merge
if err := decode(data, &m); err != nil {
return Merge{}, err
}
if err := m.check(); err != nil {
return Merge{}, err
}
return m, nil
}
// EncodeAsk is a as JSON at Version; it refuses an ask whose id is not the one its group and heads give, or with
// no option.
func EncodeAsk(a Ask) ([]byte, error) {
if err := a.check(); err != nil {
return nil, err
}
a.V = Version
return json.Marshal(a)
}
// DecodeAsk reads an ask; it refuses an unknown version (ErrUnknownVersion) and what EncodeAsk refuses.
func DecodeAsk(data []byte) (Ask, error) {
var a Ask
if err := decode(data, &a); err != nil {
return Ask{}, err
}
if err := a.check(); err != nil {
return Ask{}, err
}
return a, nil
}
// EncodeAnswer is a as JSON at Version; it refuses an answer missing its option, warrant or words.
func EncodeAnswer(a Answer) ([]byte, error) {
if err := a.check(); err != nil {
return nil, err
}
a.V = Version
return json.Marshal(a)
}
// DecodeAnswer reads an answer; it refuses an unknown version (ErrUnknownVersion) and what EncodeAnswer refuses.
func DecodeAnswer(data []byte) (Answer, error) {
var a Answer
if err := decode(data, &a); err != nil {
return Answer{}, err
}
if err := a.check(); err != nil {
return Answer{}, err
}
return a, nil
}
// EncodeOwed is o as JSON at Version; it refuses an act missing what it needs.
func EncodeOwed(o Owed) ([]byte, error) {
if err := o.check(); err != nil {
return nil, err
}
o.V = Version
return json.Marshal(o)
}
// DecodeOwed reads an owed act; it refuses an unknown version (ErrUnknownVersion) and what EncodeOwed refuses.
func DecodeOwed(data []byte) (Owed, error) {
var o Owed
if err := decode(data, &o); err != nil {
return Owed{}, err
}
if err := o.check(); err != nil {
return Owed{}, err
}
return o, nil
}
// SameMerge says whether two merges are the same value, so a writer writes only when they differ.
func SameMerge(a, b Merge) bool { return reflect.DeepEqual(a, b) }
// SameAsk says whether two asks are the same value.
func SameAsk(a, b Ask) bool { return reflect.DeepEqual(a, b) }
// SameAnswer says whether two answers are the same value.
func SameAnswer(a, b Answer) bool { return a == b }
// SameOwed says whether two owed acts are the same value.
func SameOwed(a, b Owed) bool { return a == b }
+301
View File
@@ -0,0 +1,301 @@
package delivery
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
)
func gateHeads() []Head {
return []Head{
{Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa"},
{Repository: "novox/mesh-control", Number: 7, Head: "bbbb"},
}
}
func TestMergeKeys(t *testing.T) { // 1201/AC1
if k, err := MergesKey("feat/x"); err != nil || k != "feat/x" {
t.Fatalf("merges key = %q, %v", k, err)
}
if _, err := MergesKey("a.b"); err == nil {
t.Error("a group with a dot accepted")
}
hh, _ := HeadsHash(gateHeads())
key, err := MergeAskKey("feat/x", gateHeads())
if err != nil || key != "merge.feat/x."+hh {
t.Fatalf("ask key = %q, %v", key, err)
}
kind, group, hash, err := ParseAskKey(key)
if err != nil || kind != AskKindMerge || group != "feat/x" || hash != hh {
t.Fatalf("parse ask key = %q %q %q %v", kind, group, hash, err)
}
if AnswerKey(key) != key {
t.Error("the answer's key is not the ask's key")
}
if _, err := MergeAskKey("feat/x", nil); err == nil {
t.Error("an ask key without heads accepted")
}
if _, err := MergeAskKey("", gateHeads()); err == nil {
t.Error("an ask key without a group accepted")
}
for _, bad := range []string{"", "merge", "merge.g", "merge.g.h.x", "other.g.0123456789ab", "merge..0123456789ab", "merge.g.nothex"} {
if _, _, _, err := ParseAskKey(bad); err == nil {
t.Errorf("parse ask key %q accepted", bad)
}
}
}
func TestAskID(t *testing.T) { // 1201/AC2
k1, _ := MergeAskKey("feat/x", gateHeads())
k2, _ := MergeAskKey("feat/y", gateHeads())
k3, _ := MergeAskKey("feat/x", gateHeads()[:1])
long, _ := MergeAskKey("feat/"+strings.Repeat("long-", 20), gateHeads())
long2, _ := MergeAskKey("feat/"+strings.Repeat("long-", 20)+"2", gateHeads())
slash, _ := MergeAskKey("a/b", gateHeads())
under, _ := MergeAskKey("a_b", gateHeads())
eq, _ := MergeAskKey("feat=x", gateHeads())
eqs, _ := MergeAskKey("feat_x", gateHeads())
seen := map[string]string{}
for _, k := range []string{k1, k2, k3, long, long2, slash, under, eq, eqs, "=odd.key"} {
id := AskID(k)
if strings.Contains(id, ".") {
t.Errorf("id %q holds a dot", id)
}
if !asks.UsableID(id) {
t.Errorf("id %q is not usable by the router", id)
}
if other, dup := seen[id]; dup {
t.Errorf("keys %q and %q give the id %q", k, other, id)
}
seen[id] = k
}
if AskID(k1) != AskID(k1) {
t.Error("the id is not stable")
}
}
func TestOwedKey(t *testing.T) { // 1201/AC3
for _, act := range []string{"status=mesh/merge-gate", "view", "note", "comment"} {
key, err := OwedKey("novox/mesh-sdk", 36, "aaaa", act)
if err != nil || key != "novox/mesh-sdk.36.aaaa."+act {
t.Fatalf("owed key for %s = %q, %v", act, key, err)
}
repo, number, head, gotAct, err := ParseOwedKey(key)
if err != nil || repo != "novox/mesh-sdk" || number != 36 || head != "aaaa" || gotAct != act {
t.Fatalf("parse %q = %q %d %q %q %v", key, repo, number, head, gotAct, err)
}
}
kind, context, err := ParseOwedAct("status=mesh/merge-gate")
if err != nil || kind != OwedStatus || context != "mesh/merge-gate" {
t.Errorf("owed act = %q %q %v", kind, context, err)
}
for _, bad := range []string{"", "merge", "status", "status=", "view=x", "status=a.b", "statuses=x"} {
if _, err := OwedKey("novox/mesh-sdk", 36, "aaaa", bad); err == nil {
t.Errorf("owed act %q accepted", bad)
}
}
if _, err := OwedKey("nope", 36, "aaaa", "view"); err == nil {
t.Error("a bad repository accepted")
}
if _, err := OwedKey("novox/mesh-sdk", 0, "aaaa", "view"); err == nil {
t.Error("pull request number 0 accepted")
}
if _, err := OwedKey("novox/mesh-sdk", 36, "", "view"); err == nil {
t.Error("an empty head accepted")
}
for _, bad := range []string{"", "novox/mesh-sdk.36.aaaa", "novox/mesh-sdk.x.aaaa.view", "novox/mesh-sdk.36.aaaa.merge", "a.b.36.aaaa.view"} {
if _, _, _, _, err := ParseOwedKey(bad); err == nil {
t.Errorf("parse owed key %q accepted", bad)
}
}
}
func sampleMerge() Merge {
return Merge{Group: "feat/x", Ask: "merge.feat/x.0123456789ab", Pulls: []MergePull{
{Head: gateHeads()[0], State: MergeMerged, Commit: "cccc"},
{Head: gateHeads()[1], State: MergeRefused, Words: "the forge says no"},
{Head: Head{Repository: "novox/other", Number: 1, Head: "dddd"}, State: MergeMerging},
}}
}
func sampleAsk() Ask {
key, _ := MergeAskKey("feat/x", gateHeads())
return Ask{Group: "feat/x", Heads: gateHeads(), Ask: asks.Ask{
ID: AskID(key), Headline: "Merge into main: group feat/x", Explanation: "The group passed its checks.",
Who: asks.Operator, Expires: time.Date(2026, 10, 20, 8, 0, 0, 0, time.UTC), OnExpiry: "nothing is merged",
Options: []asks.Option{{ID: "approve", Label: "Approve", Does: "merges", Level: asks.Approve, Binds: "sha256:00"}},
}}
}
func sampleOwed() []Owed {
return []Owed{
{Act: OwedStatus, Context: "mesh/merge-gate", State: "success"},
{Act: OwedView, Text: "the delivery reaches ace"},
{Act: OwedNote, Text: "plan: mesh-mail to ace", Ref: "mesh-plan"},
{Act: OwedComment, Text: "merged"},
}
}
func TestGateValuesRoundTrip(t *testing.T) { // 1201/AC4
hasV := func(name string, data []byte, err error) {
t.Helper()
if err != nil {
t.Fatalf("%s: %v", name, err)
}
var probe map[string]any
if json.Unmarshal(data, &probe) != nil || probe["v"] != float64(1) {
t.Fatalf("%s: %s does not carry v:1", name, data)
}
}
m := sampleMerge()
data, err := EncodeMerge(m)
hasV("merge", data, err)
m.V = Version
if got, err := DecodeMerge(data); err != nil || !SameMerge(got, m) {
t.Errorf("merge round trip: %+v %v", got, err)
}
a := sampleAsk()
data, err = EncodeAsk(a)
hasV("ask", data, err)
a.V = Version
if got, err := DecodeAsk(data); err != nil || !SameAsk(got, a) {
t.Errorf("ask round trip: %+v %v", got, err)
}
an := Answer{Option: "approve", Warrant: "w-1", Words: "approved on telegram"}
data, err = EncodeAnswer(an)
hasV("answer", data, err)
an.V = Version
if got, err := DecodeAnswer(data); err != nil || !SameAnswer(got, an) {
t.Errorf("answer round trip: %+v %v", got, err)
}
for _, o := range sampleOwed() {
data, err = EncodeOwed(o)
hasV("owed "+o.Act, data, err)
o.V = Version
if got, err := DecodeOwed(data); err != nil || !SameOwed(got, o) {
t.Errorf("owed %s round trip: %+v %v", o.Act, got, err)
}
}
for name, dec := range map[string]func([]byte) error{
"merge": func(b []byte) error { _, err := DecodeMerge(b); return err },
"ask": func(b []byte) error { _, err := DecodeAsk(b); return err },
"answer": func(b []byte) error { _, err := DecodeAnswer(b); return err },
"owed": func(b []byte) error { _, err := DecodeOwed(b); return err },
} {
for _, in := range []string{`{"group":"g"}`, `{"v":2}`} {
if err := dec([]byte(in)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("%s decode of %s = %v, want ErrUnknownVersion", name, in, err)
}
}
}
}
func TestGateValuesRefused(t *testing.T) { // 1201/AC5
wantErr := func(name string, err error, words ...string) {
t.Helper()
if err == nil {
t.Errorf("%s accepted", name)
return
}
for _, w := range words {
if !strings.Contains(err.Error(), w) {
t.Errorf("%s: error %q does not name %q", name, err, w)
}
}
}
m := sampleMerge()
m.Pulls[0].Commit = ""
_, err := EncodeMerge(m)
wantErr("merged without a merge commit", err, "commit")
m = sampleMerge()
m.Pulls[1].Words = ""
_, err = EncodeMerge(m)
wantErr("refused without words", err, "words")
m = sampleMerge()
m.Pulls[2].State = "done"
_, err = EncodeMerge(m)
wantErr("unknown state", err, "done")
m = sampleMerge()
m.Pulls = nil
_, err = EncodeMerge(m)
wantErr("merge without pulls", err, "pull")
m = sampleMerge()
m.Ask = ""
_, err = EncodeMerge(m)
wantErr("merge without its ask", err, "ask")
a := sampleAsk()
a.Ask.ID = "something-else"
_, err = EncodeAsk(a)
wantErr("ask with another id", err, "something-else", AskID("merge.feat/x."+mustHash(gateHeads())))
a = sampleAsk()
a.Ask.Options = nil
_, err = EncodeAsk(a)
wantErr("ask without options", err, "option")
a = sampleAsk()
a.Heads = nil
_, err = EncodeAsk(a)
wantErr("ask without heads", err, "pull request")
for name, an := range map[string]Answer{
"option": {Warrant: "w", Words: "x"},
"warrant": {Option: "approve", Words: "x"},
"words": {Option: "approve", Warrant: "w"},
} {
_, err = EncodeAnswer(an)
wantErr("answer without "+name, err, name)
}
for name, o := range map[string]Owed{
"context": {Act: OwedStatus, State: "success"},
"state": {Act: OwedStatus, Context: "mesh/merge-gate", State: "green"},
"text": {Act: OwedView},
"ref": {Act: OwedNote, Text: "line"},
"comment": {Act: OwedComment},
"act": {Act: "merge"},
} {
_, err = EncodeOwed(o)
wantErr("owed missing "+name, err)
}
_, err = EncodeOwed(Owed{Act: OwedNote, Text: "line"})
wantErr("note without ref", err, "ref")
_, err = EncodeOwed(Owed{Act: OwedStatus, State: "success"})
wantErr("status without context", err, "context")
}
func mustHash(h []Head) string { s, _ := HeadsHash(h); return s }
func TestGateSame(t *testing.T) { // 1201/AC6
if !SameMerge(sampleMerge(), sampleMerge()) || !SameAsk(sampleAsk(), sampleAsk()) {
t.Error("equal values differ")
}
m := sampleMerge()
m.Pulls[0].Commit = "other"
if SameMerge(m, sampleMerge()) {
t.Error("different merges are the same")
}
a := sampleAsk()
a.Ask.Headline = "other"
if SameAsk(a, sampleAsk()) {
t.Error("different asks are the same")
}
if !SameAnswer(Answer{Option: "a", Warrant: "w", Words: "x"}, Answer{Option: "a", Warrant: "w", Words: "x"}) ||
SameAnswer(Answer{Option: "a", Warrant: "w", Words: "x"}, Answer{Option: "b", Warrant: "w", Words: "x"}) {
t.Error("SameAnswer wrong")
}
if !SameOwed(sampleOwed()[0], sampleOwed()[0]) || SameOwed(sampleOwed()[0], sampleOwed()[1]) {
t.Error("SameOwed wrong")
}
}
func TestGateBuckets(t *testing.T) {
if MergesBucket != "mesh-delivery-gate_merges" || AsksBucket != "mesh-delivery-gate_asks" ||
OwedBucket != "mesh-delivery-gate_owed" || AnswersBucket != "mesh-delivery-teller_answers" ||
MergesHistory != 10 || AsksHistory != 5 || OwedHistory != 5 || AnswersHistory != 5 {
t.Error("a bucket or history differs from the decided layout")
}
}
+420
View File
@@ -0,0 +1,420 @@
package delivery
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"reflect"
"sort"
"strings"
)
// The modules of the parts release 3 adds (novox/hq ADR 0437): their buckets are their own state, `<module>_<name>`.
const (
GateModule = "mesh-delivery-gate"
IndexerModule = "mesh-delivery-indexer"
ForgeBridgeModule = "mesh-delivery-forge-bridge"
TellerModule = "mesh-delivery-teller"
)
// The gate's source bucket and the indexer's candidates and modules buckets (ADR 0437 §5). The source bucket has
// one key, SourceMainKey: every repository's merged head and the group that moved it, written by the gate alone by
// compare-and-set; its revision is the mesh's source revision. The indexer writes the other two.
const (
SourceBucket = GateModule + "_source"
SourceMainKey = "main"
SourceHistory = 64
CandidatesBucket = IndexerModule + "_candidates"
// CandidatesHistory is the revisions the candidates bucket keeps per key.
CandidatesHistory = 10
ModulesBucket = IndexerModule + "_modules"
// ModulesHistory is the revisions the modules bucket keeps per key.
ModulesHistory = 20
)
// ForgeStream is the forge bridge's stream as its manifest declares it; the broker derives the real names.
const ForgeStream = "forge"
// The kinds of forge message, the last part of a forge event's name.
const (
ForgePull = "pull"
ForgeMerged = "merged"
ForgeRefused = "refused"
)
// The actions of a pull message.
const (
PullOpened = "opened"
PullUpdated = "updated"
PullClosed = "closed"
)
// SourcePrefix opens every source fingerprint.
const SourcePrefix = "src1:"
// CandidatesKey is the key, in CandidatesBucket, of a group: `<group>`, the branch name the group's pull requests
// share (a slash is allowed); a lone pull request is a group of one under its branch name.
func CandidatesKey(group string) (string, error) {
if err := checkName("group", group); err != nil {
return "", err
}
return group, nil
}
// ParseCandidatesKey is the inverse of CandidatesKey.
func ParseCandidatesKey(key string) (group string, err error) {
if group, err = CandidatesKey(key); err != nil {
return "", fmt.Errorf("delivery: %q is not a candidates key: %w", key, err)
}
return group, nil
}
// ModulesKey is the key, in ModulesBucket, of a module: `<module>`.
func ModulesKey(module string) (string, error) {
if err := checkName("module", module); err != nil {
return "", err
}
return module, nil
}
// ParseModulesKey is the inverse of ModulesKey.
func ParseModulesKey(key string) (module string, err error) {
if module, err = ModulesKey(key); err != nil {
return "", fmt.Errorf("delivery: %q is not a modules key: %w", key, err)
}
return module, nil
}
// checkRepository refuses a repository that is not `<owner>/<name>`: exactly one slash, each part a valid name.
func checkRepository(repository string) error {
owner, name, ok := strings.Cut(repository, "/")
if !ok || strings.Contains(name, "/") {
return fmt.Errorf("delivery: the repository %q is not <owner>/<name>, with exactly one slash", repository)
}
if err := checkName("repository owner", owner); err != nil {
return fmt.Errorf("delivery: the repository %q: %w", repository, err)
}
if err := checkName("repository name", name); err != nil {
return fmt.Errorf("delivery: the repository %q: %w", repository, err)
}
if strings.Contains(owner, "/") {
return fmt.Errorf("delivery: the repository %q is not <owner>/<name>, with exactly one slash", repository)
}
return nil
}
func checkForgeKind(kind string) error {
switch kind {
case ForgePull, ForgeMerged, ForgeRefused:
return nil
}
return fmt.Errorf("delivery: the forge kind %q is not pull, merged or refused", kind)
}
// ForgeEvent is the name of a forge message of the given kind: `forge.<owner>.<name>.<kind>`.
func ForgeEvent(repository, kind string) (string, error) {
if err := checkRepository(repository); err != nil {
return "", err
}
if err := checkForgeKind(kind); err != nil {
return "", err
}
owner, name, _ := strings.Cut(repository, "/")
return "forge." + owner + "." + name + "." + kind, nil
}
// ParseForgeEvent is the inverse of ForgeEvent.
func ParseForgeEvent(event string) (repository, kind string, err error) {
parts := strings.Split(event, ".")
if len(parts) != 4 || parts[0] != "forge" {
return "", "", fmt.Errorf("delivery: %q is not a forge event, forge.<owner>.<name>.<kind>", event)
}
repository = parts[1] + "/" + parts[2]
if _, err := ForgeEvent(repository, parts[3]); err != nil {
return "", "", fmt.Errorf("delivery: %q is not a forge event: %w", event, err)
}
return repository, parts[3], nil
}
// Head is one pull request at one commit: the repository as `<owner>/<name>` and the full sha.
type Head struct {
Repository string `json:"repository"`
Number int `json:"number"`
Head string `json:"head"`
}
// HeadsHash is the first 12 hex digits of the sha256 over the lines `<repository>#<number>=<head>`, sorted and
// joined by a newline: the same for every order of heads, different when any head changes. An empty list is refused.
func HeadsHash(heads []Head) (string, error) {
if len(heads) == 0 {
return "", fmt.Errorf("delivery: a heads hash needs at least one pull request")
}
lines := make([]string, len(heads))
for i, h := range heads {
lines[i] = fmt.Sprintf("%s#%d=%s", h.Repository, h.Number, h.Head)
}
sort.Strings(lines)
sum := sha256.Sum256([]byte(strings.Join(lines, "\n")))
return hex.EncodeToString(sum[:])[:12], nil
}
// Source is the value of SourceMainKey: every repository's merged head (repository to full sha, not only the
// group's) and the group that moved it.
type Source struct {
V int `json:"v"`
Group string `json:"group"`
Heads map[string]string `json:"heads"`
}
func (s Source) check() error {
if len(s.Heads) == 0 {
return fmt.Errorf("delivery: a source names no repository")
}
for repository, head := range s.Heads {
if repository == "" {
return fmt.Errorf("delivery: a source names a repository with no name")
}
if head == "" {
return fmt.Errorf("delivery: the source's head of %s is empty", repository)
}
}
return nil
}
// Candidate is the value of a candidates key: the group's pull requests and heads, the heads hash, the base (the
// source revision it was previewed onto) and the modules each pull request touches. It carries no fingerprint
// until release 4 (ADR 0437 §5).
type Candidate struct {
V int `json:"v"`
Group string `json:"group"`
Base uint64 `json:"base"`
HeadsHash string `json:"heads"`
Pulls []CandidatePull `json:"pulls"`
}
// CandidatePull is one pull request of a candidate and the modules it touches.
type CandidatePull struct {
Head
Modules []string `json:"modules"`
}
func (c Candidate) check() error {
if len(c.Pulls) == 0 {
return fmt.Errorf("delivery: a candidate names no pull request")
}
heads := make([]Head, len(c.Pulls))
for i, p := range c.Pulls {
heads[i] = p.Head
}
want, _ := HeadsHash(heads)
if c.HeadsHash != want {
return fmt.Errorf("delivery: the candidate's heads hash %q is not the hash %q of its pull requests' heads", c.HeadsHash, want)
}
return nil
}
// Module is the value of a modules key: the module's source fingerprint, or none and why (a build that resolves
// packages when built gets none, said here), and where it was computed: Repository and Commit are the trunk merge,
// Src the source revision then (0 before the gate writes source.main). Exactly one of FP and None is set.
type Module struct {
V int `json:"v"`
FP string `json:"fp,omitempty"`
None string `json:"none,omitempty"`
Src uint64 `json:"src,omitempty"`
Repository string `json:"repository"`
Commit string `json:"commit"`
}
func (m Module) check() error {
switch {
case m.FP != "" && m.None != "":
return fmt.Errorf("delivery: a module has both a fingerprint and a reason for none")
case m.FP == "" && m.None == "":
return fmt.Errorf("delivery: a module has neither a fingerprint nor a reason for none")
case m.FP != "" && !strings.HasPrefix(m.FP, SourcePrefix):
return fmt.Errorf("delivery: the module's fingerprint %q does not start with %s", m.FP, SourcePrefix)
}
return nil
}
// Forge is a message of the forge bridge's stream, trimmed to repository, branch, pull request and head. A pull
// message says Action; a merged message carries the merge commit, the tree of every directory holding a
// module.json and the repository's root tree; a refused message names the owed act it answers and the forge's own
// words. A pull message may also carry the module directories its head changes (Dirs, in the order the forge said) and
// DirsKnown, true when the forge could say them: known and empty is DirsKnown with no Dirs, not known is neither
// (the forge's module_dirs and module_dirs_said, mesh-catalog modules/gitea/trunk.ts). Other messages carry neither.
type Forge struct {
V int `json:"v"`
Kind string `json:"kind"`
Repository string `json:"repository"`
Branch string `json:"branch,omitempty"`
Number int `json:"number,omitempty"`
Head string `json:"head,omitempty"`
Action string `json:"action,omitempty"`
Dirs []string `json:"dirs,omitempty"`
DirsKnown bool `json:"dirs-known,omitempty"`
Merge string `json:"merge,omitempty"`
Trees map[string]string `json:"trees,omitempty"`
Root string `json:"root,omitempty"`
Act string `json:"act,omitempty"`
Words string `json:"words,omitempty"`
}
func (f Forge) check() error {
if err := checkRepository(f.Repository); err != nil {
return err
}
if err := checkForgeKind(f.Kind); err != nil {
return err
}
switch f.Kind {
case ForgePull:
if f.Number <= 0 {
return fmt.Errorf("delivery: a pull message needs its pull request number")
}
if f.Head == "" {
return fmt.Errorf("delivery: a pull message needs its head")
}
switch f.Action {
case PullOpened, PullUpdated, PullClosed:
default:
return fmt.Errorf("delivery: the pull action %q is not opened, updated or closed", f.Action)
}
if len(f.Dirs) > 0 && !f.DirsKnown {
return fmt.Errorf("delivery: a pull message with module directories must say it knows them (dirs-known)")
}
case ForgeMerged:
if len(f.Dirs) > 0 || f.DirsKnown {
return fmt.Errorf("delivery: a merged message carries no module directories")
}
if f.Number <= 0 {
return fmt.Errorf("delivery: a merged message needs its pull request number")
}
if f.Head == "" {
return fmt.Errorf("delivery: a merged message needs its head")
}
if f.Merge == "" {
return fmt.Errorf("delivery: a merged message needs its merge commit")
}
if f.Root == "" {
return fmt.Errorf("delivery: a merged message needs the repository's root tree")
}
if len(f.Trees) == 0 {
return fmt.Errorf("delivery: a merged message needs the tree of at least one directory holding a module.json")
}
for dir, tree := range f.Trees {
if tree == "" {
return fmt.Errorf("delivery: a merged message's tree of the directory %q is empty", dir)
}
}
case ForgeRefused:
if len(f.Dirs) > 0 || f.DirsKnown {
return fmt.Errorf("delivery: a refused message carries no module directories")
}
if f.Act == "" {
return fmt.Errorf("delivery: a refused message needs the act it answers")
}
if f.Words == "" {
return fmt.Errorf("delivery: a refused message needs the forge's words")
}
}
return nil
}
// EncodeSource is s as JSON at Version; it refuses a source with no repository or an empty head.
func EncodeSource(s Source) ([]byte, error) {
if err := s.check(); err != nil {
return nil, err
}
s.V = Version
return json.Marshal(s)
}
// DecodeSource reads a source; it refuses an unknown version (ErrUnknownVersion) and what EncodeSource refuses.
func DecodeSource(data []byte) (Source, error) {
var s Source
if err := decode(data, &s); err != nil {
return Source{}, err
}
if err := s.check(); err != nil {
return Source{}, err
}
return s, nil
}
// EncodeCandidate is c as JSON at Version; it refuses a heads hash that is not the hash of its pull requests' heads.
func EncodeCandidate(c Candidate) ([]byte, error) {
if err := c.check(); err != nil {
return nil, err
}
c.V = Version
return json.Marshal(c)
}
// DecodeCandidate reads a candidate; it refuses an unknown version (ErrUnknownVersion) and what EncodeCandidate refuses.
func DecodeCandidate(data []byte) (Candidate, error) {
var c Candidate
if err := decode(data, &c); err != nil {
return Candidate{}, err
}
if err := c.check(); err != nil {
return Candidate{}, err
}
return c, nil
}
// EncodeModule is m as JSON at Version; it refuses both or neither of a fingerprint and a reason, and a
// fingerprint not of the form src1:.
func EncodeModule(m Module) ([]byte, error) {
if err := m.check(); err != nil {
return nil, err
}
m.V = Version
return json.Marshal(m)
}
// DecodeModule reads a module; it refuses an unknown version (ErrUnknownVersion) and what EncodeModule refuses.
func DecodeModule(data []byte) (Module, error) {
var m Module
if err := decode(data, &m); err != nil {
return Module{}, err
}
if err := m.check(); err != nil {
return Module{}, err
}
return m, nil
}
// EncodeForge is f as JSON at Version; it refuses a message missing what its kind needs.
func EncodeForge(f Forge) ([]byte, error) {
if err := f.check(); err != nil {
return nil, err
}
f.V = Version
return json.Marshal(f)
}
// DecodeForge reads a forge message; it refuses an unknown version (ErrUnknownVersion) and what EncodeForge refuses.
func DecodeForge(data []byte) (Forge, error) {
var f Forge
if err := decode(data, &f); err != nil {
return Forge{}, err
}
if err := f.check(); err != nil {
return Forge{}, err
}
return f, nil
}
// SameSource says whether two sources are the same value, so a writer writes only when they differ.
func SameSource(a, b Source) bool { return reflect.DeepEqual(a, b) }
// SameCandidate says whether two candidates are the same value.
func SameCandidate(a, b Candidate) bool { return reflect.DeepEqual(a, b) }
// SameModule says whether two modules are the same value.
func SameModule(a, b Module) bool { return a == b }
// SameForge says whether two forge messages are the same value.
func SameForge(a, b Forge) bool { return reflect.DeepEqual(a, b) }
+340
View File
@@ -0,0 +1,340 @@
package delivery
import (
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
)
func TestSourceKeys(t *testing.T) { // 1200/AC1
if k, err := CandidatesKey("feat/delivery-source-keys"); err != nil || k != "feat/delivery-source-keys" {
t.Fatalf("candidates key = %q, %v", k, err)
}
if g, err := ParseCandidatesKey("feat/x"); err != nil || g != "feat/x" {
t.Fatalf("parse candidates = %q, %v", g, err)
}
if k, err := ModulesKey("mesh-mail"); err != nil || k != "mesh-mail" {
t.Fatalf("modules key = %q, %v", k, err)
}
if m, err := ParseModulesKey("mesh-mail"); err != nil || m != "mesh-mail" {
t.Fatalf("parse modules = %q, %v", m, err)
}
for _, bad := range []string{"", "a.b", "a*b", "a>b", "a b", "a\tb"} {
if _, err := CandidatesKey(bad); err == nil {
t.Errorf("candidates key %q accepted", bad)
}
if _, err := ModulesKey(bad); err == nil {
t.Errorf("modules key %q accepted", bad)
}
if _, err := ParseCandidatesKey(bad); err == nil {
t.Errorf("parse candidates %q accepted", bad)
}
if _, err := ParseModulesKey(bad); err == nil {
t.Errorf("parse modules %q accepted", bad)
}
}
ev, err := ForgeEvent("novox/mesh-sdk", ForgeMerged)
if err != nil || ev != "forge.novox.mesh-sdk.merged" {
t.Fatalf("forge event = %q, %v", ev, err)
}
repo, kind, err := ParseForgeEvent(ev)
if err != nil || repo != "novox/mesh-sdk" || kind != ForgeMerged {
t.Fatalf("parse forge event = %q %q %v", repo, kind, err)
}
for _, k := range []string{ForgePull, ForgeMerged, ForgeRefused} {
if _, err := ForgeEvent("a/b", k); err != nil {
t.Errorf("kind %s refused: %v", k, err)
}
}
for _, bad := range []struct{ repo, kind string }{
{"", "pull"}, {"novox", "pull"}, {"a/b/c", "pull"}, {"/b", "pull"}, {"a/", "pull"},
{"a.b/c", "pull"}, {"a/b c", "pull"}, {"a/b", "closed"}, {"a/b", ""},
} {
if _, err := ForgeEvent(bad.repo, bad.kind); err == nil {
t.Errorf("forge event %q %q accepted", bad.repo, bad.kind)
} else if bad.kind == "closed" && !strings.Contains(err.Error(), "closed") {
t.Errorf("error does not name the kind: %v", err)
}
}
for _, bad := range []string{"", "forge.a.b", "forge.a.b.closed", "x.a.b.pull", "forge.a.b.pull.x", "forge..b.pull"} {
if _, _, err := ParseForgeEvent(bad); err == nil {
t.Errorf("parse forge event %q accepted", bad)
}
}
}
func TestHeadsHash(t *testing.T) { // 1200/AC2
a := Head{Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa"}
b := Head{Repository: "novox/mesh-control", Number: 7, Head: "bbbb"}
h1, err := HeadsHash([]Head{a, b})
if err != nil {
t.Fatal(err)
}
h2, _ := HeadsHash([]Head{b, a})
if h1 != h2 {
t.Errorf("order changes the hash: %s %s", h1, h2)
}
if len(h1) != 12 || strings.Trim(h1, "0123456789abcdef") != "" {
t.Errorf("hash %q is not 12 hex digits", h1)
}
for _, changed := range []Head{
{Repository: "novox/mesh-sdk", Number: 36, Head: "aaab"},
{Repository: "novox/mesh-sdk", Number: 37, Head: "aaaa"},
{Repository: "novox/other", Number: 36, Head: "aaaa"},
} {
h3, _ := HeadsHash([]Head{changed, b})
if h3 == h1 {
t.Errorf("changing %v keeps the hash", changed)
}
}
if _, err := HeadsHash(nil); err == nil {
t.Error("an empty list is accepted")
}
}
func sampleSource() Source {
return Source{Group: "feat/x", Heads: map[string]string{"novox/mesh-sdk": "abc123", "novox/mesh-control": "def456"}}
}
func sampleCandidate() Candidate {
pulls := []CandidatePull{
{Head: Head{Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa"}, Modules: []string{"mesh-mail"}},
{Head: Head{Repository: "novox/mesh-control", Number: 7, Head: "bbbb"}, Modules: []string{"mesh-controller"}},
}
hh, _ := HeadsHash([]Head{pulls[0].Head, pulls[1].Head})
return Candidate{Group: "feat/x", Base: 12, HeadsHash: hh, Pulls: pulls}
}
func sampleForges() []Forge {
return []Forge{
{Kind: ForgePull, Repository: "novox/mesh-sdk", Branch: "feat/x", Number: 36, Head: "aaaa", Action: "opened"},
{Kind: ForgeMerged, Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa", Merge: "cccc",
Trees: map[string]string{"modules/a": "t1"}, Root: "t0"},
{Kind: ForgeRefused, Repository: "novox/mesh-sdk", Act: "novox/mesh-sdk.36.aaaa.view", Words: "403 forbidden"},
}
}
func TestSourceValuesRoundTrip(t *testing.T) { // 1200/AC3
check := func(name string, data []byte, err error) {
t.Helper()
if err != nil {
t.Fatalf("%s: %v", name, err)
}
var probe map[string]any
if err := json.Unmarshal(data, &probe); err != nil || probe["v"] != float64(1) {
t.Fatalf("%s: json %s does not carry v:1", name, data)
}
}
s := sampleSource()
data, err := EncodeSource(s)
check("source", data, err)
s.V = Version
if got, err := DecodeSource(data); err != nil || !SameSource(got, s) {
t.Errorf("source round trip: %+v %v", got, err)
}
c := sampleCandidate()
data, err = EncodeCandidate(c)
check("candidate", data, err)
c.V = Version
if got, err := DecodeCandidate(data); err != nil || !SameCandidate(got, c) {
t.Errorf("candidate round trip: %+v %v", got, err)
}
for _, m := range []Module{
{FP: "src1:abc", Src: 5, Repository: "novox/mesh-sdk", Commit: "cccc"},
{None: "a build that resolves packages when built", Repository: "novox/mesh-sdk", Commit: "cccc"},
} {
data, err = EncodeModule(m)
check("module", data, err)
m.V = Version
if got, err := DecodeModule(data); err != nil || !SameModule(got, m) {
t.Errorf("module round trip: %+v %v", got, err)
}
}
for _, f := range sampleForges() {
data, err = EncodeForge(f)
check("forge "+f.Kind, data, err)
f.V = Version
if got, err := DecodeForge(data); err != nil || !SameForge(got, f) {
t.Errorf("forge %s round trip: %+v %v", f.Kind, got, err)
}
}
for name, dec := range map[string]func([]byte) error{
"source": func(b []byte) error { _, err := DecodeSource(b); return err },
"candidate": func(b []byte) error { _, err := DecodeCandidate(b); return err },
"module": func(b []byte) error { _, err := DecodeModule(b); return err },
"forge": func(b []byte) error { _, err := DecodeForge(b); return err },
} {
for _, in := range []string{`{"group":"g"}`, `{"v":2,"group":"g"}`} {
if err := dec([]byte(in)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("%s decode of %s = %v, want ErrUnknownVersion", name, in, err)
}
}
}
}
func TestSourceValuesRefused(t *testing.T) { // 1200/AC4
wantErr := func(name string, err error, words ...string) {
t.Helper()
if err == nil {
t.Errorf("%s accepted", name)
return
}
for _, w := range words {
if !strings.Contains(err.Error(), w) {
t.Errorf("%s: error %q does not name %q", name, err, w)
}
}
}
_, err := EncodeSource(Source{Group: "g"})
wantErr("source without repositories", err, "repositor")
_, err = EncodeSource(Source{Group: "g", Heads: map[string]string{"a/b": ""}})
wantErr("source with an empty head", err, "a/b")
_, err = EncodeSource(Source{Group: "g", Heads: map[string]string{"": "x"}})
wantErr("source with an empty repository", err, "repositor")
c := sampleCandidate()
c.HeadsHash = "000000000000"
_, err = EncodeCandidate(c)
wantErr("candidate with a stale hash", err, "heads hash")
c = sampleCandidate()
c.Pulls = nil
_, err = EncodeCandidate(c)
wantErr("candidate without pulls", err, "pull")
_, err = EncodeModule(Module{FP: "src1:x", None: "why", Repository: "a/b", Commit: "c"})
wantErr("module with both", err, "both")
_, err = EncodeModule(Module{Repository: "a/b", Commit: "c"})
wantErr("module with neither", err, "fingerprint", "none")
_, err = EncodeModule(Module{FP: "abc", Repository: "a/b", Commit: "c"})
wantErr("module with a bad fingerprint", err, "src1:")
f := sampleForges()[1]
f.Merge = ""
_, err = EncodeForge(f)
wantErr("merged without merge commit", err, "merge")
f = sampleForges()[1]
f.Root = ""
_, err = EncodeForge(f)
wantErr("merged without root", err, "root")
f = sampleForges()[1]
f.Trees = nil
_, err = EncodeForge(f)
wantErr("merged without trees", err, "tree")
f = sampleForges()[1]
f.Trees = map[string]string{"modules/a": ""}
_, err = EncodeForge(f)
wantErr("merged with an empty tree", err, "modules/a")
f = sampleForges()[0]
f.Action = "reopened"
_, err = EncodeForge(f)
wantErr("pull with an unknown action", err, "reopened")
f = sampleForges()[0]
f.Head = ""
_, err = EncodeForge(f)
wantErr("pull without head", err, "head")
f = sampleForges()[2]
f.Words = ""
_, err = EncodeForge(f)
wantErr("refused without words", err, "words")
f = sampleForges()[2]
f.Act = ""
_, err = EncodeForge(f)
wantErr("refused without act", err, "act")
_, err = EncodeForge(Forge{Kind: "closed", Repository: "a/b"})
wantErr("unknown kind", err, "closed")
_, err = EncodeForge(Forge{Kind: ForgePull, Repository: "nope", Number: 1, Head: "h", Action: "opened"})
wantErr("bad repository", err, "nope")
}
func TestSourceSame(t *testing.T) { // 1200/AC5
if !SameSource(sampleSource(), sampleSource()) {
t.Error("equal sources differ")
}
s := sampleSource()
s.Heads["novox/mesh-sdk"] = "zzz"
if SameSource(s, sampleSource()) {
t.Error("different sources are the same")
}
if !SameCandidate(sampleCandidate(), sampleCandidate()) {
t.Error("equal candidates differ")
}
c := sampleCandidate()
c.Pulls[0].Modules = []string{"other"}
if SameCandidate(c, sampleCandidate()) {
t.Error("different candidates are the same")
}
m := Module{FP: "src1:a", Repository: "a/b", Commit: "c"}
if !SameModule(m, m) || SameModule(m, Module{FP: "src1:b", Repository: "a/b", Commit: "c"}) {
t.Error("SameModule wrong")
}
f := sampleForges()[1]
g := sampleForges()[1]
if !SameForge(f, g) {
t.Error("equal forge messages differ")
}
g.Trees["modules/a"] = "other"
if SameForge(f, g) {
t.Error("different forge messages are the same")
}
}
func TestSourceBuckets(t *testing.T) {
if SourceBucket != "mesh-delivery-gate_source" || CandidatesBucket != "mesh-delivery-indexer_candidates" ||
ModulesBucket != "mesh-delivery-indexer_modules" || SourceMainKey != "main" ||
SourceHistory != 64 || CandidatesHistory != 10 || ModulesHistory != 20 {
t.Error("a bucket or history differs from the decided layout")
}
}
func TestForgeDirs(t *testing.T) {
pull := func(dirs []string, known bool) Forge {
return Forge{Kind: ForgePull, Repository: "novox/mesh-sdk", Number: 36, Head: "aaaa", Action: "updated", Dirs: dirs, DirsKnown: known}
}
round := func(f Forge) Forge {
t.Helper()
data, err := EncodeForge(f)
if err != nil {
t.Fatal(err)
}
got, err := DecodeForge(data)
if err != nil {
t.Fatal(err)
}
return got
}
// 1233/AC2: the directories come back equal and in order.
got := round(pull([]string{"modules/a", "modules/b"}, true))
if !reflect.DeepEqual(got.Dirs, []string{"modules/a", "modules/b"}) || !got.DirsKnown {
t.Errorf("dirs = %v known %v", got.Dirs, got.DirsKnown)
}
// 1233/AC3: a head changing no module directory is told apart from one whose directories could not be said.
none, unknown := round(pull(nil, true)), round(pull(nil, false))
if !none.DirsKnown || len(none.Dirs) != 0 {
t.Errorf("known and empty: dirs %v known %v", none.Dirs, none.DirsKnown)
}
if unknown.DirsKnown || len(unknown.Dirs) != 0 {
t.Errorf("not known: dirs %v known %v", unknown.Dirs, unknown.DirsKnown)
}
// 1233/AC4: a merged or a refused message carrying directories is refused.
m := sampleForges()[1]
m.Dirs, m.DirsKnown = []string{"modules/a"}, true
if _, err := EncodeForge(m); err == nil {
t.Error("merged message with dirs accepted")
}
r := sampleForges()[2]
r.Dirs, r.DirsKnown = []string{"modules/a"}, true
if _, err := EncodeForge(r); err == nil {
t.Error("refused message with dirs accepted")
}
r = sampleForges()[2]
r.DirsKnown = true
if _, err := EncodeForge(r); err == nil {
t.Error("refused message with dirs-known accepted")
}
// directories on a pull message that does not say it knows them are refused too.
if _, err := EncodeForge(pull([]string{"modules/a"}, false)); err == nil {
t.Error("dirs without dirs-known accepted")
}
}
+65
View File
@@ -0,0 +1,65 @@
package delivery
import (
"encoding/json"
"fmt"
"time"
)
// WatchmanModule is the watchman's module: its buckets are its own state, `<module>_<name>`, and the controller's
// broker grants it alone the bus's connect and disconnect events (novox/hq ADR 0431).
const WatchmanModule = "mesh-delivery-watchman"
// StuckBucket holds what stopped moving (novox/hq ADR 0433 §2): one key per key whose pulse expired, written by the
// watchman, read by the controller's self-check, which raises each as a condition until it says moved. No key of it
// is removed: a key that moves again is written as moved.
const StuckBucket = WatchmanModule + "_stuck"
// StuckKey is the key, in StuckBucket, of the key that stopped moving: `<bucket>.<key>`, formed as PulseKey forms a
// pulse's, so a node or part gone offline is `mesh-delivery-watchman_presence.<presence key>`.
func StuckKey(bucket, key string) (string, error) {
stuck, err := PulseKey(bucket, key)
if err != nil {
return "", fmt.Errorf("delivery: no stuck key: %w", err)
}
return stuck, nil
}
// ParseStuckKey is the inverse of StuckKey, splitting at the first dot.
func ParseStuckKey(stuckKey string) (bucket, key string, err error) {
bucket, key, err = ParsePulseKey(stuckKey)
if err != nil {
return "", "", fmt.Errorf("delivery: %q is not a stuck key: %w", stuckKey, err)
}
return bucket, key, nil
}
// Stuck is the value of a stuck key: the key that stopped moving, what it waited for in words, since when, and
// whether it has moved again since. The watchman writes it; the controller reads it.
type Stuck struct {
V int `json:"v"`
Bucket string `json:"bucket"`
Key string `json:"key"`
Waited string `json:"waited,omitempty"`
Since time.Time `json:"since"`
Moved bool `json:"moved,omitempty"`
}
// EncodeStuck is s as JSON at Version.
func EncodeStuck(s Stuck) ([]byte, error) {
s.V = Version
return json.Marshal(s)
}
// DecodeStuck reads a stuck value; it refuses an unknown version (ErrUnknownVersion).
func DecodeStuck(data []byte) (Stuck, error) {
var s Stuck
err := decode(data, &s)
return s, err
}
// SameStuck says whether two stuck values are the same, so a writer writes only when they differ.
func SameStuck(a, b Stuck) bool {
return a.V == b.V && a.Bucket == b.Bucket && a.Key == b.Key && a.Waited == b.Waited && a.Since.Equal(b.Since) &&
a.Moved == b.Moved
}
+88
View File
@@ -0,0 +1,88 @@
package delivery
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
)
// 1153/AC4: the watchman's module and its three buckets, each that module's state.
func TestWatchmanBuckets(t *testing.T) {
if WatchmanModule != "mesh-delivery-watchman" {
t.Fatalf("WatchmanModule = %q", WatchmanModule)
}
for name, bucket := range map[string]string{"presence": PresenceBucket, "pulse": PulseBucket, "stuck": StuckBucket} {
if bucket != WatchmanModule+"_"+name {
t.Errorf("the %s bucket is %q, not %s_%s", name, bucket, WatchmanModule, name)
}
}
}
// 1153/AC1: the stuck key is the stopped key's bucket and key, split at the first dot, refused as a pulse key is.
func TestStuckKey(t *testing.T) {
key, err := StuckKey(PresenceBucket, "ace")
if err != nil || key != "mesh-delivery-watchman_presence.ace" {
t.Fatalf("StuckKey = %q, %v", key, err)
}
key, err = StuckKey("mesh-controller_builds", "zsh.build.b-1")
if err != nil || key != "mesh-controller_builds.zsh.build.b-1" {
t.Fatalf("StuckKey = %q, %v", key, err)
}
bucket, rest, err := ParseStuckKey(key)
if err != nil || bucket != "mesh-controller_builds" || rest != "zsh.build.b-1" {
t.Fatalf("ParseStuckKey = %q %q %v", bucket, rest, err)
}
for _, c := range [][2]string{{"", "k"}, {"a.b", "k"}, {"b", ""}, {"b", "k x"}, {"b", ".k"}, {"b", "k."}, {"b", "k>"}} {
if _, err := StuckKey(c[0], c[1]); err == nil {
t.Errorf("StuckKey(%q, %q) accepted", c[0], c[1])
}
}
for _, bad := range []string{"", "nodot", ".k", "b."} {
if _, _, err := ParseStuckKey(bad); err == nil {
t.Errorf("ParseStuckKey(%q) accepted", bad)
}
}
}
// 1153/AC2: a stuck value round-trips with "v":1, and an unknown or missing version is refused loudly.
func TestStuckValue(t *testing.T) {
s := Stuck{Bucket: PresenceBucket, Key: "ace", Waited: "ace to come back online",
Since: time.Date(2026, 10, 18, 20, 0, 0, 0, time.UTC)}
raw, err := EncodeStuck(s)
if err != nil {
t.Fatal(err)
}
var probe map[string]any
if err := json.Unmarshal(raw, &probe); err != nil || probe["v"] != float64(1) {
t.Fatalf("encoded %s: v = %v (%v)", raw, probe["v"], err)
}
back, err := DecodeStuck(raw)
s.V = Version
if err != nil || !SameStuck(back, s) {
t.Fatalf("decoded %+v, %v; want %+v", back, err, s)
}
for _, bad := range []string{`{"v":2,"bucket":"b","key":"k"}`, `{"bucket":"b","key":"k"}`} {
_, err := DecodeStuck([]byte(bad))
if !errors.Is(err, ErrUnknownVersion) {
t.Errorf("DecodeStuck(%s) = %v, want ErrUnknownVersion", bad, err)
}
if strings.Contains(bad, `"v":2`) && (err == nil || !strings.Contains(err.Error(), "2") || !strings.Contains(err.Error(), "1")) {
t.Errorf("DecodeStuck(%s) = %v: does not name the version seen and the one read", bad, err)
}
}
}
// 1153/AC3: equal stuck values are the same; one moved is not.
func TestSameStuck(t *testing.T) {
a := Stuck{V: Version, Bucket: "b", Key: "k", Waited: "w", Since: time.Unix(100, 0).UTC()}
b := a
if !SameStuck(a, b) {
t.Fatal("equal values are not the same")
}
b.Moved = true
if SameStuck(a, b) {
t.Fatal("a moved value is the same as a stuck one")
}
}
+1
View File
@@ -0,0 +1 @@
{"id":"build-1792395447088936520","state":"built","repository":"novox/mesh-controller","ref":"main","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","manifest-digest":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","source-inputs":{"Module":"build-agent","Tree":"11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca","Bases":["sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc","sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c"],"Contexts":{"server":"narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"},"Toolchains":null,"Unpinned":""},"made":[{"name":"server","kind":"image","reference":"registry.example/build-agent@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"}],"at":"2026-10-19T09:38:00Z"}
+110
View File
@@ -0,0 +1,110 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"testing"
"time"
)
// emitRuntime drives a bundle over pipes as the runtime does, reading every line it says.
type emitRuntime struct {
t *testing.T
lines *bufio.Scanner
in io.Writer
}
func (r *emitRuntime) next() map[string]any {
done := make(chan map[string]any, 1)
go func() {
if r.lines.Scan() {
var m map[string]any
_ = json.Unmarshal(r.lines.Bytes(), &m)
done <- m
}
}()
select {
case m := <-done:
return m
case <-time.After(3 * time.Second):
r.t.Fatal("the bundle said nothing")
return nil
}
}
func serveEmit(t *testing.T, tools []Tool) *emitRuntime {
t.Helper()
t.Setenv(ServedAs, "chime")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() { _ = ServeOn("", tools, inR, outW) }()
t.Cleanup(func() { _ = inW.Close() })
return &emitRuntime{t: t, lines: bufio.NewScanner(outR), in: inW}
}
func (r *emitRuntime) call(id int, tool string) {
b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": id, "method": "tools/call",
"params": map[string]any{"name": tool, "arguments": map[string]any{}}})
_, _ = r.in.Write(append(b, '\n'))
}
func (r *emitRuntime) answer(id any) {
b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": id, "result": map[string]any{}})
_, _ = r.in.Write(append(b, '\n'))
}
// eventIDOf is a "mesh/publish" ask's x-event-id header.
func eventIDOf(asked map[string]any) string {
return asked["params"].(map[string]any)["headers"].(map[string]any)["x-event-id"].(string)
}
// 1187/AC1: given EmitOptions with a message id, when a bundle emits, then the event's x-event-id header is
// that id; and a random one when none is given, as on main.
func TestEmitSendsTheChosenMessageIDAsXEventID(t *testing.T) {
rt := serveEmit(t, []Tool{
{Name: "chosen", Run: func(map[string]any) (any, error) {
return nil, Emit("sent", map[string]any{}, EmitOptions{MessageID: "send-42"})
}},
{Name: "unchosen", Run: func(map[string]any) (any, error) {
return nil, Emit("sent", map[string]any{})
}},
})
rt.call(1, "chosen")
asked := rt.next()
if got := eventIDOf(asked); got != "send-42" {
t.Fatalf("x-event-id: %q, want the chosen id %q", got, "send-42")
}
rt.answer(asked["id"])
rt.next() // the tool's own answer
rt.call(2, "unchosen")
asked = rt.next()
if got := eventIDOf(asked); len(got) != 36 {
t.Fatalf("x-event-id with no message id chosen: %q, want a random UUID as on main", got)
}
rt.answer(asked["id"])
rt.next()
}
// 1187/AC2: given two emits with the same chosen id, when sent through the runtime's protocol stand-in,
// then both carry the same x-event-id, which the tool runner gives the bus as Nats-Msg-Id (mesh-tools
// node-tools/internal/bus/bus.go ~748-773, :770): a repeated publish is kept to one by the bus's dedup.
func TestEmitRepeatedWithTheSameChosenIDCarriesTheSameXEventIDTwice(t *testing.T) {
rt := serveEmit(t, []Tool{
{Name: "chosen", Run: func(map[string]any) (any, error) {
return nil, Emit("sent", map[string]any{}, EmitOptions{MessageID: "send-77"})
}},
})
var ids []string
for i := 1; i <= 2; i++ {
rt.call(i, "chosen")
asked := rt.next()
ids = append(ids, eventIDOf(asked))
rt.answer(asked["id"])
rt.next()
}
if ids[0] != "send-77" || ids[1] != "send-77" {
t.Fatalf("the two emits' x-event-id: %v, want %q both times", ids, "send-77")
}
}
+19 -1
View File
@@ -1,3 +1,21 @@
module git.novox.be/novox/mesh-sdk/go
go 1.22
go 1.26.0
require (
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+25
View File
@@ -0,0 +1,25 @@
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+191
View File
@@ -0,0 +1,191 @@
// Package health is the answer a module's health tool gives, and how the node-engine reads it (novox/hq ADR 0240
// rule 2, ADR 0283).
//
// A health tool answers one of three:
//
// - healthy: everything the module runs works;
// - unhealthy: something is wrong, said in Why;
// - waits: nothing is wrong but a part that cannot work until the operator gives something — one of the module's
// own secrets, or one of its settings. Each Wait names exactly one of them, the part, and what the operator
// gives, in words with no path, address or value.
//
// The answer keeps `healthy` beside `state`, so a node-engine older than ADR 0283 reads a wait as not healthy, as
// it did before. An answer without `state` is read from `healthy`.
//
// The module writes its answer with Healthy, Unhealthy or Waiting, and the node-engine reads it with Read. Read
// never trusts a shape it cannot check: a wait it cannot read makes the answer unhealthy, saying why.
package health
import (
"encoding/json"
"fmt"
"regexp"
"strings"
)
// The three states an answer says.
const (
StateHealthy = "healthy"
StateUnhealthy = "unhealthy"
StateWaits = "waits"
)
// Answer is a health tool's answer.
type Answer struct {
// State is healthy, unhealthy or waits.
State string `json:"state"`
// Healthy is State == healthy, kept for a reader older than State.
Healthy bool `json:"healthy"`
// Why is the answer in one or more plain sentences.
Why string `json:"why"`
// Waits is what the module waits for, when State is waits; and, beside a fault, what it also waits for.
Waits []Wait `json:"waits,omitempty"`
}
// Wait is one thing a module waits for the operator to give: exactly one of Secret and Setting.
type Wait struct {
// Part is what waits, in words: "the source games".
Part string `json:"part"`
// Secret is one of the module's own secrets, by name (a member of a secret family by its full name).
Secret string `json:"secret,omitempty"`
// Setting is one of the module's settings, by name.
Setting string `json:"setting,omitempty"`
// What is what the operator gives, in words: "the password of the source games".
What string `json:"what"`
}
// Healthy is a healthy answer.
func Healthy(why string) Answer {
return Answer{State: StateHealthy, Healthy: true, Why: why}
}
// Unhealthy is an unhealthy answer. Waits may be given beside the fault: they are said, and excuse nothing.
func Unhealthy(why string, waits ...Wait) Answer {
return Answer{State: StateUnhealthy, Why: why, Waits: waits}
}
// Waiting is an answer that says nothing is wrong but what the operator has yet to give. With no wait it is
// unhealthy: a module that waits names what for.
func Waiting(why string, waits ...Wait) Answer {
if len(waits) == 0 {
return Unhealthy(orSaid(why, "it says it waits") + " (and names nothing it waits for)")
}
for _, w := range waits {
if err := w.Check(); err != nil {
return Unhealthy(orSaid(why, "it says it waits") + " (" + err.Error() + ")")
}
}
return Answer{State: StateWaits, Why: why, Waits: waits}
}
// name is a secret's or a setting's name: what a manifest may declare.
var name = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// looksLikeAddress finds what the operator's words never carry: a URL, an IPv4 or IPv6 address, a host and port, a
// path (absolute, from a home or relative, on its own or after a bracket or a quote), a UNC path or a drive path.
// It cannot find a value typed in plain words ("the password hunter2"): a module never puts one there, and the
// words are its own, not the operator's.
var looksLikeAddress = regexp.MustCompile(strings.Join([]string{
`://`,
`\b[0-9]{1,3}(\.[0-9]{1,3}){3}\b`,
`[0-9A-Fa-f]{0,4}::[0-9A-Fa-f]{0,4}`,
`\b[0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4}){3,}\b`,
`\b[A-Za-z0-9.-]+:[0-9]{1,5}\b`,
`(^|[\s(\[{"'\x60])[~.]{0,2}/`,
`\\\\`,
`\b[A-Za-z]:\\`,
}, "|"))
// Check says why a wait cannot be read, or nil.
func (w Wait) Check() error {
switch {
case w.Secret != "" && w.Setting != "":
return fmt.Errorf("a wait names both the secret %q and the setting %q; it names one", w.Secret, w.Setting)
case w.Secret == "" && w.Setting == "":
return fmt.Errorf("a wait names neither a secret nor a setting")
case w.Secret != "" && !name.MatchString(w.Secret):
return fmt.Errorf("a wait names the secret %q, which is not a name", w.Secret)
case w.Setting != "" && !name.MatchString(w.Setting):
return fmt.Errorf("a wait names the setting %q, which is not a name", w.Setting)
case strings.TrimSpace(w.What) == "":
return fmt.Errorf("a wait for %s does not say what the operator gives", w.Names())
case strings.TrimSpace(w.Part) == "":
return fmt.Errorf("a wait for %s does not say which part waits", w.Names())
}
for _, words := range []string{w.What, w.Part} {
if strings.ContainsAny(words, "\n\r") {
return fmt.Errorf("a wait for %s says more than one line", w.Names())
}
if looksLikeAddress.MatchString(words) {
return fmt.Errorf("a wait for %s says a path or an address, which the operator's words never carry", w.Names())
}
}
return nil
}
// Names is what the wait names, as "the secret x" or "the setting y".
func (w Wait) Names() string {
if w.Secret != "" {
return "the secret " + w.Secret
}
if w.Setting != "" {
return "the setting " + w.Setting
}
return "nothing"
}
// Read reads a health tool's result. An answer it cannot read at all is an error; an answer whose wait it cannot
// read is unhealthy, saying why, so a malformed wait never excuses anything.
func Read(result []byte) (Answer, error) {
var a Answer
if err := json.Unmarshal(result, &a); err != nil {
return Answer{}, fmt.Errorf("it answered something other than healthy or not: %w", err)
}
switch a.State {
case "":
if a.Healthy {
a.State = StateHealthy
} else {
a.State = StateUnhealthy
}
case StateHealthy:
if !a.Healthy {
return Unhealthy(fmt.Sprintf("it answered the state healthy and healthy false, which contradict each other (%s)",
orSaid(a.Why, "and said nothing more"))), nil
}
// A healthy module waits for nothing: a wait beside healthy is a contradiction, and is dropped.
a.Waits = nil
case StateUnhealthy:
a.Healthy = false
case StateWaits:
a.Healthy = false
if len(a.Waits) == 0 {
return Unhealthy(orSaid(a.Why, "it says it waits") + " (and names nothing it waits for)"), nil
}
for _, w := range a.Waits {
if err := w.Check(); err != nil {
return Unhealthy(orSaid(a.Why, "it says it waits") + " (" + err.Error() + ")"), nil
}
}
default:
return Unhealthy(fmt.Sprintf("it answered the state %q, which is none of healthy, unhealthy or waits", a.State)), nil
}
if a.State != StateWaits {
// Waits beside a fault are said and excuse nothing; one that cannot be read is dropped, not believed.
var kept []Wait
for _, w := range a.Waits {
if w.Check() == nil {
kept = append(kept, w)
}
}
a.Waits = kept
}
return a, nil
}
func orSaid(s, otherwise string) string {
if strings.TrimSpace(s) == "" {
return otherwise
}
return s
}
+150
View File
@@ -0,0 +1,150 @@
package health
import (
"encoding/json"
"strings"
"testing"
)
func roundTrip(t *testing.T, a Answer) Answer {
t.Helper()
b, err := json.Marshal(a)
if err != nil {
t.Fatal(err)
}
got, err := Read(b)
if err != nil {
t.Fatal(err)
}
return got
}
var password = Wait{Part: "the source games", Secret: "smb-password-games", What: "the password of the source games"}
func TestEachConstructorSaysItsStateAndKeepsHealthyForAnOlderReader(t *testing.T) {
for _, c := range []struct {
a Answer
state string
healthy bool
}{
{Healthy("all armed"), StateHealthy, true},
{Unhealthy("the share is unreachable"), StateUnhealthy, false},
{Waiting("the source games waits for its password", password), StateWaits, false},
} {
b, _ := json.Marshal(c.a)
var old struct {
Healthy bool `json:"healthy"`
Why string `json:"why"`
}
if err := json.Unmarshal(b, &old); err != nil || old.Healthy != c.healthy || old.Why == "" {
t.Fatalf("an older reader of %s reads %+v (%v)", b, old, err)
}
if got := roundTrip(t, c.a); got.State != c.state || got.Healthy != c.healthy {
t.Fatalf("%s read as %+v", b, got)
}
}
}
func TestAWaitIsReadWithWhatItNames(t *testing.T) {
got := roundTrip(t, Waiting("waits", password, Wait{Part: "the source games", Setting: "smb-users", What: "the username of the source games"}))
if got.State != StateWaits || len(got.Waits) != 2 || got.Waits[0].Secret != "smb-password-games" || got.Waits[1].Setting != "smb-users" {
t.Fatalf("read as %+v", got)
}
}
func TestAnAnswerWithoutStateIsReadFromHealthy(t *testing.T) {
for raw, want := range map[string]string{`{"healthy":true,"why":"ok"}`: StateHealthy, `{"healthy":false,"why":"down"}`: StateUnhealthy} {
got, err := Read([]byte(raw))
if err != nil || got.State != want {
t.Fatalf("%s read as %+v, %v", raw, got, err)
}
}
}
func TestAWaitThatCannotBeReadIsUnhealthyAndSaysWhy(t *testing.T) {
for name, c := range map[string]struct {
w []Wait
says string
}{
"no wait": {nil, "names nothing"},
"neither": {[]Wait{{Part: "p", What: "w"}}, "neither a secret nor a setting"},
"both": {[]Wait{{Part: "p", Secret: "a", Setting: "b", What: "w"}}, "both"},
"not a name": {[]Wait{{Part: "p", Secret: "Smb Password", What: "w"}}, "not a name"},
"no what": {[]Wait{{Part: "p", Secret: "a"}}, "does not say what"},
"no part": {[]Wait{{Secret: "a", What: "w"}}, "which part"},
"two lines": {[]Wait{{Part: "p", Secret: "a", What: "the password\nhunter2"}}, "more than one line"},
"an address": {[]Wait{{Part: "p", Secret: "a", What: "the password of 192.0.2.76"}}, "path or an address"},
"a url": {[]Wait{{Part: "p", Secret: "a", What: "the password of smb://host/share"}}, "path or an address"},
"a path": {[]Wait{{Part: "the source at /mnt/games", Secret: "a", What: "the password"}}, "path or an address"},
} {
b, _ := json.Marshal(Answer{State: StateWaits, Why: "it waits", Waits: c.w})
got, err := Read(b)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if got.State != StateUnhealthy || got.Healthy || !strings.Contains(got.Why, c.says) {
t.Fatalf("%s: read as %+v, want unhealthy saying %q", name, got, c.says)
}
}
}
func TestAnUnknownStateIsUnhealthy(t *testing.T) {
got, err := Read([]byte(`{"state":"fine","healthy":true}`))
if err != nil || got.State != StateUnhealthy || got.Healthy {
t.Fatalf("read as %+v, %v", got, err)
}
}
func TestWaitsBesideAFaultAreSaidAndExcuseNothing(t *testing.T) {
got := roundTrip(t, Unhealthy("the share is unreachable", password, Wait{Part: "p", What: "w"}))
if got.State != StateUnhealthy || len(got.Waits) != 1 {
t.Fatalf("read as %+v: the readable wait is kept, the other dropped, the state stays unhealthy", got)
}
if h := roundTrip(t, Answer{State: StateHealthy, Healthy: true, Why: "ok", Waits: []Wait{password}}); len(h.Waits) != 0 {
t.Fatalf("a healthy answer kept its waits: %+v", h)
}
}
func TestWaitingWithNothingIsUnhealthy(t *testing.T) {
if a := Waiting("it waits"); a.State != StateUnhealthy {
t.Fatalf("Waiting with no wait is %+v", a)
}
}
func TestNotAnAnswerIsAnError(t *testing.T) {
if _, err := Read([]byte(`"yes"`)); err == nil {
t.Fatal("a string read as an answer")
}
}
func TestEveryFormOfPathOrAddressIsRefused(t *testing.T) {
for _, words := range []string{
"the source (/mnt/games)", `the source "/mnt/games"`, "the folder ~/games", "the folder ./games",
"2001:db8::1", "fe80::1%eth0", "2001:0db8:0000:0000:0000:0000:0000:0001", "nas.lan:445", "192.0.2.7:445",
`\\nas\games`, `C:\Users\x`, "smb://nas/games", "198.51.100.4",
} {
w := Wait{Part: "the source games", Secret: "smb-password-games", What: "the password of " + words}
if err := w.Check(); err == nil {
t.Errorf("%q passed as words for the operator", words)
}
}
for _, words := range []string{"the password of the source games", "the username of the source games-2", "the API key, issued by the vendor"} {
w := Wait{Part: "the source games", Secret: "smb-password-games", What: words}
if err := w.Check(); err != nil {
t.Errorf("%q refused: %v", words, err)
}
}
}
func TestHealthyStateWithHealthyFalseIsUnhealthy(t *testing.T) {
got, err := Read([]byte(`{"state":"healthy","healthy":false,"why":"x"}`))
if err != nil || got.State != StateUnhealthy || !strings.Contains(got.Why, "contradict") {
t.Fatalf("read as %+v, %v", got, err)
}
}
func TestWaitingChecksItsWaits(t *testing.T) {
if a := Waiting("it waits", Wait{Part: "p", What: "w"}); a.State != StateUnhealthy || !strings.Contains(a.Why, "neither") {
t.Fatalf("Waiting with a malformed wait is %+v", a)
}
}
+74
View File
@@ -0,0 +1,74 @@
// Package testbus is the bus an integration test of the SDK's Go binding runs against: a server of
// its own, of the release go.mod pins, started for the one test and gone after it.
//
// **Why one per test.** A test that reaches a shared bus asserts, reads and removes objects by fixed
// name — a stream, a consumer — and two tests at once would step on each other's: one deleting what
// the other is reading, one leaving a stream behind for the next to trip on. A server per test makes
// every test start from nothing and leave nothing, so the suite is the same run in any order, at any
// parallelism, under the race detector, on a workstation or on the build seat.
//
// **No catalogue pin check.** mesh-controller's testbus also checks the server's release against the
// catalogue's bus image and the mesh's facts snapshot (novox/hq ADR 0227 rule 9) — the SDK has no
// catalogue of its own to check against, so that test is left out here. Only the version this
// module's go.mod pins is what these tests run.
//
// A person may still point a run at a bus of their own — another release, a lab's — with
// MESH_TEST_NATS_EXTERNAL=1 and MESH_TEST_NATS; then the run is theirs to serialise.
package testbus
import (
"os"
"sync"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
)
// Version is the release of the server the tests run.
const Version = server.VERSION
// URL is the bus of this test alone: started at its first ask, the same one at every ask after — a
// test that dials twice, or hands the address to the code it tests, reaches one bus — and shut down
// when the test ends.
func URL(t testing.TB) string {
t.Helper()
if os.Getenv("MESH_TEST_NATS_EXTERNAL") == "1" {
if url := os.Getenv("MESH_TEST_NATS"); url != "" {
return url
}
t.Fatal("MESH_TEST_NATS_EXTERNAL=1 and MESH_TEST_NATS names no bus")
}
if url, ok := buses.Load(t); ok {
return url.(string)
}
url := Start(t).ClientURL()
buses.Store(t, url)
t.Cleanup(func() { buses.Delete(t) })
return url
}
// buses are the running tests' buses, by test.
var buses sync.Map
// Start is a server with JetStream, for this test alone: on loopback, on a port the system chooses, its
// store in the test's own directory.
func Start(t testing.TB) *server.Server {
t.Helper()
opts := &server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(),
NoLog: true, NoSigs: true}
s, err := server.NewServer(opts)
if err != nil {
t.Fatalf("a bus for this test could not be made: %v", err)
}
go s.Start()
if !s.ReadyForConnections(30 * time.Second) {
s.Shutdown()
t.Fatal("a bus for this test did not come up within 30s")
}
t.Cleanup(func() {
s.Shutdown()
s.WaitForShutdown()
})
return s
}
@@ -0,0 +1,87 @@
//go:build integration
package testbus
import (
"context"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// 1175/AC3, 1171/AC1: a JetStream bus of this test alone answers a publish and a key-value create.
func TestBusAnswersPublishAndKeyValueCreate(t *testing.T) {
conn, err := nats.Connect(URL(t))
if err != nil {
t.Fatal(err)
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
ctx := context.Background()
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.>"}}); err != nil {
t.Fatalf("the bus did not answer a stream create: %v", err)
}
ack, err := js.Publish(ctx, "mesh.test", []byte("hello"))
if err != nil {
t.Fatalf("the bus did not answer a publish: %v", err)
}
if ack.Stream != "EVENTS" {
t.Errorf("published into %q, want EVENTS", ack.Stream)
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "TESTKV"})
if err != nil {
t.Fatalf("the bus did not answer a key-value create: %v", err)
}
if _, err := kv.Put(ctx, "k", []byte("v")); err != nil {
t.Fatal(err)
}
entry, err := kv.Get(ctx, "k")
if err != nil {
t.Fatal(err)
}
if string(entry.Value()) != "v" {
t.Errorf("read back %q, want %q", entry.Value(), "v")
}
}
// 1175/AC3, 1171/AC1: the bus is gone when the test that started it ends.
func TestBusGoneWhenItsTestEnds(t *testing.T) {
var url string
t.Run("one", func(t *testing.T) {
url = URL(t)
conn, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
conn.Close()
})
if _, err := nats.Connect(url, nats.Timeout(time.Second)); err == nil {
t.Fatal("the subtest's bus still answers now that the subtest has ended")
}
}
// 1175/AC3: two tests never share a bus.
func TestEveryTestHasABusOfItsOwn(t *testing.T) {
seen := map[string]bool{}
for i := 0; i < 2; i++ {
t.Run("one", func(t *testing.T) {
url := URL(t)
if seen[url] {
t.Fatalf("two tests were handed %s", url)
}
seen[url] = true
})
}
// Asked twice, one test reaches one bus.
if a, b := URL(t), URL(t); a != b {
t.Errorf("one test was handed two buses: %s and %s", a, b)
}
}
+36
View File
@@ -0,0 +1,36 @@
package stdio
import (
"os"
"os/exec"
"testing"
)
// 1171/AC4: before Serve, a bundle's bus functions still refuse exactly as on main (stdio.go:391-398,
// unchanged for bundles). Run in a fresh process: once any other test in this package has called Serve,
// current stays set for the rest of the test binary, so "before Serve" can only be checked honestly in a
// process of its own (the self-exec pattern Go's own testing package uses for os.Exit).
func TestBusFunctionsRefuseBeforeServeInAFreshProcess(t *testing.T) {
const marker = "MESH_SDK_TEST_FRESH_PROCESS"
if os.Getenv(marker) == "1" {
if err := Emit("x", map[string]any{}); err == nil ||
err.Error() != "emit before Serve: a bundle emits through the runtime that launched it" {
t.Fatalf("Emit before Serve: %v", err)
}
if _, err := Ask("m.t", nil); err == nil ||
err.Error() != "before Serve: a bundle reaches the bus through the runtime that launched it" {
t.Fatalf("Ask before Serve: %v", err)
}
if err := Subscribe("x", func(Envelope) error { return nil }); err == nil ||
err.Error() != "before Serve: a bundle reaches the bus through the runtime that launched it" {
t.Fatalf("Subscribe before Serve: %v", err)
}
return
}
cmd := exec.Command(os.Args[0], "-test.run=^TestBusFunctionsRefuseBeforeServeInAFreshProcess$", "-test.v")
cmd.Env = append(os.Environ(), marker+"=1")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("a fresh process did not refuse every bus function before Serve: %v\n%s", err, out)
}
}
+123
View File
@@ -0,0 +1,123 @@
package stdio
import (
"encoding/json"
"fmt"
"time"
)
// Log — a module's record of operations on the bus, kept as long as the log is (novox/hq ADR 0297). Not the
// current value: that is state, and a log's state is its fold. Each entry is one operation on one key, its
// sequence in the log is its identity, and the order of the log is the order of the operations. A module
// declares each log it owns in its manifest (`logs`) and names it here as it named it there; a log is its
// owner's alone, and the runtime refuses, with the reason, one it did not issue.
// LogEntry is one operation on one key, as the log holds it.
type LogEntry struct {
Sequence uint64 `json:"sequence"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
At time.Time `json:"at"`
}
// LogPage is one page of a read: its entries, in order, the sequence to read on from (0 when nothing
// follows) and the log's last sequence when it was read.
type LogPage struct {
Entries []LogEntry `json:"entries"`
Next uint64 `json:"next"`
Last uint64 `json:"last"`
}
// LogInfo is how much a log holds: its entries and bytes, its cap in bytes, and its first and last sequence.
type LogInfo struct {
Entries uint64 `json:"entries"`
Bytes uint64 `json:"bytes"`
MaxBytes uint64 `json:"max_bytes"`
First uint64 `json:"first"`
Last uint64 `json:"last"`
}
// KeptLog is one log this module owns.
type KeptLog struct{ name string }
// Log is a log this module owns, by its local name.
func Log(name string) KeptLog { return KeptLog{name: name} }
const logTimeout = 30 * time.Second
func (l KeptLog) ask(verb string, params map[string]any) (json.RawMessage, error) {
c, err := live()
if err != nil {
return nil, err
}
params["log"] = l.name
return c.ask("mesh/log."+verb, params, logTimeout)
}
func (l KeptLog) append(params map[string]any) (uint64, error) {
raw, err := l.ask("append", params)
if err != nil {
return 0, err
}
var appended struct {
Sequence uint64 `json:"sequence"`
}
return appended.Sequence, json.Unmarshal(raw, &appended)
}
// Append writes an entry for key, whatever came before it, and returns its sequence. A key is one or more
// names of letters, digits, `-` and `_`, joined by `.`; a value is JSON of at most 256 KiB. A full log
// refuses the entry: it never drops an older one to make room.
func (l KeptLog) Append(key string, value any) (uint64, error) {
return l.append(map[string]any{"key": key, "value": value})
}
// AppendAt writes an entry for key only if the key's last entry is still at expect — the sequence the caller
// read, 0 for a key with no entry yet — and returns its sequence. When another entry for the key came in
// between, nothing is written and the error is ErrChanged, as for a lost State update: read again and
// decide again.
func (l KeptLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
return l.append(map[string]any{"key": key, "value": value, "expect": expect})
}
// Read is the entries from the sequence from on, oldest first, at most limit of them (0 is the runtime's
// page, 100; it gives at most 1000). keys narrows it: a key, with `*` for one name and `>` for the rest;
// empty is every key. Read on from the page's Next until it is 0.
func (l KeptLog) Read(from uint64, keys string, limit int) (*LogPage, error) {
if limit < 0 {
return nil, fmt.Errorf("a read of %s asks for at least no entries: limit %d", l.name, limit)
}
params := map[string]any{"from": from}
if keys != "" {
params["keys"] = keys
}
if limit > 0 {
params["limit"] = limit
}
raw, err := l.ask("read", params)
if err != nil {
return nil, err
}
var p LogPage
return &p, json.Unmarshal(raw, &p)
}
// Last is a key's last entry, or nil when the key has none.
func (l KeptLog) Last(key string) (*LogEntry, error) {
raw, err := l.ask("last", map[string]any{"key": key})
if err != nil || len(raw) == 0 || string(raw) == "null" {
return nil, err
}
var e LogEntry
return &e, json.Unmarshal(raw, &e)
}
// Info is how much the log holds and how much it may.
func (l KeptLog) Info() (*LogInfo, error) {
raw, err := l.ask("info", map[string]any{})
if err != nil {
return nil, err
}
var i LogInfo
return &i, json.Unmarshal(raw, &i)
}
+354
View File
@@ -0,0 +1,354 @@
package stdio
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"sync"
"testing"
"time"
)
// logRuntime is the runtime's side of mesh/log.append, read, last and info as the bus does them (novox/hq
// ADR 0297): one sequence per entry across the log, an append with expect refused unless the key's last
// entry is still at it (0 for a key with none), that refusal answered with CodeChanged as a lost state
// update is, and a read paged by its limit with the sequence to read on from. asked keeps every ask's
// method and params, as the bundle sent them.
type logRuntime struct {
mu sync.Mutex
entries []LogEntry
asked []message
}
func (r *logRuntime) last(key string) uint64 {
for i := len(r.entries) - 1; i >= 0; i-- {
if r.entries[i].Key == key {
return r.entries[i].Sequence
}
}
return 0
}
// matches is a key pattern as the runtime reads it: `*` one name, `>` the rest.
func matches(pattern, key string) bool {
if pattern == "" {
return true
}
p, k := strings.Split(pattern, "."), strings.Split(key, ".")
for i, t := range p {
if t == ">" {
return len(k) > i
}
if i >= len(k) || (t != "*" && t != k[i]) {
return false
}
}
return len(p) == len(k)
}
func (r *logRuntime) answer(m message) map[string]any {
var p struct {
Log string `json:"log"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Expect *uint64 `json:"expect"`
From uint64 `json:"from"`
Keys string `json:"keys"`
Limit int `json:"limit"`
}
_ = json.Unmarshal(m.Params, &p)
r.mu.Lock()
defer r.mu.Unlock()
r.asked = append(r.asked, m)
if p.Log != "changes" {
return map[string]any{"error": map[string]any{"code": -32000, "message": "issues was issued no log " + p.Log}}
}
switch strings.TrimPrefix(m.Method, "mesh/log.") {
case "append":
if p.Expect != nil && r.last(p.Key) != *p.Expect {
return map[string]any{"error": map[string]any{"code": CodeChanged, "message": p.Key + ": the key was written by another writer first"}}
}
seq := uint64(len(r.entries)) + 1
r.entries = append(r.entries, LogEntry{Sequence: seq, Key: p.Key, Value: p.Value,
At: time.Date(2026, 10, 11, 18, 0, int(seq), 0, time.UTC)})
return map[string]any{"result": map[string]any{"sequence": seq}}
case "read":
limit := p.Limit
if limit == 0 {
limit = 100
}
page := map[string]any{"entries": []LogEntry{}, "next": 0, "last": len(r.entries)}
var got []LogEntry
for _, e := range r.entries {
if e.Sequence < p.From || !matches(p.Keys, e.Key) {
continue
}
if len(got) == limit {
page["next"] = e.Sequence
break
}
got = append(got, e)
}
if got != nil {
page["entries"] = got
}
return map[string]any{"result": page}
case "last":
seq := r.last(p.Key)
if seq == 0 {
return map[string]any{"result": nil}
}
return map[string]any{"result": r.entries[seq-1]}
case "info":
var bytes int
for _, e := range r.entries {
bytes += len(e.Value)
}
info := map[string]any{"entries": len(r.entries), "bytes": bytes, "max_bytes": 1 << 30, "first": 0, "last": len(r.entries)}
if len(r.entries) > 0 {
info["first"] = 1
}
return map[string]any{"result": info}
}
return map[string]any{"error": map[string]any{"code": -32601, "message": "no " + m.Method}}
}
// logging serves a bundle whose one tool, "log", runs run against r, answering each mesh/log ask beside
// the loop as the runtime does. It returns what the tool answered.
func logging(t *testing.T, r *logRuntime, run func() (any, error)) map[string]any {
t.Helper()
t.Setenv(ServedAs, "issues")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() {
_ = ServeOn("", []Tool{{Name: "log", Run: func(map[string]any) (any, error) { return run() }}}, inR, outW)
}()
var wmu sync.Mutex
send := func(m map[string]any) {
m["jsonrpc"] = "2.0"
b, _ := json.Marshal(m)
wmu.Lock()
defer wmu.Unlock()
_, _ = inW.Write(append(b, '\n'))
}
done := make(chan map[string]any, 1)
go func() {
lines := bufio.NewScanner(outR)
lines.Buffer(make([]byte, 1<<20), 1<<24)
for lines.Scan() {
var m message
if json.Unmarshal(lines.Bytes(), &m) != nil {
continue
}
if strings.HasPrefix(m.Method, "mesh/log.") {
go func(m message) {
a := r.answer(m)
a["id"] = m.ID
send(a)
}(m)
continue
}
var answer map[string]any
_ = json.Unmarshal(lines.Bytes(), &answer)
if answer["id"] == float64(1) {
done <- answer
return
}
}
}()
send(map[string]any{"id": 1, "method": "tools/call", "params": map[string]any{"name": "log"}})
t.Cleanup(func() { _ = inW.Close() })
select {
case a := <-done:
return a
case <-time.After(10 * time.Second):
t.Fatal("the bundle did not answer")
return nil
}
}
// params is what the bundle sent with its n-th ask, and the ask's method.
func (r *logRuntime) params(t *testing.T, n int) (string, map[string]any) {
t.Helper()
r.mu.Lock()
defer r.mu.Unlock()
if n >= len(r.asked) {
t.Fatalf("the bundle asked %d times, not %d", len(r.asked), n+1)
}
var p map[string]any
_ = json.Unmarshal(r.asked[n].Params, &p)
return r.asked[n].Method, p
}
// A bundle appends to, reads, and measures its log through the runtime (novox/hq ADR 0297): each verb is
// asked as mesh/log.<verb> naming the log, its answer is handed back whole, a key with no entry is nil,
// and a read pages on from Next until it is 0.
func TestABundleKeepsALogThroughTheRuntime(t *testing.T) {
r := &logRuntime{}
var (
first, second, third uint64
pages []*LogPage
last, none *LogEntry
info *LogInfo
narrowed *LogPage
)
got := logging(t, r, func() (any, error) {
changes := Log("changes")
var err error
if first, err = changes.Append("401", map[string]any{"op": "opened", "by": "fix@g14"}); err != nil {
return nil, err
}
if second, err = changes.Append("402", map[string]any{"op": "opened"}); err != nil {
return nil, err
}
if third, err = changes.AppendAt("401", map[string]any{"op": "moved", "to": "fixing"}, first); err != nil {
return nil, err
}
for _, k := range []string{"403", "404", "405"} {
if _, err = changes.Append(k, map[string]any{"op": "opened"}); err != nil {
return nil, err
}
}
for from := uint64(1); ; {
p, err := changes.Read(from, "", 2)
if err != nil {
return nil, err
}
pages = append(pages, p)
if p.Next == 0 {
break
}
from = p.Next
}
if narrowed, err = changes.Read(0, "401", 0); err != nil {
return nil, err
}
if last, err = changes.Last("401"); err != nil {
return nil, err
}
if none, err = changes.Last("999"); err != nil {
return nil, err
}
info, err = changes.Info()
return "done", err
})
if out, isError := text(t, got); isError || out != `"done"` {
t.Fatalf("the tool answered %s", out)
}
if first != 1 || second != 2 || third != 3 {
t.Fatalf("appended at %d, %d, %d", first, second, third)
}
// Asked as the contract names them: the verb, the log, and only what was given.
method, p := r.params(t, 0)
if method != "mesh/log.append" || p["log"] != "changes" || p["key"] != "401" || p["value"].(map[string]any)["by"] != "fix@g14" {
t.Fatalf("an append asked %s %v", method, p)
}
if _, has := p["expect"]; has {
t.Fatalf("an Append without expect named one: %v", p)
}
if method, p = r.params(t, 2); method != "mesh/log.append" || p["expect"] != float64(1) {
t.Fatalf("an AppendAt asked %s %v", method, p)
}
if method, p = r.params(t, 6); method != "mesh/log.read" || p["from"] != float64(1) || p["limit"] != float64(2) || p["keys"] != nil {
t.Fatalf("a read asked %s %v", method, p)
}
// Paged two at a time through six entries: three pages, every entry once, in order.
if len(pages) != 3 {
t.Fatalf("read %d pages, not 3", len(pages))
}
var seqs []uint64
for _, pg := range pages {
if pg.Last != 6 {
t.Fatalf("a page said the last sequence is %d", pg.Last)
}
for _, e := range pg.Entries {
seqs = append(seqs, e.Sequence)
}
}
if fmt.Sprint(seqs) != "[1 2 3 4 5 6]" || pages[0].Next != 3 || pages[2].Next != 0 {
t.Fatalf("paged %v, next %d then %d", seqs, pages[0].Next, pages[2].Next)
}
if e := pages[1].Entries[0]; e.Key != "401" || string(e.Value) != `{"op":"moved","to":"fixing"}` ||
!e.At.Equal(time.Date(2026, 10, 11, 18, 0, 3, 0, time.UTC)) {
t.Fatalf("an entry read as %+v", e)
}
// Narrowed to a key, with the runtime's page: no limit sent.
method, p = r.params(t, 9)
if method != "mesh/log.read" || p["keys"] != "401" || p["limit"] != nil {
t.Fatalf("a narrowed read asked %s %v", method, p)
}
if len(narrowed.Entries) != 2 || narrowed.Entries[0].Sequence != 1 || narrowed.Entries[1].Sequence != 3 || narrowed.Next != 0 {
t.Fatalf("narrowed to 401: %+v", narrowed)
}
if last == nil || last.Sequence != 3 || last.Key != "401" {
t.Fatalf("the last of 401 is %+v", last)
}
if method, p = r.params(t, 10); method != "mesh/log.last" || p["key"] != "401" {
t.Fatalf("a last asked %s %v", method, p)
}
if none != nil {
t.Fatalf("a key with no entry has a last: %+v", none)
}
if info == nil || info.Entries != 6 || info.MaxBytes != 1<<30 || info.First != 1 || info.Last != 6 || info.Bytes == 0 {
t.Fatalf("the log's info is %+v", info)
}
if method, p = r.params(t, 12); method != "mesh/log.info" || p["log"] != "changes" || len(p) != 1 {
t.Fatalf("an info asked %s %v", method, p)
}
}
// An AppendAt on a stale expect is ErrChanged and writes nothing: another entry for the key came in
// between. Expect 0 is a key with no entry yet, and is ErrChanged once the key has one. A log not issued
// is refused with the runtime's reason, and is not ErrChanged.
func TestAnAppendAtAStaleSequenceIsErrChangedAndWritesNothing(t *testing.T) {
r := &logRuntime{}
var errs []error
got := logging(t, r, func() (any, error) {
changes := Log("changes")
read, err := changes.AppendAt("401", map[string]any{"op": "opened"}, 0)
if err != nil {
return nil, err
}
// Another writer is first.
if _, err := changes.Append("401", map[string]any{"op": "noted"}); err != nil {
return nil, err
}
_, stale := changes.AppendAt("401", map[string]any{"op": "moved"}, read)
_, again := changes.AppendAt("401", map[string]any{"op": "opened"}, 0)
_, other := Log("elsewhere").Append("401", map[string]any{})
errs = []error{stale, again, other}
return "done", nil
})
if out, isError := text(t, got); isError || out != `"done"` {
t.Fatalf("the tool answered %s", out)
}
if !errors.Is(errs[0], ErrChanged) || !errors.Is(errs[1], ErrChanged) {
t.Fatalf("a stale expect answered %v and %v, not ErrChanged", errs[0], errs[1])
}
if errs[2] == nil || errors.Is(errs[2], ErrChanged) || !strings.Contains(errs[2].Error(), "no log elsewhere") {
t.Fatalf("a log not issued answered %v", errs[2])
}
if len(r.entries) != 2 {
t.Fatalf("the log holds %d entries, not 2: a refused append wrote", len(r.entries))
}
}
// A read asking for fewer than no entries is refused before the runtime is asked.
func TestAReadOfANegativeLimitIsRefused(t *testing.T) {
r := &logRuntime{}
var err error
logging(t, r, func() (any, error) {
_, err = Log("changes").Read(0, "", -1)
return "done", nil
})
if err == nil || len(r.asked) != 0 {
t.Fatalf("a negative limit answered %v after %d asks", err, len(r.asked))
}
}
+115 -4
View File
@@ -5,16 +5,127 @@
# seat in the mesh's Go toolchain, its own container, because the SDK's merge-check.sh declares it
# (`# mesh-check-also: go go/merge-check.sh`) — and by hand, from anywhere: `sh go/merge-check.sh`.
#
# Formatted, vetted, and its whole suite run: the stdio loop and the local protocol mesh-cli speaks to the
# node-engine. Standard library only, so nothing is fetched.
# Formatted, vetted (with and without the build tag `integration`), and run at two levels (novox/hq ADR 0391):
#
# 1. the unit level: `go test ./...` untagged, no bus or other process, timed and held to a bound
# (unit_bound below); over it the check fails and names the slowest tests (issue 895);
# 2. the integration level: the whole suite with `-tags integration`, each test against a bus of its
# own (internal/testbus, which issue 1171 brings with the first test that needs a bus) so nothing is
# serialised. With no tagged test yet, the level runs the unit tests again and says their count.
#
# **A test that starts a bus or another process carries `//go:build integration`** as the first line of
# its file; a file that held both kinds is two files (`<name>_integration_test.go` beside `<name>_test.go`).
# Each level says its test count and its time.
#
# No dependency is committed (no vendor directory: .gitignore; the operator, 2026-10-19; issue 1304). Every dependency
# is resolved by the version and checksum go.sum names (novox/hq ADR 0405 §5), from its public source: this binding
# requires nothing of the mesh's own forge. `go mod verify` fails by module name when a downloaded dependency does not
# match go.sum, and -mod=readonly fails a build whose go.sum lacks an entry instead of writing it. gofmt reads only
# the binding's own files.
#
# Exit 0 is a pass; anything else fails `mesh/repo-check` with the last line printed.
set -eu
cd "$(dirname "$0")"
# The toolchain the check runs in sets -mod=vendor; there is no vendor directory, so it is replaced.
GOFLAGS="$(printf '%s' "${GOFLAGS:-}" | sed 's/-mod=[a-z]*//g') -mod=readonly"
export GOFLAGS
export GOPRIVATE=git.novox.be
unformatted=$(gofmt -l .)
# **The unit level's bound, in seconds** (issue 895): the slowest package's own test time of the untagged run,
# the compile not counted. A run over it fails the check and names the slowest tests. The SDK's unit suite is
# small (its slowest package measured well under a second on a quiet machine), so 2 s leaves room for a busier
# build node without hiding a test that should carry `integration` instead.
unit_bound=2
# judge_unit_run says the unit level as `go test -v` reported it and fails when it was over the bound: $1 the
# file holding the output of the untagged `go test -v` of every package (packages one after another, each
# ending in its `ok <package> <time>` line), $2 the bound in seconds. Packages run side by side, so the
# level's time is that of its slowest package; the five slowest tests of the run are named when it is over,
# so that the bound is kept by making them cheap or by tagging them `integration`, and never by raising it
# (issue 895).
judge_unit_run() {
out=$1 bound=$2
awk '
/^--- (PASS|FAIL|SKIP): / {
name = $3; took = $4; gsub(/[()s]/, "", took)
pending[++n] = took "\t" name
if ($2 == "SKIP:") skipped++
next
}
/^(ok|FAIL)[ \t]/ && NF >= 3 && $3 ~ /^[0-9.]+s$/ {
t = $3; sub(/s$/, "", t)
for (i = 1; i <= n; i++) print "T\t" pending[i] "\t" $2
n = 0
print "P\t" t "\t" $2
}
END { print "S\t" skipped + 0 }
' "$out" >"$out.judged"
tests=$(grep -c '^T' "$out.judged" || true)
skipped=$(awk -F'\t' '$1 == "S" {print $2}' "$out.judged")
level=$(awk -F'\t' '$1 == "P" && $2 + 0 > m {m = $2 + 0} END {printf "%.2f", m}' "$out.judged")
echo "unit level: $tests tests ($skipped skipped), $level s (bound $bound s)"
if awk -v t="$level" -v b="$bound" 'BEGIN {exit !(t + 0 > b + 0)}'; then
echo "the unit level took $level s, over its bound of $bound s; the slowest tests:"
awk -F'\t' '$1 == "T" {print $2 "\t" $4 "\t" $3}' "$out.judged" | sort -t"$(printf '\t')" -k1,1 -nr | head -n 5 |
awk -F'\t' '{printf " %s s %s %s\n", $1, $3, $2}'
echo "make them cheap (no wait, no process), or give their files //go:build integration if they start a bus or another process"
return 1
fi
}
# `sh go/merge-check.sh judge-unit-run <output> <bound>` runs only the judging of the unit level, on a
# recorded output.
if [ "${1:-}" = judge-unit-run ]; then
shift
judge_unit_run "$@"
exit
fi
if ! fetched=$(go mod download 2>&1); then
echo "FAILED: the dependencies cannot be fetched: $(printf '%s\n' "$fetched" | tail -n 1)"
exit 1
fi
if ! verified=$(go mod verify 2>&1); then
echo "FAILED: a dependency does not match go.sum: $verified"
exit 1
fi
unformatted=$(find . -name '*.go' -not -path './vendor/*' | xargs gofmt -l)
if [ -n "$unformatted" ]; then
echo "not gofmt'd:"
echo "$unformatted"
exit 1
fi
go vet ./...
go test -count=1 ./...
# The files tagged `integration` are vetted too, or nothing would read them.
go vet -tags integration ./...
# NOT RACE-CHECKED here: the SDK's suite is small and carries no store or other shared state between
# packages, so the race detector is not required of this check (novox/hq issue 1175); a module that needs
# it asks for it in its own merge-check.sh.
echo "NOT RACE-CHECKED: the Go binding's check runs without the race detector"
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
# The unit level first and alone: every package untagged, no bus or other process (judged above).
status=0
go test -count=1 -v ./... >"$work/unit.out" 2>&1 || status=$?
if [ "$status" -ne 0 ]; then
grep -vE '^(=== |[[:space:]]*--- PASS)' "$work/unit.out"
echo "the unit level failed (go test exited $status)"
exit 1
fi
judge_unit_run "$work/unit.out" "$unit_bound"
# The integration level: the whole suite with the tag, each test against its own bus (internal/testbus).
integration_start=$(date +%s)
status=0
go test -tags integration -count=1 -v ./... >"$work/integration.out" 2>&1 || status=$?
grep -vE '^(=== |[[:space:]]*--- PASS)' "$work/integration.out" || true
tests=$(grep -cE '^--- (PASS|FAIL|SKIP): ' "$work/integration.out" || true)
skipped=$(grep -cE '^--- SKIP: ' "$work/integration.out" || true)
echo "integration level: $tests tests ($skipped skipped), $(($(date +%s) - integration_start)) s"
if [ "$status" -ne 0 ]; then
echo "the integration level failed (go test exited $status)"
exit 1
fi
+276
View File
@@ -0,0 +1,276 @@
// Package outward is the content rule (novox/hq ADR 0310 §4, to-be 45 §5, research 028 Q8, ADR 0227): what
// may leave the mesh, and what an answer may carry at all. One rule, here, so every module that answers outward
// applies the same one rather than a copy: the operator channel's router and every node-reads answer.
//
// Two strengths:
//
// - Check is the operator channel's: roles and words only. A message carrying an address, a path or
// anything shaped like a secret is refused, because a channel such as Telegram is not end-to-end
// encrypted for bots and this rule is the only thing between a condition's words and someone else's
// server. Deliberately wider than it must be: a commit id or a long random name is refused too.
// - CheckSecret is the secret class alone: what leaves the mesh nowhere and is answered nowhere. A path
// or an address is what a file read or an operator's approval is about; a secret's shape never is.
//
// A refusal says its class and a few words, never the text itself.
package outward
import (
"math"
"regexp"
"strings"
"unicode"
)
// Refusal says why a text may not leave: the class of what it carried, never the text itself.
type Refusal struct {
Class string // address, path or secret
What string // a few words: "an IPv4 address", "a URL", …
}
func (r Refusal) String() string { return r.Class + " (" + r.What + ")" }
var (
reURL = regexp.MustCompile(`(?i)\b[a-z][a-z0-9+.-]*://`)
reEmail = regexp.MustCompile(`[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,}`)
reIPv4 = regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}\b`)
reIPv6 = regexp.MustCompile(`(?i)(^|[^0-9a-z:])(([0-9a-f]{1,4}:){4,7}[0-9a-f]{1,4}|([0-9a-f]{1,4}:)*[0-9a-f]{0,4}::([0-9a-f]{1,4}:)*[0-9a-f]{0,4})([^0-9a-z:]|$)`)
reMAC = regexp.MustCompile(`(?i)\b([0-9a-f]{2}[:-]){5}[0-9a-f]{2}\b`)
rePEM = regexp.MustCompile(`-----BEGIN [A-Z ]+-----`)
reJWT = regexp.MustCompile(`\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}`)
reBotToken = regexp.MustCompile(`\b\d{6,}:[A-Za-z0-9_-]{30,}`)
reKnown = regexp.MustCompile(`\b(gh[pousr]_[A-Za-z0-9]{20,}|glpat-[A-Za-z0-9_-]{16,}|sk-[A-Za-z0-9_-]{16,}|xox[abprs]-[A-Za-z0-9-]{10,}|AKIA[0-9A-Z]{16})`)
reAssigned = regexp.MustCompile(`(?i)\b(password|passwd|passphrase|secret|token|api[_-]?key|apikey|credential|private[_-]?key)\s*[=:]\s*\S`)
// reNamedSecret is a value given to a name that ends in a secret's word after `_` or `-`, as an environment
// or configuration name is written: MESH_TOKEN=, DB_PASS=, aws_secret_access_key = …, SUDO_PASSWORD value.
// The word ends the name: what follows it is `=`, `:` or a space and then a value.
reNamedSecret = regexp.MustCompile(`(?i)(^|[^a-z0-9_-])[a-z0-9]+[_-]([a-z0-9]+[_-])*(password|passwd|passphrase|pass|pwd|secret|token|api[_-]?key|apikey|access[_-]?key|secret[_-]?key|private[_-]?key|credentials?|psk)(\s*[=:]\s*|\s+)[^\s=:]`)
// reURLCredential is a password inside an address: scheme://user:password@host.
reURLCredential = regexp.MustCompile(`(?i)\b[a-z][a-z0-9+.-]*://[^/\s:@]+:[^/\s@]+@`)
// reSecretRun is a run CheckSecret judges: reRun with `-`, so a base64url value is judged whole.
reSecretRun = regexp.MustCompile(`[A-Za-z0-9+/=_-]{20,}`)
reUUID = regexp.MustCompile(`^(?i)[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
reHex = regexp.MustCompile(`(?i)\b[0-9a-f]{32,}\b`)
reRun = regexp.MustCompile(`[A-Za-z0-9+/=_]{20,}`)
reWinPath = regexp.MustCompile(`(?i)\b[a-z]:\\`)
)
// topLevel are names that end a host name — the generic and country ones a mesh's names use, and the
// private ones (.internal, .lan, .home, .local) a mesh is likelier to.
var topLevel = map[string]bool{}
func init() {
for _, t := range strings.Fields(`com net org edu gov mil int io dev app cloud ai co me info biz xyz
site online tech page link
be nl de fr uk lu eu ch at it es pt se no dk fi pl cz us ca au nz jp cn ru in br ie
internal lan home local localdomain corp intranet private arpa test example invalid localhost`) {
topLevel[t] = true
}
}
// Check says whether a text may leave the mesh, and when not, why.
func Check(text string) (Refusal, bool) {
switch {
case reURL.MatchString(text):
return Refusal{"address", "a URL"}, false
case reEmail.MatchString(text):
return Refusal{"address", "a mail address"}, false
case reIPv4.MatchString(text):
return Refusal{"address", "an IPv4 address"}, false
case reMAC.MatchString(text):
return Refusal{"address", "a hardware address"}, false
case reIPv6.MatchString(text):
return Refusal{"address", "an IPv6 address"}, false
}
if refusal, ok := secretShape(text); !ok {
return refusal, false
}
if reWinPath.MatchString(text) {
return Refusal{"path", "a drive path"}, false
}
if refusal, ok := randomRun(text); !ok {
return refusal, false
}
for _, word := range strings.FieldsFunc(text, func(r rune) bool {
return unicode.IsSpace(r) || strings.ContainsRune("\"'`()[]{}<>,;|", r)
}) {
w := strings.TrimRight(word, ".:!?")
if isPath(w) {
return Refusal{"path", "a file path"}, false
}
if isHostName(w) {
return Refusal{"address", "a host name"}, false
}
}
return Refusal{}, true
}
// CheckSecret says whether a text carries a secret's shape — the one class that leaves the mesh nowhere. It is
// what an ask's whole words (asks.Ask.Whole, novox/hq issue 383) are held to: on a kind that proves who answers,
// a path or an address is what the operator approves and may be shown, a secret never. A path is read as one:
// a run with a slash in it is judged piece by piece between the slashes, so "/mnt/Photos_2024/Jochen" is a
// path and not a random string (the review of 2026-10-10); the named shapes — a key block, a signed token, a
// bot token, a known token shape, a value given to a secret's name, a long hexadecimal string — hold whatever
// the run holds.
func CheckSecret(text string) (Refusal, bool) {
if refusal, ok := secretShape(text); !ok {
return refusal, false
}
return randomRunOutsidePaths(text)
}
// randomRunOutsidePaths judges every long run of a value's characters, a path's pieces between its slashes
// alone (the review of mesh-sdk #23): a run is read as a path only where it is one — it starts with a slash,
// follows a dot or a tilde (a host's or a home's path), or a name's `=` gives it one — so a base64 value, whose
// slashes fall anywhere, is judged whole. A UUID is a name, never a secret.
func randomRunOutsidePaths(text string) (Refusal, bool) {
for _, loc := range reSecretRun.FindAllStringIndex(text, -1) {
var before byte
if loc[0] > 0 {
before = text[loc[0]-1]
}
run := text[loc[0]:loc[1]]
// Standard base64 ends in its padding: such a run is a value, even when it starts with a slash.
padded := strings.HasSuffix(run, "=") && !strings.Contains(strings.TrimRight(run, "="), "=")
for i, seg := range strings.Split(run, "=") {
path := !padded && (strings.HasPrefix(seg, "/") || (i == 0 && (before == '.' || before == '~')))
pieces := []string{seg}
if path {
pieces = strings.Split(seg, "/")
}
for _, piece := range pieces {
if len(piece) >= 20 && !reUUID.MatchString(piece) && looksRandomValue(piece) {
return Refusal{"secret", "a long random-looking string"}, false
}
}
}
}
return Refusal{}, true
}
// secretShape is the secret shapes a pattern names.
func secretShape(text string) (Refusal, bool) {
switch {
case rePEM.MatchString(text):
return Refusal{"secret", "a key block"}, false
case reJWT.MatchString(text):
return Refusal{"secret", "a signed token"}, false
case reBotToken.MatchString(text):
return Refusal{"secret", "a bot token"}, false
case reKnown.MatchString(text):
return Refusal{"secret", "a known token shape"}, false
case reURLCredential.MatchString(text):
return Refusal{"secret", "a password in an address"}, false
case reAssigned.MatchString(text), reNamedSecret.MatchString(text):
return Refusal{"secret", "a value given to a secret's name"}, false
case reHex.MatchString(text):
return Refusal{"secret", "a long hexadecimal string"}, false
}
return Refusal{}, true
}
// randomRun is a long unbroken run of characters spread as a random string's are.
func randomRun(text string) (Refusal, bool) {
for _, run := range reRun.FindAllString(text, -1) {
if looksRandom(run) {
return Refusal{"secret", "a long random-looking string"}, false
}
}
return Refusal{}, true
}
// isPath: absolute, home-relative or dot-relative, or two separators deep. A mesh address names one
// machine and one tool (`ace/postgres.query`) and has one; a ratio ("3/4") has digits only.
func isPath(w string) bool {
if w == "" {
return false
}
if strings.HasPrefix(w, "/") && len(w) > 1 {
return true
}
for _, p := range []string{"~/", "./", "../", "$HOME", "${"} {
if strings.HasPrefix(w, p) {
return true
}
}
return strings.Count(w, "/") >= 2 || strings.Contains(w, "\\")
}
// isHostName: two names or more, the last a top-level one. A condition key's last name is its kind
// (`machine.ace.silent`), which none of these is.
func isHostName(w string) bool {
w = strings.ToLower(w)
if w == "localhost" {
return true
}
parts := strings.Split(w, ".")
if len(parts) < 2 {
return false
}
for _, p := range parts {
if p == "" {
return false
}
}
return topLevel[parts[len(parts)-1]]
}
// looksRandom: letters and digits mixed, and the characters spread as a random string's are. A
// sentence's words are separated, so only an unbroken run reaches here.
func looksRandom(s string) bool {
var letters, digits int
counts := map[rune]int{}
for _, r := range s {
counts[r]++
switch {
case unicode.IsLetter(r):
letters++
case unicode.IsDigit(r):
digits++
}
}
if letters == 0 || digits == 0 {
// One class only: a word, or a number. A long number of digits alone is a count or a time.
return letters > 0 && hasUpperAndLower(s) && entropy(counts, len(s)) >= 4.0
}
return entropy(counts, len(s)) >= 3.3
}
// looksRandomValue is looksRandom for a value in a file: letters and digits mixed and spread as a random string's
// are; letters alone when the case changes as a random string's does — about as many capitals as small letters,
// where a CamelCase name (sshd's PasswordAuthentication) has one a word.
func looksRandomValue(s string) bool {
var upper, lower, digits int
counts := map[rune]int{}
for _, r := range s {
counts[r]++
switch {
case unicode.IsUpper(r):
upper++
case unicode.IsLower(r):
lower++
case unicode.IsDigit(r):
digits++
}
}
if upper+lower == 0 {
return false
}
if digits > 0 {
return entropy(counts, len(s)) >= 3.3
}
share := float64(upper) / float64(upper+lower)
return share >= 0.3 && share <= 0.7 && entropy(counts, len(s)) >= 3.5
}
func hasUpperAndLower(s string) bool {
return strings.IndexFunc(s, unicode.IsUpper) >= 0 && strings.IndexFunc(s, unicode.IsLower) >= 0
}
func entropy(counts map[rune]int, n int) float64 {
e := 0.0
for _, c := range counts {
p := float64(c) / float64(n)
e -= p * math.Log2(p)
}
return e
}
+164
View File
@@ -0,0 +1,164 @@
package outward
// The operator channel's cases, carried here with the rule (novox/hq ADR 0310 §4): the rule moved, its meaning did not.
import (
"encoding/base64"
"fmt"
"math/rand"
"testing"
)
func TestRolesAndWordsMayLeave(t *testing.T) {
for _, s := range []string{
"URGENT: the home server has not been heard for 15 min",
"key: provider.keycloak.ace.gitea.failing",
"more: conditions show machine.ace.silent",
"since: 2026-10-06 12:30 UTC",
"call call-1759752000123456789-12 has run past its bound of 10 min",
"ace/postgres.query answers no more",
"3/4 machines answered; plan 41 waits on build 7f3a9c1e",
"the bus advisory slow-consumer for ace_records",
"core.controller.novox.rolled-back",
"seat node-notifier has no live holder",
"HELD BACK: 5 message(s) over the cap of 20 an hour",
"STILL OPEN after 1.5 h: the controller's event loop takes no message",
"node-engine and node tools builds differ from the plan's",
} {
if r, ok := Check(s); !ok {
t.Errorf("refused %q: %s", s, r)
}
}
}
func TestAnAddressAPathOrASecretMayNot(t *testing.T) {
for s, class := range map[string]string{
"cannot reach 192.168.1.20": "address",
"listening on 10.0.0.7:5432": "address",
"route fd00:1234:5678::1 is gone": "address",
"fe80::1 answered": "address",
"2001:db8:0:0:0:0:2:1 answered": "address",
"see https://git.example.org/x": "address",
"git.novox.internal does not answer": "address",
"the mail for admin@example.org bounced": "address",
"zurag.be is down": "address",
"localhost refused": "address",
"the card aa:bb:cc:dd:ee:ff went away": "address",
"/var/lib/mesh-controller is full": "path",
"~/.config/hal/env changed": "path",
"read ./grants.json": "path",
"services/postgres/data/pg_hba.conf": "path",
"C:\\Users\\x": "path",
"token 123456789:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw": "secret",
"ghp_abcdefghijklmnopqrstuvwxyz0123456789": "secret",
"password=hunter2": "secret",
"api_key: x": "secret",
"-----BEGIN OPENSSH PRIVATE KEY-----": "secret",
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0": "secret",
"commit 3f2a9c1e5b7d4f6a8c0e2b4d6f8a0c2e4b6d8f0a": "secret",
"value Zx8Qp2Lm9Rt4Vb7Nc1Kd5Hs3Jf6": "secret",
} {
r, ok := Check(s)
if ok {
t.Errorf("let %q leave", s)
continue
}
if r.Class != class {
t.Errorf("%q refused as %s, want %s", s, r, class)
}
}
}
// The secret class alone (novox/hq ADR 0310 §4, issue 383): a path and an address pass, a path's pieces are
// judged between its slashes, and every named secret shape and a random run outside a path are refused.
func TestCheckSecretRefusesASecretsShapeAndNothingElse(t *testing.T) {
for text, may := range map[string]bool{
"+ shares: photos=/mnt/Photos_2024/Jochen": true,
"+ sources: games=smb://nas.lan/Recalbox_Games2024": true,
"+ library: /srv/media/Series_Archive/Season01": true,
"root:x:0:0::/root:/usr/bin/bash": true,
"nameserver 192.168.1.1": true,
"drwxr-x--- jochen users /home/jochen": true,
"+ key: xK9mQ2vL8pR4tW7yB3nF6hJ1dG5sA0zC": false,
"-----BEGIN OPENSSH PRIVATE KEY-----": false,
"password=hunter2": false,
"Authorization: token ghp_abcdefghijklmnopqrstuvwxyz0123456789": false,
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0": false,
"sha 3f2a9c1e5b7d4f6a8c0e2b4d6f8a0c2e4b6d8f0a": false,
} {
r, ok := CheckSecret(text)
if ok != may {
t.Errorf("CheckSecret(%q) = %v (%s), want %v", text, ok, r, may)
}
if !ok && r.Class != "secret" {
t.Errorf("CheckSecret(%q) refused as %s: only the secret class is refused there", text, r)
}
}
}
// The review of mesh-sdk #23: the shapes node-reads meets in env files, unit files, configurations and
// history files, each refused by its shape whatever the value looks like.
func TestCheckSecretRefusesTheShapesOfConfigurationFiles(t *testing.T) {
for _, text := range []string{
"url = nats://user:pa55word@host:4222",
"DATABASE_URL=postgres://app:hunter2@db/app",
"MESH_TOKEN=s3cr3t-v4lue",
"DB_PASS=hunter2",
"export GITEA_SECRET=abc",
"Environment=APP_API_KEY=xyz",
"SUDO_PASSWORD hunter2",
"aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"password = hunter2",
"wifi_psk: correcthorse",
"value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
} {
if r, ok := CheckSecret(text); ok || r.Class != "secret" {
t.Errorf("CheckSecret(%q) let it through (%v)", text, r)
}
}
// What a configuration file says that is not a secret.
for _, text := range []string{
"ExecStart=/usr/lib/systemd/systemd-resolved",
"UUID=3f2a9c1e-5b7d-4f6a-8c0e-2b4d6f8a0c2e /home btrfs defaults 0 0",
"password required pam_unix.so",
"bypass = true",
"PasswordAuthentication no",
"url = https://git.example.org/novox/mesh-sdk.git",
"HOME=/var/lib/mesh-reads",
"TOKEN_FILE=/var/lib/app/token",
} {
if r, ok := CheckSecret(text); !ok {
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
}
}
}
// Random values as base64 and base64url, wherever their slashes fall: every one is refused.
func TestCheckSecretRefusesRandomValuesAsBase64(t *testing.T) {
rng := rand.New(rand.NewSource(418))
passed := map[string]int{}
for i := 0; i < 1000; i++ {
for _, n := range []int{16, 32} {
b := make([]byte, n)
rng.Read(b)
for name, enc := range map[string]*base64.Encoding{"std": base64.StdEncoding, "url": base64.RawURLEncoding} {
v := enc.EncodeToString(b)
if _, ok := CheckSecret("key " + v); ok {
passed[fmt.Sprintf("%s/%d", name, n)]++
}
}
}
}
if len(passed) > 0 {
t.Errorf("random values passed as clean, of 1000 each: %v", passed)
}
}
// A CamelCase name is a word, not a value: the case changes once a word.
func TestCheckSecretLetsACamelCaseNameThrough(t *testing.T) {
for _, text := range []string{"PasswordAuthenticationIsOff", "ChallengeResponseAuthentication no", "KbdInteractiveAuthentication"} {
if r, ok := CheckSecret(text); !ok {
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
}
}
}
+54
View File
@@ -0,0 +1,54 @@
// Package process is a module's own process's connection to the bus (novox/hq ADR 0436 clause 4): it reads
// the account the controller delivers, connects as it with its pinned fingerprint and its inbox prefix, and
// offers emitting an event, the module's states with compare-and-set, consuming a work queue, reading the
// router's records of the module's own asks, and calling a tool or verb — each only as the account's grants
// allow. It offers no Serve and no way to serve a tool: a process never answers a call. A sibling of stdio,
// which it imports only for stdio's pure types (ErrChanged, CodeChanged, StateEntry, EmitOptions); it never
// touches stdio's current, channel or live(), which stay the bundle's own way to the bus.
package process
import (
"encoding/json"
"fmt"
"os"
)
// Account is a process's own bus account, exactly as the controller delivers it as the module's own secret
// broker-<process id> (mesh-controller cmd/mesh-controller/modules.go issueProcess, heldProcessAccount, and
// issueWith for a module's own broker account, which also carries claims). A process account carries no
// claims at all; this type reads either file without error.
type Account struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
// Claims rides on a module's own broker account only; a process holds no seat, so its own account
// carries none (novox/hq ADR 0436 §2, §3). Kept raw: this package never reads it.
Claims json.RawMessage `json:"claims,omitempty"`
}
// InboxPrefix is this account's own inbox, and nothing wider: `_INBOX.<user>`.
func (a Account) InboxPrefix() string { return "_INBOX." + a.User }
// ReadAccount reads and validates the account file the controller delivers: it refuses a file missing url,
// user or password, naming the field — never the password, which does not appear in an error.
func ReadAccount(path string) (Account, error) {
data, err := os.ReadFile(path)
if err != nil {
return Account{}, fmt.Errorf("reading the process account at %s: %w", path, err)
}
var a Account
if err := json.Unmarshal(data, &a); err != nil {
return Account{}, fmt.Errorf("the process account at %s is not JSON: %w", path, err)
}
for _, f := range []struct{ name, value string }{
{"url", a.URL}, {"user", a.User}, {"password", a.Password},
} {
if f.value == "" {
return Account{}, fmt.Errorf("the process account at %s has no %s", path, f.name)
}
}
return a, nil
}
+84
View File
@@ -0,0 +1,84 @@
package process
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// 1171/AC2: given an account file broker-<process id> in the controller's form {url, fingerprint, node,
// module, user, password, claims}, when the process package reads it, then it yields those fields and
// tolerates claims.
func TestReadAccountYieldsEveryFieldAndToleratesClaims(t *testing.T) {
path := writeAccount(t, map[string]any{
"url": "nats://novox:4222", "fingerprint": "sha256:ab", "node": "novox", "module": "mesh-mail",
"user": "novox.mesh-mail.send", "password": "secret-pw",
"claims": []map[string]any{{"seat": "operator-channel", "verbs": []string{"tell"}}},
})
a, err := ReadAccount(path)
if err != nil {
t.Fatalf("a well-formed account file was refused: %v", err)
}
if a.URL != "nats://novox:4222" || a.Fingerprint != "sha256:ab" || a.Node != "novox" ||
a.Module != "mesh-mail" || a.User != "novox.mesh-mail.send" || a.Password != "secret-pw" {
t.Fatalf("the account read back: %+v", a)
}
}
// 1171/AC2: a process account carries no claims at all (heldProcessAccount); reading it is not refused for
// the field being absent.
func TestReadAccountToleratesNoClaimsAtAll(t *testing.T) {
path := writeAccount(t, map[string]any{
"url": "nats://novox:4222", "node": "novox", "module": "mesh-mail",
"user": "novox.mesh-mail.send", "password": "secret-pw",
})
a, err := ReadAccount(path)
if err != nil {
t.Fatalf("a process account with no claims was refused: %v", err)
}
if a.User != "novox.mesh-mail.send" {
t.Fatalf("the account read back: %+v", a)
}
}
// 1171/AC2: a file missing url, user or password is refused, naming the field; the password never appears
// in the error even when it is the field present and another is missing.
func TestReadAccountRefusesAFileMissingAField(t *testing.T) {
for _, tc := range []struct {
field string
values map[string]any
}{
{"url", map[string]any{"node": "novox", "module": "mesh-mail", "user": "novox.mesh-mail.send", "password": "secret-pw"}},
{"user", map[string]any{"url": "nats://novox:4222", "node": "novox", "module": "mesh-mail", "password": "secret-pw"}},
{"password", map[string]any{"url": "nats://novox:4222", "node": "novox", "module": "mesh-mail", "user": "novox.mesh-mail.send"}},
} {
t.Run(tc.field, func(t *testing.T) {
path := writeAccount(t, tc.values)
_, err := ReadAccount(path)
if err == nil {
t.Fatalf("an account file missing %s was accepted", tc.field)
}
if !strings.Contains(err.Error(), tc.field) {
t.Errorf("the error %q does not name the missing field %s", err.Error(), tc.field)
}
if strings.Contains(err.Error(), "secret-pw") {
t.Errorf("the password appears in the error: %q", err.Error())
}
})
}
}
func writeAccount(t *testing.T, fields map[string]any) string {
t.Helper()
b, err := json.Marshal(fields)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "broker-send")
if err := os.WriteFile(path, b, 0o600); err != nil {
t.Fatal(err)
}
return path
}
+68
View File
@@ -0,0 +1,68 @@
package process
import (
"context"
"encoding/json"
"fmt"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Ask reads the router's record of one of this process's module's own asks, under its own module and no
// other's (novox/hq ADR 0436 clause 4, ADR 0259 §3): by direct get on bucket, the same subject the tool
// runner reads (mesh-tools node-tools/internal/bus/seat.go ~300-345):
//
// $JS.API.DIRECT.GET.KV_<bucket>.$KV.<bucket>.<module>.<id>
//
// bucket is the router's records bucket as the caller's own manifest names it (operator-channel's is
// messenger_asks in the controller's own test fixtures) — never hard-coded to one bucket here, since a
// process may read records of more than one seat's router. A missing, deleted or purged key is nil, not
// an error.
func (c *Conn) Ask(bucket, id string) (*stdio.StateEntry, error) {
if err := checkAskID(id); err != nil {
return nil, err
}
subject := "$JS.API.DIRECT.GET.KV_" + bucket + ".$KV." + bucket + "." + c.account.Module + "." + id
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
reply, err := c.nc.RequestWithContext(ctx, subject, nil)
if err != nil {
return nil, fmt.Errorf("reading %s's ask %s from %s: %w", c.account.Module, id, bucket, err)
}
if status := reply.Header.Get("Status"); status == "404" {
return nil, nil
} else if status != "" {
return nil, fmt.Errorf("the record %s.%s could not be read: %s %s", c.account.Module, id, status,
reply.Header.Get("Description"))
}
if op := reply.Header.Get("KV-Operation"); op == "DEL" || op == "PURGE" {
return nil, nil
}
var revision uint64
if _, err := fmt.Sscan(reply.Header.Get("Nats-Sequence"), &revision); err != nil {
return nil, fmt.Errorf("the record %s.%s carries no readable revision: %w", c.account.Module, id, err)
}
return &stdio.StateEntry{Key: c.account.Module + "." + id, Value: askValueOf(reply.Data), Revision: revision}, nil
}
// checkAskID refuses an id holding a dot or a wildcard before anything is sent: either would name a
// different key under the direct-get subject's own dot-joined path, never this one (novox/hq issue 1189
// AC2).
func checkAskID(id string) error {
if id == "" || strings.ContainsAny(id, ".*>") {
return fmt.Errorf("%q is not an ask id: an ask id holds no dot and no wildcard, naming one key and "+
"no other", id)
}
return nil
}
// askValueOf is a record's raw bytes as JSON, quoting them when they are not already valid JSON
// (mesh-tools node-tools/internal/bus/state.go:351-357, valueOf).
func askValueOf(raw []byte) json.RawMessage {
if len(raw) == 0 || !json.Valid(raw) {
b, _ := json.Marshal(string(raw))
return b
}
return json.RawMessage(append([]byte(nil), raw...))
}
+60
View File
@@ -0,0 +1,60 @@
//go:build integration
package process
import (
"testing"
"github.com/nats-io/nats.go/jetstream"
)
// 1189/AC1: given the router's records bucket on a test bus holding a record under <module>.<ask id>,
// when the process connection reads its module's ask of that id, then it gets the record's bytes and
// revision by direct get; a deleted or purged key is said as absent; a missing key as absent.
func TestAskReadsTheRoutersRecordOfItsOwnAsk(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
kv, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "messenger_asks"})
if err != nil {
t.Fatalf("the admin could not make the records bucket: %v", err)
}
rev, err := kv.Put(ctxFor(t), "mesh-mail.ask-1", []byte(`{"state":"decided"}`))
if err != nil {
t.Fatal(err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
e, err := conn.Ask("messenger_asks", "ask-1")
if err != nil {
t.Fatalf("reading an existing record: %v", err)
}
if e == nil || e.Revision != rev || string(e.Value) != `{"state":"decided"}` {
t.Fatalf("read back %+v, want revision %d and {\"state\":\"decided\"}", e, rev)
}
if err := kv.Delete(ctxFor(t), "mesh-mail.ask-1"); err != nil {
t.Fatal(err)
}
if e, err := conn.Ask("messenger_asks", "ask-1"); err != nil || e != nil {
t.Fatalf("a deleted record: %+v, %v, want absent", e, err)
}
if _, err := kv.Create(ctxFor(t), "mesh-mail.ask-2", []byte(`{"state":"asked"}`)); err != nil {
t.Fatal(err)
}
if err := kv.Purge(ctxFor(t), "mesh-mail.ask-2"); err != nil {
t.Fatal(err)
}
if e, err := conn.Ask("messenger_asks", "ask-2"); err != nil || e != nil {
t.Fatalf("a purged record: %+v, %v, want absent", e, err)
}
if e, err := conn.Ask("messenger_asks", "ask-missing"); err != nil || e != nil {
t.Fatalf("a missing record: %+v, %v, want absent", e, err)
}
}
+28
View File
@@ -0,0 +1,28 @@
package process
import "testing"
// 1189/AC2: given an ask id holding a dot or a wildcard, when read, then it is refused before anything is
// sent — proven on a Conn with no live connection at all: if Ask tried to send, it would panic on the nil
// *nats.Conn instead of returning a clean error.
func TestAskRefusesAnInvalidIDBeforeSendingAnything(t *testing.T) {
c := &Conn{account: Account{Module: "mesh-mail"}}
for _, id := range []string{"", "a.b", "a*", "a>", "ask.1", "ask.*"} {
if _, err := c.Ask("messenger_asks", id); err == nil {
t.Errorf("Ask(%q) was not refused", id)
}
}
}
func TestCheckAskID(t *testing.T) {
for _, id := range []string{"", "a.b", "a*b", "a>b", ".", "*"} {
if err := checkAskID(id); err == nil {
t.Errorf("checkAskID(%q) was accepted", id)
}
}
for _, id := range []string{"ask-1", "a1b2c3", "send_42"} {
if err := checkAskID(id); err != nil {
t.Errorf("checkAskID(%q) was refused: %v", id, err)
}
}
}
+84
View File
@@ -0,0 +1,84 @@
package process
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
)
// CallTimeout bounds a call to a tool or a seat verb (mesh-tools node-tools/internal/bus/bus.go:35,
// RequestTimeout): what a module already expects.
var CallTimeout = 30 * time.Second
// Call invokes a tool or a seat verb as this process's own account (novox/hq ADR 0436 clause 4), on the
// caller-named subject, falling back to the subject that names no caller when nothing answers it —
// exactly the provisioner client's own pattern (mesh-catalog modules/mesh-provisioner-client/cmd/
// mesh-provisioner-client/secrets.go:59-61, 93-97, 119, 125-133). key is "seat:<seat>.<verb>" for a seat's
// verb, or "<module>.<tool>" for a module's own tool, as a process's bus.invokes names it. The reply is
// {result, error, node} (mesh-tools node-tools/internal/bus/bus.go answerOf); a non-empty error is the
// call's failure. It offers no way to answer a call — a process reaches the bus only as itself, and
// never serves anything.
func (c *Conn) Call(key string, body any) (json.RawMessage, error) {
called, fallback, err := callSubjects(key, c.callerToken())
if err != nil {
return nil, err
}
data, err := json.Marshal(body)
if err != nil {
return nil, err
}
result, err := c.request(called, data)
if errors.Is(err, nats.ErrNoResponders) {
result, err = c.request(fallback, data)
}
return result, err
}
// callerToken is this account's user name as the last token of a call it publishes, each dot written ~
// (mesh-tools node-tools/internal/bus/caller.go:43, CallerToken).
func (c *Conn) callerToken() string { return strings.ReplaceAll(c.account.User, ".", "~") }
// callSubjects is the caller-named subject key is called on, and its plain fallback (mesh-tools
// node-tools/internal/bus/caller.go CalledSubject, toolShape, calledPattern).
func callSubjects(key, token string) (called, fallback string, err error) {
if rest, ok := strings.CutPrefix(key, "seat:"); ok {
seat, verb, ok := strings.Cut(rest, ".")
if !ok || seat == "" || verb == "" {
return "", "", fmt.Errorf("%q names no seat and verb (seat:<seat>.<verb>)", key)
}
return "mesh.seat." + seat + ".call." + verb + "." + token, "mesh.seat." + seat + ".tool." + verb, nil
}
module, tool, ok := strings.Cut(key, ".")
if !ok || module == "" || tool == "" {
return "", "", fmt.Errorf("%q is not <module>.<tool>, and does not start seat:<seat>.<verb>", key)
}
return "mesh.mod." + module + ".call." + tool + "." + token, "mesh.mod." + module + ".tool." + tool, nil
}
// request asks one subject once and reads its reply ({result, error, node}); a reply naming an error is
// the call's failure, not a transport one.
func (c *Conn) request(subject string, data []byte) (json.RawMessage, error) {
ctx, cancel := context.WithTimeout(context.Background(), CallTimeout)
defer cancel()
msg, err := c.nc.RequestWithContext(ctx, subject, data)
if err != nil {
return nil, err
}
var r struct {
Result json.RawMessage `json:"result"`
Error string `json:"error"`
Node string `json:"node"`
}
if err := json.Unmarshal(msg.Data, &r); err != nil {
return nil, fmt.Errorf("%s answered something that is not a seat's answer: %w", subject, err)
}
if r.Error != "" {
return nil, errors.New(r.Error)
}
return r.Result, nil
}
+237
View File
@@ -0,0 +1,237 @@
//go:build integration
package process
import (
"context"
"encoding/json"
"errors"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// 1191/AC1: given a stand-in holder answering seat operator-channel verb tell on a test bus, when the
// process connection of novox.mesh-mail.send calls seat:operator-channel.tell with {headline, lines,
// subject}, then the request goes to mesh.seat.operator-channel.call.tell.novox~mesh-mail~send with the
// arguments as the body, and the holder's {result} comes back; a reply with error set comes back as an
// error with those words.
func TestCallReachesTheCallerNamedSubjectAndReadsTheReply(t *testing.T) {
url, account := grantedSendProcess(t)
adminNC, _ := adminJS(t, url)
heard := make(chan *nats.Msg, 1)
sub, err := adminNC.Subscribe("mesh.seat.operator-channel.call.tell.novox~mesh-mail~send", func(m *nats.Msg) {
heard <- m
_ = m.Respond([]byte(`{"result":{"sent":true}}`))
})
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
if err := adminNC.Flush(); err != nil {
t.Fatal(err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
result, err := conn.Call("seat:operator-channel.tell",
map[string]any{"headline": "h", "lines": []string{"l1"}, "subject": "s"})
if err != nil {
t.Fatalf("Call: %v", err)
}
if string(result) != `{"sent":true}` {
t.Fatalf("result: %s", result)
}
select {
case msg := <-heard:
var body map[string]any
if err := json.Unmarshal(msg.Data, &body); err != nil {
t.Fatal(err)
}
if body["headline"] != "h" || body["subject"] != "s" {
t.Errorf("the arguments were not the call's body: %v", body)
}
case <-time.After(3 * time.Second):
t.Fatal("the holder never heard the call")
}
}
// 1191/AC1: a reply with error set comes back as an error with those words.
func TestCallReadsAnErrorReplyAsAnError(t *testing.T) {
url, account := grantedSendProcess(t)
adminNC, _ := adminJS(t, url)
sub, err := adminNC.Subscribe("mesh.seat.operator-channel.call.tell.novox~mesh-mail~send", func(m *nats.Msg) {
_ = m.Respond([]byte(`{"error":"refused: no such warrant"}`))
})
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
if err := adminNC.Flush(); err != nil {
t.Fatal(err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
_, err = conn.Call("seat:operator-channel.tell", map[string]any{})
if err == nil || err.Error() != "refused: no such warrant" {
t.Fatalf("Call with an error reply: %v, want the words of the error", err)
}
}
// 1191/AC2: given no responder on the caller-named subject, when the call is made, then it is asked
// again on mesh.seat.operator-channel.tool.tell (the tool runner's fallback).
func TestCallFallsBackToThePlainSubjectWhenNothingAnswersTheCallerNamedOne(t *testing.T) {
url, account := grantedSendProcess(t)
adminNC, _ := adminJS(t, url)
sub, err := adminNC.Subscribe("mesh.seat.operator-channel.tool.tell", func(m *nats.Msg) {
_ = m.Respond([]byte(`{"result":{"via":"fallback"}}`))
})
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
if err := adminNC.Flush(); err != nil {
t.Fatal(err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
result, err := conn.Call("seat:operator-channel.tell", map[string]any{})
if err != nil {
t.Fatalf("Call: %v", err)
}
if string(result) != `{"via":"fallback"}` {
t.Fatalf("result: %s, want the fallback holder's answer", result)
}
}
// 1191/AC4, release 2's acceptance test: given the test bus with accounts composed as the controller
// composes a process account (the subjects of mesh-controller internal/broker/process_grants.go), when
// one process connection emits, updates a state, reads an own ask's record, consumes and acknowledges its
// queue and calls tell, then each succeeds; and its attempt to subscribe a tool-call subject is refused
// by the bus.
func TestReleaseAcceptanceAProcessDoesExactlyItsBusDeclaration(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_sends"}); err != nil {
t.Fatalf("the admin could not make the state bucket: %v", err)
}
asksKV, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "messenger_asks"})
if err != nil {
t.Fatalf("the admin could not make the records bucket: %v", err)
}
if _, err := asksKV.Put(ctxFor(t), "mesh-mail.ask-1", []byte(`{"state":"decided"}`)); err != nil {
t.Fatal(err)
}
if _, err := adminJetStream.CreateStream(ctxFor(t), jetstream.StreamConfig{
Name: "MAIL_EVENTS", Subjects: []string{"mesh.mod.mesh-mail.event.>"}, Duplicates: time.Minute,
}); err != nil {
t.Fatalf("the admin could not make the events stream: %v", err)
}
makeQueue(t, url, account, 30*time.Second)
publishApproved(t, url, `{"send":"s1"}`)
adminNC, _ := adminJS(t, url)
holderSub, err := adminNC.Subscribe("mesh.seat.operator-channel.call.tell.novox~mesh-mail~send", func(m *nats.Msg) {
_ = m.Respond([]byte(`{"result":{}}`))
})
if err != nil {
t.Fatal(err)
}
defer func() { _ = holderSub.Unsubscribe() }()
if err := adminNC.Flush(); err != nil {
t.Fatal(err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// 1. Emits.
if err := conn.Emit("mail.send.sent", map[string]any{"send": "s1"}, stdio.EmitOptions{MessageID: "ev-1"}); err != nil {
t.Fatalf("Emit: %v", err)
}
// 2. Updates a state.
s := conn.State("sends")
rev, err := s.Create("k", map[string]any{"v": 1})
if err != nil {
t.Fatalf("State.Create: %v", err)
}
if _, err := s.Update("k", map[string]any{"v": 2}, rev); err != nil {
t.Fatalf("State.Update: %v", err)
}
// 3. Reads an own ask's record.
if e, err := conn.Ask("messenger_asks", "ask-1"); err != nil || e == nil {
t.Fatalf("Ask: %+v, %v", e, err)
}
// 4. Consumes and acknowledges its queue.
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
consumed := make(chan error, 1)
go func() {
consumed <- conn.Consume(ctx, "MAIL_SENDS", func(m *Message) {
if err := m.Ack(); err != nil {
t.Errorf("Ack: %v", err)
}
cancel()
})
}()
select {
case <-consumed:
case <-time.After(8 * time.Second):
t.Fatal("the process did not consume its own queue")
}
// 5. Calls tell.
if _, err := conn.Call("seat:operator-channel.tell", map[string]any{}); err != nil {
t.Fatalf("Call: %v", err)
}
// 6. Refused: subscribing a tool-call subject — a fresh connection, so the error handler catches
// exactly this one refusal and nothing earlier.
refusals := make(chan error, 1)
watched, err := Connect(account, nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refusals <- err }))
if err != nil {
t.Fatal(err)
}
defer watched.Close()
if _, err := watched.nc.Subscribe("mesh.mod.mesh-mail.tool.>", func(*nats.Msg) {}); err != nil {
t.Fatal(err)
}
if err := watched.nc.Flush(); err != nil {
t.Fatal(err)
}
select {
case err := <-refusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("subscribing a tool-call subject was not refused as a permissions violation: %v", err)
}
case <-time.After(3 * time.Second):
t.Error("subscribing a tool-call subject the process does not serve was not refused")
}
}
+41
View File
@@ -0,0 +1,41 @@
package process
import "testing"
// 1191/AC3: given a module tool key <module>.<tool>, when called, then the subject is
// mesh.mod.<module>.call.<tool>.<token> with the same fallback (mesh.mod.<module>.tool.<tool>).
func TestCallSubjectsForAModuleTool(t *testing.T) {
called, fallback, err := callSubjects("mesh-mail.send", "novox~mesh-mail~send")
if err != nil {
t.Fatal(err)
}
if called != "mesh.mod.mesh-mail.call.send.novox~mesh-mail~send" {
t.Errorf("called: %q", called)
}
if fallback != "mesh.mod.mesh-mail.tool.send" {
t.Errorf("fallback: %q", fallback)
}
}
// 1191/AC1's subject (the seat form): seat:<seat>.<verb> calls mesh.seat.<seat>.call.<verb>.<token>,
// falling back to mesh.seat.<seat>.tool.<verb>.
func TestCallSubjectsForASeatVerb(t *testing.T) {
called, fallback, err := callSubjects("seat:operator-channel.tell", "novox~mesh-mail~send")
if err != nil {
t.Fatal(err)
}
if called != "mesh.seat.operator-channel.call.tell.novox~mesh-mail~send" {
t.Errorf("called: %q", called)
}
if fallback != "mesh.seat.operator-channel.tool.tell" {
t.Errorf("fallback: %q", fallback)
}
}
func TestCallSubjectsRefusesAKeyNamingNeither(t *testing.T) {
for _, key := range []string{"", "nodot", "seat:", "seat:noverb", "."} {
if _, _, err := callSubjects(key, "token"); err == nil {
t.Errorf("callSubjects(%q) was accepted", key)
}
}
}
+162
View File
@@ -0,0 +1,162 @@
package process
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"errors"
"fmt"
"regexp"
"sync"
"github.com/nats-io/nats.go"
)
// Conn is one process's own connection to the bus (novox/hq ADR 0436 clause 4): dialled with the account
// the controller delivered, pinned to its fingerprint, on its own inbox prefix. It offers no Serve and no
// way to serve a tool — a process reaches the bus only as itself, and never answers a call.
type Conn struct {
nc *nats.Conn
account Account
mu sync.Mutex
next uint64
pending map[string]map[uint64]context.CancelCauseFunc
}
// errPermissionRefused marks a context cancelled because the bus refused this account's publish of the
// subject a pending call was waiting an answer on: a denied publish is refused only as an async protocol
// error with no reply (mesh-tools node-tools/internal/bus/caller.go publishRefused), so without this a
// call would wait out its whole timeout instead of failing at once.
var errPermissionRefused = errors.New("the bus refused this account's publish")
// refusedPublish reads the subject out of the bus's "-ERR 'Permissions Violation for Publish to
// "<subject>"'" (mesh-tools node-tools/internal/bus/caller.go's own regexp).
var refusedPublish = regexp.MustCompile(`(?i)Publish to "(\S+)"`)
// Connect dials the bus with account: pinned to its fingerprint when it carries one (refused with any
// other certificate), authenticated as its user and password, on the inbox prefix _INBOX.<user> and no
// other, reconnecting without limit as the tool runner does (novox/hq issue 1192: a process outlives a
// bus restart, never giving up the way the client library's own default of 60 attempts would). Ported
// from mesh-controller internal/broker/jetstream.go:106-132 (DialPinned, PinnedToFingerprint) since this
// package may not import the controller.
//
// An ErrorHandler among opts is chained behind this package's own, which watches every async error for a
// permission violation refusing a publish this connection is waiting an answer to, and cancels that wait
// at once.
func Connect(account Account, opts ...nats.Option) (*Conn, error) {
c := &Conn{account: account, pending: map[string]map[uint64]context.CancelCauseFunc{}}
nc, err := nats.Connect(account.URL, c.options(opts)...)
if err != nil {
return nil, fmt.Errorf("connecting the process account %s: %w", account.User, err)
}
c.nc = nc
return c, nil
}
// options is the full option list Connect dials with, apart from Connect so a unit test can check what
// it carries without a live bus: this package's own (the inbox prefix, the credentials, the pin,
// reconnecting without limit, and an ErrorHandler chaining any the caller passed), then opts, then this
// package's own ErrorHandler again last, so it is the one AsyncErrorCB nats.Connect keeps however opts
// ordered theirs.
func (c *Conn) options(opts []nats.Option) []nats.Option {
// opts are functions over a *nats.Options; applied to a scratch value, this reads off a caller's own
// ErrorHandler (if any) to chain, without needing nats.Connect's own URL/Servers parsing here.
var scratch nats.Options
for _, opt := range opts {
if opt != nil {
_ = opt(&scratch)
}
}
chained := scratch.AsyncErrorCB
all := []nats.Option{
nats.CustomInboxPrefix(c.account.InboxPrefix()),
nats.UserInfo(c.account.User, c.account.Password),
nats.MaxReconnects(-1),
}
if c.account.Fingerprint != "" {
all = append(all, nats.Secure(pinnedToFingerprint(c.account.Fingerprint)))
}
all = append(all, opts...)
return append(all, nats.ErrorHandler(func(nc *nats.Conn, sub *nats.Subscription, err error) {
c.publishRefused(err)
if chained != nil {
chained(nc, sub, err)
}
}))
}
// Close ends the connection.
func (c *Conn) Close() { c.nc.Close() }
// withRefusal derives a context from ctx that is also cancelled, with errPermissionRefused as its cause,
// the moment the bus refuses this account's publish of subject — before ctx's own deadline, which a
// denied publish would otherwise run out, refused only as an async error with no reply. Every call that
// requests on the bus registers the subject it is about to publish to here first.
func (c *Conn) withRefusal(ctx context.Context, subject string) (context.Context, func()) {
derived, cancel := context.WithCancelCause(ctx)
c.mu.Lock()
c.next++
id := c.next
if c.pending[subject] == nil {
c.pending[subject] = map[uint64]context.CancelCauseFunc{}
}
c.pending[subject][id] = cancel
c.mu.Unlock()
return derived, func() {
c.mu.Lock()
if list := c.pending[subject]; list != nil {
delete(list, id)
if len(list) == 0 {
delete(c.pending, subject)
}
}
c.mu.Unlock()
cancel(nil)
}
}
// publishRefused ends every call waiting on the subject the bus's error names, with errPermissionRefused
// as the cause (mesh-tools node-tools/internal/bus/caller.go publishRefused): true when there was one
// waiting.
func (c *Conn) publishRefused(err error) bool {
if !errors.Is(err, nats.ErrPermissionViolation) {
return false
}
m := refusedPublish.FindStringSubmatch(err.Error())
if len(m) < 2 {
return false
}
c.mu.Lock()
cancels := c.pending[m[1]]
delete(c.pending, m[1])
c.mu.Unlock()
for _, cancel := range cancels {
cancel(errPermissionRefused)
}
return len(cancels) > 0
}
// pinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other — the leaf alone, its
// name and chain deliberately not consulted, as mesh-controller's own pin does (jetstream.go:116-132).
func pinnedToFingerprint(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
got := "sha256:" + hex.EncodeToString(sum[:])
if got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%s…), expected %s…",
got[:23], want[:23])
}
return nil
},
}
}
+115
View File
@@ -0,0 +1,115 @@
//go:build integration
package process
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"encoding/json"
"math/big"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
)
// selfSignedServer is a nats-server with JetStream and TLS, self-signed, for this test alone (mirroring
// go/internal/testbus.Start, with a certificate added since testbus has none): a mesh's bus presents its own
// certificate in no public trust store, which is exactly what a process pins to (ADR 0436 clause 4).
func selfSignedServer(t *testing.T) (url, fingerprint string) {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatal(err)
}
tmpl := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "mesh-sdk-test-bus"},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(time.Hour),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(der)
fingerprint = "sha256:" + hex.EncodeToString(sum[:])
cert := tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key}
opts := &server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(),
NoLog: true, NoSigs: true, TLSConfig: &tls.Config{Certificates: []tls.Certificate{cert}},
Username: "novox.mesh-mail.send", Password: "the-pw"}
s, err := server.NewServer(opts)
if err != nil {
t.Fatalf("a TLS bus for this test could not be made: %v", err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
s.Shutdown()
t.Fatal("a TLS bus for this test did not come up within 10s")
}
t.Cleanup(func() {
s.Shutdown()
s.WaitForShutdown()
})
return s.ClientURL(), fingerprint
}
// mailSenderAccount is the account novox.mesh-mail.send would be delivered as, the module's own secret
// broker-send: written to a file and read back through ReadAccount (1171/AC2), exactly as a process
// reads it, rather than built by hand.
func mailSenderAccount(t *testing.T, url, fingerprint string) Account {
t.Helper()
data, err := json.Marshal(map[string]any{
"url": url, "fingerprint": fingerprint, "node": "novox", "module": "mesh-mail",
"user": "novox.mesh-mail.send", "password": "the-pw",
})
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "broker-send")
if err := os.WriteFile(path, data, 0o600); err != nil {
t.Fatal(err)
}
account, err := ReadAccount(path)
if err != nil {
t.Fatalf("ReadAccount on a well-formed file: %v", err)
}
return account
}
// 1171/AC3: given a test bus with TLS and the account composed for novox.mesh-mail.send, when the package
// connects with the file, then it connects with the inbox prefix _INBOX.novox.mesh-mail.send.
func TestConnectPinsTheFingerprintAndSetsTheInboxPrefix(t *testing.T) {
url, fingerprint := selfSignedServer(t)
account := mailSenderAccount(t, url, fingerprint)
conn, err := Connect(account)
if err != nil {
t.Fatalf("connecting with the right fingerprint was refused: %v", err)
}
defer conn.Close()
if !conn.nc.IsConnected() {
t.Fatal("the connection did not come up")
}
if got := conn.nc.Opts.InboxPrefix; got != "_INBOX.novox.mesh-mail.send" {
t.Errorf("inbox prefix: %q, want %q", got, "_INBOX.novox.mesh-mail.send")
}
}
// 1171/AC3: connecting with a fingerprint that is not the bus's own leaf certificate is refused.
func TestConnectRefusesTheWrongFingerprint(t *testing.T) {
url, _ := selfSignedServer(t)
account := mailSenderAccount(t, url, "sha256:0000000000000000000000000000000000000000000000000000000000000000")
if _, err := Connect(account); err == nil {
t.Fatal("connecting with the wrong fingerprint was not refused")
}
}
+69
View File
@@ -0,0 +1,69 @@
package process
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// Emit publishes an event as this process's module (novox/hq ADR 0436 clause 4), under its module's own
// namespace (mesh.mod.<module>.event.<type>, the same subject the module's own principal would publish it
// on), with the SDK's own envelope and headers (stdio.go:498-509). A chosen message id
// (EmitOptions.MessageID) is sent as x-event-id, and as the bus's own dedup id (Nats-Msg-Id), so a second
// publish inside the stream's duplicate window is dropped; empty is a random one, as stdio.Emit on main.
func (c *Conn) Emit(eventType string, body any, opts ...stdio.EmitOptions) error {
data, err := json.Marshal(body)
if err != nil {
return err
}
id := ""
for _, o := range opts {
if o.MessageID != "" {
id = o.MessageID
}
}
if id == "" {
id = eventID()
}
msg := nats.NewMsg("mesh.mod." + c.account.Module + ".event." + eventType)
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", c.account.Module)
msg.Header.Set("x-node", c.account.Node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
for _, o := range opts {
if o.CausationID != "" {
msg.Header.Set("x-causation-id", o.CausationID)
}
if o.Schema != "" {
msg.Header.Set("x-schema", o.Schema)
}
}
msg.Data = data
js, err := jetstream.New(c.nc)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
_, err = js.PublishMsg(ctx, msg, jetstream.WithMsgID(id))
return err
}
// eventID is a random UUID, version 4, as stdio's own eventID (go/stdio.go:515-522) — duplicated rather
// than imported, since stdio's is unexported and this package touches none of stdio's bundle machinery.
func eventID() string {
var b [16]byte
_, _ = rand.Read(b[:])
b[6] = (b[6] & 0x0f) | 0x40
b[8] = (b[8] & 0x3f) | 0x80
h := hex.EncodeToString(b[:])
return h[0:8] + "-" + h[8:12] + "-" + h[12:16] + "-" + h[16:20] + "-" + h[20:32]
}
+57
View File
@@ -0,0 +1,57 @@
//go:build integration
package process
import (
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// 1188/AC3: given a stream accepting mail.send.sent, when the process emits it twice with the same
// message id inside the duplicate window, then the stream holds one message; with the headers stdio's
// Emit sets (stdio.go:498-509).
func TestEmitTwiceWithTheSameIDIsKeptToOneWithTheSDKsHeaders(t *testing.T) {
url, account := grantedSendProcess(t)
account.Node = "novox"
_, adminJetStream := adminJS(t, url)
stream, err := adminJetStream.CreateStream(ctxFor(t), jetstream.StreamConfig{
Name: "MAIL_EVENTS", Subjects: []string{"mesh.mod.mesh-mail.event.>"}, Duplicates: time.Minute,
})
if err != nil {
t.Fatalf("the admin could not make the events stream: %v", err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
for i := 0; i < 2; i++ {
if err := conn.Emit("mail.send.sent", map[string]any{"send": "s1"}, stdio.EmitOptions{MessageID: "send-s1"}); err != nil {
t.Fatalf("emit %d: %v", i, err)
}
}
info, err := stream.Info(ctxFor(t))
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != 1 {
t.Fatalf("the stream holds %d messages, want exactly 1 (the second publish is a duplicate)", info.State.Msgs)
}
msg, err := stream.GetMsg(ctxFor(t), 1)
if err != nil {
t.Fatal(err)
}
if msg.Header.Get("x-event-id") != "send-s1" || msg.Header.Get("x-source") != "mesh-mail" ||
msg.Header.Get("x-node") != "novox" || msg.Header.Get("content-type") != "application/json" ||
msg.Header.Get("x-time") == "" {
t.Errorf("the stored message's headers are not the SDK's: %v", msg.Header)
}
}
+100
View File
@@ -0,0 +1,100 @@
//go:build integration
package process
import (
"context"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
// sendUser and sendPassword are mesh-mail's process "send" on novox, as mesh-controller would mint it
// (mesh-controller internal/broker/process_grants_test.go mailSenderProcess, Principal.Username).
const (
sendUser = "novox.mesh-mail.send"
sendPassword = "the-pw"
adminUser = "admin"
adminPass = "admin-pw"
)
// sendGrants is the process principal mailSenderProcess's whole publish set, copied from
// mesh-controller internal/broker/process_grants_test.go (TestAProcessIsGrantedExactlyItsBusDeclaration,
// "dumped from a passing run... 16 subjects, correct today"), since this package may not import the
// controller: the exact grant the controller composes for novox.mesh-mail.send.
var sendGrants = []string{
"$JS.ACK.MAIL_SENDS.novox_mesh-mail_send.>",
"$JS.API.CONSUMER.CREATE.KV_mesh-mail_sends.>",
"$JS.API.CONSUMER.DELETE.KV_mesh-mail_sends.>",
"$JS.API.CONSUMER.INFO.MAIL_SENDS.novox_mesh-mail_send",
"$JS.API.CONSUMER.MSG.NEXT.MAIL_SENDS.novox_mesh-mail_send",
"$JS.API.DIRECT.GET.KV_mesh-mail_sends.>",
"$JS.API.DIRECT.GET.KV_messenger_asks.$KV.messenger_asks.mesh-mail.>",
"$JS.API.STREAM.INFO.KV_mesh-mail_sends",
"$JS.FC.KV_mesh-mail_sends.>",
"$KV.mesh-mail_sends.>",
"mesh.mod.mesh-mail.event.mail.send.failed",
"mesh.mod.mesh-mail.event.mail.send.sent",
"mesh.seat.operator-channel.call.tell.*.novox~mesh-mail~send",
"mesh.seat.operator-channel.call.tell.novox~mesh-mail~send",
"mesh.seat.operator-channel.tool.tell",
"mesh.seat.operator-channel.tool.tell.*",
}
// grantedSendProcess is a test bus with JetStream, with two users: novox.mesh-mail.send, permissioned to
// exactly sendGrants and its own inbox (nats-server server.Options Users with Permissions, as
// mesh-controller's cmd/mesh-controller/process_live_test.go composes accounts for a real bus — copied as
// literal grants here since this package may not import the controller); and an unrestricted admin for the
// test's own setup (creating streams and buckets, standing in for the holder and the runtime).
func grantedSendProcess(t *testing.T) (url string, account Account) {
t.Helper()
opts := &server.Options{Host: "127.0.0.1", Port: server.RANDOM_PORT, JetStream: true, StoreDir: t.TempDir(),
NoLog: true, NoSigs: true,
Users: []*server.User{
{Username: adminUser, Password: adminPass},
{Username: sendUser, Password: sendPassword, Permissions: &server.Permissions{
Publish: &server.SubjectPermission{Allow: sendGrants},
Subscribe: &server.SubjectPermission{Allow: []string{"_INBOX." + sendUser + ".>"}},
}},
},
}
s, err := server.NewServer(opts)
if err != nil {
t.Fatalf("a granted test bus could not be made: %v", err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
s.Shutdown()
t.Fatal("a granted test bus did not come up within 10s")
}
t.Cleanup(func() {
s.Shutdown()
s.WaitForShutdown()
})
url = s.ClientURL()
return url, Account{URL: url, Node: "novox", Module: "mesh-mail", User: sendUser, Password: sendPassword}
}
// adminJS is the test's own unrestricted connection to url, for setting up fixtures.
func adminJS(t *testing.T, url string) (*nats.Conn, jetstream.JetStream) {
t.Helper()
nc, err := nats.Connect(url, nats.UserInfo(adminUser, adminPass), nats.CustomInboxPrefix("_INBOX."+adminUser))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
return nc, js
}
func ctxFor(t *testing.T) context.Context {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
t.Cleanup(cancel)
return ctx
}
+37
View File
@@ -0,0 +1,37 @@
package process
import (
"reflect"
"testing"
"github.com/nats-io/nats.go"
)
// 1171/AC3: it offers no Serve and no way to serve a tool — a process reaches the bus only as itself, and
// never answers a call. Checked by reflection over *Conn's whole method set and field list, so a later
// addition cannot bring one back without this test naming it.
func TestConnOffersNoServeAndExposesNoRawConnection(t *testing.T) {
ptrType := reflect.TypeOf(&Conn{})
for _, forbidden := range []string{"Serve", "ServeOn", "Handle", "Subscribe", "Respond", "Answer"} {
if _, ok := ptrType.MethodByName(forbidden); ok {
t.Errorf("*Conn has a method %s: a process offers no way to serve a tool", forbidden)
}
}
natsConnType := reflect.TypeOf(&nats.Conn{})
structType := reflect.TypeOf(Conn{})
for i := 0; i < structType.NumField(); i++ {
f := structType.Field(i)
if f.IsExported() && (f.Type == natsConnType || f.Type == natsConnType.Elem()) {
t.Errorf("Conn exposes its raw connection through the exported field %s", f.Name)
}
}
for i := 0; i < ptrType.NumMethod(); i++ {
m := ptrType.Method(i)
for o := 0; o < m.Type.NumOut(); o++ {
if m.Type.Out(o) == natsConnType || m.Type.Out(o) == natsConnType.Elem() {
t.Errorf("*Conn.%s returns the raw *nats.Conn", m.Name)
}
}
}
}
+127
View File
@@ -0,0 +1,127 @@
package process
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// Message is one piece of work this process's queue consumer delivered, with how many times the bus has
// delivered it. Nothing acts on its behalf: the handler decides its fate exactly once, by calling Ack,
// NakWithDelay or Term (mesh-tools node-tools/internal/bus/seat.go's seat work Consume, with the same
// three outcomes, offered here explicitly instead of by a handler's return value, since the caller
// chooses its own delay before a message comes back).
type Message struct {
Subject string
Body []byte
Headers map[string]string
// Delivered is how many times the bus has handed this message over, this time included.
Delivered uint64
ack func() error
nak func(time.Duration) error
term func() error
}
// Ack acknowledges the message: the bus never delivers it again.
func (m *Message) Ack() error { return m.ack() }
// NakWithDelay asks for the message to be delivered again no sooner than delay, its delivery count
// raised by one.
func (m *Message) NakWithDelay(delay time.Duration) error { return m.nak(delay) }
// Term terminates the message: the bus never delivers it again, and does not count it a dead letter the
// way running out of deliveries does.
func (m *Message) Term() error { return m.term() }
// ProcessID is this account's own process id, the last segment of its user name <node>.<module>.<id>
// (mesh-controller internal/broker, Principal.Username for KindProcess).
func (a Account) ProcessID() string {
prefix := a.Node + "." + a.Module + "."
if !strings.HasPrefix(a.User, prefix) {
return ""
}
return strings.TrimPrefix(a.User, prefix)
}
// QueueConsumerName is the one durable consumer a process's bus.queues names it on its work-queue stream
// (novox/hq ADR 0436 clauses 1, 2, 6; mesh-controller internal/broker/modstreams.go QueueConsumer):
// <node>_<module>_<process id>.
func (a Account) QueueConsumerName() string { return a.Node + "_" + a.Module + "_" + a.ProcessID() }
// errNoMessage is one pull that fetched nothing within its wait: not an error to the caller, just a
// reason to pull again.
var errNoMessage = errors.New("no message this pull")
// Consume binds this process's one durable consumer on stream (QueueConsumerName) and pulls from it, one
// message at a time, handing each to handle until ctx ends: it then stops pulling and returns ctx.Err(),
// never acknowledging a message still in handle's hands — only handle decides a message's fate, by
// calling Ack, NakWithDelay or Term on it.
func (c *Conn) Consume(ctx context.Context, stream string, handle func(*Message)) error {
js, err := jetstream.New(c.nc)
if err != nil {
return err
}
name := c.account.QueueConsumerName()
cons, err := js.Consumer(ctx, stream, name)
if err != nil {
return fmt.Errorf("binding %s's queue consumer %s on %s: %w", c.account.Module, name, stream, err)
}
return runQueue(ctx, fetchOne(cons), handle)
}
// fetchOne is one pull of at most one message, non-blocking past ctx: Fetch itself does not take a
// context, so this selects on its message channel and ctx together, and returns promptly once ctx ends
// even while a fetch is still outstanding.
func fetchOne(cons jetstream.Consumer) func(ctx context.Context) (*Message, error) {
return func(ctx context.Context) (*Message, error) {
batch, err := cons.Fetch(1, jetstream.FetchMaxWait(5*time.Second))
if err != nil {
return nil, err
}
select {
case msg, ok := <-batch.Messages():
if !ok {
return nil, errNoMessage
}
headers := map[string]string{}
for k := range msg.Headers() {
headers[k] = msg.Headers().Get(k)
}
var delivered uint64 = 1
if meta, err := msg.Metadata(); err == nil && meta != nil {
delivered = meta.NumDelivered
}
return &Message{Subject: msg.Subject(), Body: append([]byte(nil), msg.Data()...), Headers: headers,
Delivered: delivered, ack: msg.Ack, nak: msg.NakWithDelay, term: msg.Term}, nil
case <-ctx.Done():
return nil, ctx.Err()
}
}
}
// runQueue is Consume's loop, apart from the bus so a unit test can drive it with a fake next
// (novox/hq issue 1190 AC4): it checks ctx before every pull, and stops as soon as next or the check
// says ctx has ended, without touching the message past handing it to handle.
func runQueue(ctx context.Context, next func(context.Context) (*Message, error), handle func(*Message)) error {
for {
if err := ctx.Err(); err != nil {
return err
}
msg, err := next(ctx)
if err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
if errors.Is(err, errNoMessage) {
continue
}
return err
}
handle(msg)
}
}
+223
View File
@@ -0,0 +1,223 @@
//go:build integration
package process
import (
"context"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
)
// mailSendsApprovedSubject is where the runtime publishes a work-queue message on mesh-mail's own stream
// (mesh-controller internal/broker/modstreams.go ModuleStreamSubject, mail-sends' one event).
const mailSendsApprovedSubject = "mesh.stream.mesh-mail.event.mail.send.approved"
// makeQueue sets up MAIL_SENDS as a work-queue stream and binds the one durable consumer a process's
// bus.queues names it (QueueConsumerName), both as the admin, mirroring what the controller composes
// (mesh-controller internal/broker/modstreams.go QueueConsumer), named <node>_<module>_<process id>.
// ackWait is the controller's 30s in most tests, shortened in the Term test so waiting past it (to prove
// no redelivery) does not make that test slow.
func makeQueue(t *testing.T, url string, account Account, ackWait time.Duration) {
t.Helper()
_, adminJetStream := adminJS(t, url)
if _, err := adminJetStream.CreateStream(ctxFor(t), jetstream.StreamConfig{
Name: "MAIL_SENDS", Subjects: []string{mailSendsApprovedSubject}, Retention: jetstream.WorkQueuePolicy,
Discard: jetstream.DiscardNew,
}); err != nil {
t.Fatalf("the admin could not make the work-queue stream: %v", err)
}
if _, err := adminJetStream.CreateConsumer(ctxFor(t), "MAIL_SENDS", jetstream.ConsumerConfig{
Durable: account.QueueConsumerName(), AckPolicy: jetstream.AckExplicitPolicy, AckWait: ackWait,
MaxDeliver: 5,
}); err != nil {
t.Fatalf("the admin could not make the queue consumer: %v", err)
}
}
func publishApproved(t *testing.T, url string, body string) {
t.Helper()
nc, _ := adminJS(t, url)
if err := nc.Publish(mailSendsApprovedSubject, []byte(body)); err != nil {
t.Fatal(err)
}
if err := nc.FlushTimeout(3 * time.Second); err != nil {
t.Fatal(err)
}
}
// recv reads one value off ch, failing the test rather than hanging forever if nothing arrives within
// timeout.
func recv[T any](t *testing.T, ch <-chan T, timeout time.Duration, what string) T {
t.Helper()
select {
case v := <-ch:
return v
case <-time.After(timeout):
t.Fatalf("%s: nothing arrived within %s", what, timeout)
var zero T
return zero
}
}
// streamMsgs is how many messages MAIL_SENDS still holds.
func streamMsgs(t *testing.T, url string) uint64 {
t.Helper()
_, adminJetStream := adminJS(t, url)
stream, err := adminJetStream.Stream(ctxFor(t), "MAIL_SENDS")
if err != nil {
t.Fatal(err)
}
info, err := stream.Info(ctxFor(t))
if err != nil {
t.Fatal(err)
}
return info.State.Msgs
}
// 1190/AC1: given a work-queue stream with one durable pull consumer and a process connection, when a
// message is consumed and acknowledged, then it is gone from the stream.
func TestConsumeAckRemovesTheMessageFromTheStream(t *testing.T) {
url, account := grantedSendProcess(t)
makeQueue(t, url, account, 30*time.Second)
publishApproved(t, url, `{"send":"s1"}`)
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
got := make(chan *Message, 1)
go func() {
_ = conn.Consume(ctx, "MAIL_SENDS", func(m *Message) {
got <- m
cancel()
})
}()
msg := recv(t, got, 8*time.Second, "the message")
<-ctx.Done()
if err := msg.Ack(); err != nil {
t.Fatalf("Ack: %v", err)
}
deadline := time.Now().Add(5 * time.Second)
for {
if n := streamMsgs(t, url); n == 0 {
return
} else if time.Now().After(deadline) {
t.Fatalf("the stream still holds %d messages after the ack", n)
}
time.Sleep(100 * time.Millisecond)
}
}
// 1190/AC2: given a consumed message, when the handler asks for it to be delivered again after a delay,
// then it comes back no sooner than the delay with its delivery count raised by one.
func TestConsumeNakWithDelayRedeliversAfterTheDelayWithARaisedCount(t *testing.T) {
url, account := grantedSendProcess(t)
makeQueue(t, url, account, 30*time.Second)
publishApproved(t, url, `{"send":"s2"}`)
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
const delay = 2 * time.Second
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
type delivery struct {
at time.Time
delivered uint64
}
deliveries := make(chan delivery, 2)
go func() {
_ = conn.Consume(ctx, "MAIL_SENDS", func(m *Message) {
deliveries <- delivery{time.Now(), m.Delivered}
if m.Delivered == 1 {
_ = m.NakWithDelay(delay)
} else {
_ = m.Ack()
cancel()
}
})
}()
first := recv(t, deliveries, 8*time.Second, "the first delivery")
if first.delivered != 1 {
t.Fatalf("first delivery count: %d, want 1", first.delivered)
}
second := recv(t, deliveries, delay+8*time.Second, "the redelivery")
if second.delivered != 2 {
t.Fatalf("second delivery count: %d, want 2", second.delivered)
}
if elapsed := second.at.Sub(first.at); elapsed < delay {
t.Fatalf("redelivered after %s, want no sooner than %s", elapsed, delay)
}
}
// 1190/AC3: given a consumed message, when the handler terminates it, then it is never delivered again.
// A short AckWait (well under the 4s this then waits past) is what would otherwise make this pass
// whether or not Term did anything at all: without it, the message simply sits unredelivered until the
// controller's ordinary 30s AckWait, and the test would see the same "nothing arrived" result either way.
func TestConsumeTermNeverDeliversTheMessageAgain(t *testing.T) {
url, account := grantedSendProcess(t)
const ackWait = time.Second
makeQueue(t, url, account, ackWait)
publishApproved(t, url, `{"send":"s3"}`)
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
deliveries := make(chan *Message, 4)
go func() {
_ = conn.Consume(ctx, "MAIL_SENDS", func(m *Message) {
deliveries <- m
_ = m.Term()
})
}()
first := recv(t, deliveries, 8*time.Second, "the message")
if first.Delivered != 1 {
t.Fatalf("first delivery count: %d, want 1", first.Delivered)
}
// Term removes it from the work-queue stream and leaves the consumer with nothing pending — not just
// "has not come back yet", which a message merely sitting past an un-acted AckWait would also show.
deadline := time.Now().Add(5 * time.Second)
for {
if n := streamMsgs(t, url); n == 0 {
break
} else if time.Now().After(deadline) {
t.Fatalf("the stream still holds %d messages after Term", n)
}
time.Sleep(100 * time.Millisecond)
}
_, adminJetStream := adminJS(t, url)
cons, err := adminJetStream.Consumer(ctxFor(t), "MAIL_SENDS", account.QueueConsumerName())
if err != nil {
t.Fatal(err)
}
info, err := cons.Info(ctxFor(t))
if err != nil {
t.Fatal(err)
}
if info.NumAckPending != 0 {
t.Fatalf("the consumer still has %d acks pending after Term", info.NumAckPending)
}
select {
case again := <-deliveries:
t.Fatalf("a terminated message was delivered again: %+v", again)
case <-time.After(4 * ackWait):
}
}
+71
View File
@@ -0,0 +1,71 @@
package process
import (
"context"
"errors"
"testing"
"time"
)
// 1190/AC4: given the consume loop, when its context ends, then it stops pulling and returns without
// acknowledging a message in hand.
func TestRunQueueStopsPullingAndNeverActsOnAMessageWhenContextEnds(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
acked := false
msg := &Message{Subject: "test", ack: func() error { acked = true; return nil },
nak: func(time.Duration) error { return nil }, term: func() error { return nil }}
pulls := 0
next := func(ctx context.Context) (*Message, error) {
pulls++
return msg, nil
}
handled := 0
done := make(chan error, 1)
go func() {
done <- runQueue(ctx, next, func(m *Message) {
handled++
// The handler does not act on the message at all: a redelivery, a termination or an
// acknowledgement is its choice alone, never the loop's.
cancel()
})
}()
select {
case err := <-done:
if !errors.Is(err, context.Canceled) {
t.Fatalf("runQueue returned %v, want context.Canceled", err)
}
case <-time.After(3 * time.Second):
t.Fatal("runQueue did not stop when its context ended")
}
if handled != 1 {
t.Fatalf("handle was called %d times, want 1", handled)
}
if acked {
t.Error("runQueue acknowledged the message itself; only the handler may")
}
if pulls != 1 {
t.Fatalf("next was called %d times, want exactly 1: once the handler ends the context, runQueue's "+
"own check stops it before pulling again", pulls)
}
}
// 1190/AC4: a context already ended before the loop starts pulls nothing at all.
func TestRunQueueNeverPullsOnAnAlreadyEndedContext(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
calls := 0
err := runQueue(ctx, func(context.Context) (*Message, error) {
calls++
return nil, context.Canceled
}, func(*Message) { t.Fatal("handle was called on an already-ended context") })
if !errors.Is(err, context.Canceled) {
t.Fatalf("runQueue returned %v, want context.Canceled", err)
}
if calls != 0 {
t.Fatalf("next was called %d times, want 0", calls)
}
}
+27
View File
@@ -0,0 +1,27 @@
package process
import (
"testing"
"github.com/nats-io/nats.go"
)
// novox/hq issue 1192 (taken into this round, release 2's reviewer): Connect reconnects without limit, as
// the tool runner does, so a process outlives a bus restart instead of giving up after the client
// library's default of 60 attempts. Checked without a live bus: the option list Connect dials with is
// applied to a scratch nats.Options and MaxReconnect read back.
func TestConnectOptionsReconnectWithoutLimit(t *testing.T) {
c := &Conn{account: Account{Node: "novox", Module: "mesh-mail", User: "novox.mesh-mail.send", Password: "pw"}}
var o nats.Options
for _, opt := range c.options(nil) {
if opt == nil {
continue
}
if err := opt(&o); err != nil {
t.Fatal(err)
}
}
if o.MaxReconnect != -1 {
t.Errorf("MaxReconnect: %d, want -1 (no limit)", o.MaxReconnect)
}
}
+130
View File
@@ -0,0 +1,130 @@
package process
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// stateTimeout bounds a state call (Get, Create, Update) and an Emit (emit.go): a denied publish is
// caught at once by Conn's own ErrorHandler (conn.go withRefusal/publishRefused), never waited out, so
// this stays the ordinary request bound and is not exposed for a test to shrink.
const stateTimeout = 30 * time.Second
// State is one of this process's module's states, read and written on this connection (novox/hq ADR 0436
// clause 4), in the bucket <module>_<name> (mesh-controller internal/broker/state.go:49) — only as the
// account's grants allow. It satisfies go/asker's Store.
type State struct {
Module string
Name string
// kv binds this state's underlying key-value store: a live connection's own (Conn.State), or, in a
// unit test, a fake that never reaches a bus.
kv func(ctx context.Context) (jetstream.KeyValue, error)
}
// bucket is this state's name on the bus.
func (s State) bucket() string { return s.Module + "_" + s.Name }
// State is the named state this connection's module keeps, bound lazily to its bucket on each call so a
// bucket the account is not granted fails at the call that needed it, naming the bucket. The bind itself
// (js.KeyValue) is what asks $JS.API.STREAM.INFO.KV_<bucket>, so it is what withRefusal watches: a bucket
// not granted is refused there, promptly, as nats.ErrPermissionViolation.
func (c *Conn) State(name string) State {
bucket := c.account.Module + "_" + name
return State{Module: c.account.Module, Name: name, kv: func(ctx context.Context) (jetstream.KeyValue, error) {
js, err := jetstream.New(c.nc)
if err != nil {
return nil, err
}
subject := "$JS.API.STREAM.INFO.KV_" + bucket
ctx, stop := c.withRefusal(ctx, subject)
defer stop()
kv, err := js.KeyValue(ctx, bucket)
if err != nil {
if errors.Is(context.Cause(ctx), errPermissionRefused) {
return nil, fmt.Errorf("%w: publishing %s", nats.ErrPermissionViolation, subject)
}
return nil, err
}
return kv, nil
}}
}
func (s State) store(ctx context.Context) (jetstream.KeyValue, error) {
kv, err := s.kv(ctx)
if err != nil {
return nil, fmt.Errorf("%s's state %s (bucket %s): %w", s.Module, s.Name, s.bucket(), err)
}
return kv, nil
}
// Get is a key's current value, or nil when it has none.
func (s State) Get(key string) (*stdio.StateEntry, error) {
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
kv, err := s.store(ctx)
if err != nil {
return nil, err
}
entry, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("%s's state %s, key %s: %w", s.Module, s.Name, key, err)
}
return &stdio.StateEntry{Key: key, Value: json.RawMessage(entry.Value()), Revision: entry.Revision()}, nil
}
// Create writes a key only when it has none, and returns the revision; a key that has one is refused as
// stdio.ErrChanged.
func (s State) Create(key string, value any) (uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
kv, err := s.store(ctx)
if err != nil {
return 0, err
}
data, err := json.Marshal(value)
if err != nil {
return 0, err
}
rev, err := kv.Create(ctx, key, data)
if errors.Is(err, jetstream.ErrKeyExists) {
return 0, fmt.Errorf("%s: %w", key, stdio.ErrChanged)
}
if err != nil {
return 0, fmt.Errorf("%s's state %s, key %s: %w", s.Module, s.Name, key, err)
}
return rev, nil
}
// Update writes a key only when it is still at revision, and returns the new one; a key changed meanwhile
// is refused as stdio.ErrChanged.
func (s State) Update(key string, value any, revision uint64) (uint64, error) {
ctx, cancel := context.WithTimeout(context.Background(), stateTimeout)
defer cancel()
kv, err := s.store(ctx)
if err != nil {
return 0, err
}
data, err := json.Marshal(value)
if err != nil {
return 0, err
}
rev, err := kv.Update(ctx, key, data, revision)
if errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
return 0, fmt.Errorf("%s: %w", key, stdio.ErrChanged)
}
if err != nil {
return 0, fmt.Errorf("%s's state %s, key %s: %w", s.Module, s.Name, key, err)
}
return rev, nil
}
+96
View File
@@ -0,0 +1,96 @@
//go:build integration
package process
import (
"errors"
"strings"
"testing"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// 1188/AC1: given a process connection to a test bus granted the bucket mesh-mail_sends, when it creates a
// key, reads it, and updates it with the revision it read, then each succeeds and returns the new
// revision; an update with a stale revision fails as stdio.ErrChanged; a create of an existing key fails
// as stdio.ErrChanged.
func TestStateCreateGetUpdateOnAGrantedBucket(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_sends"}); err != nil {
t.Fatalf("the admin could not make the bucket: %v", err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
s := conn.State("sends")
rev, err := s.Create("key", map[string]any{"v": 1})
if err != nil {
t.Fatalf("Create on a granted bucket: %v", err)
}
if rev == 0 {
t.Fatal("Create returned no revision")
}
e, err := s.Get("key")
if err != nil || e == nil {
t.Fatalf("Get after Create: %+v, %v", e, err)
}
if e.Revision != rev || string(e.Value) != `{"v":1}` {
t.Fatalf("Get read back %+v, want revision %d and {\"v\":1}", e, rev)
}
rev2, err := s.Update("key", map[string]any{"v": 2}, rev)
if err != nil {
t.Fatalf("Update at the revision read: %v", err)
}
if rev2 == rev {
t.Fatal("Update did not return a new revision")
}
if _, err := s.Update("key", map[string]any{"v": 3}, rev); !errors.Is(err, stdio.ErrChanged) {
t.Fatalf("Update at a stale revision: %v, want stdio.ErrChanged", err)
}
if _, err := s.Create("key", map[string]any{"v": 4}); !errors.Is(err, stdio.ErrChanged) {
t.Fatalf("Create of an existing key: %v, want stdio.ErrChanged", err)
}
}
// 1188/AC4: given an account not granted a bucket, when it writes there, then the bus's refusal comes
// back as an error naming the bucket. The bucket exists (made by the admin) so the only reason Create
// can fail is the missing grant, never "bucket not found"; and Conn's own ErrorHandler (conn.go
// withRefusal/publishRefused) catches the denied publish at once, so this returns promptly rather than
// after a timeout.
func TestStateRefusesAnUngrantedBucketNamingIt(t *testing.T) {
url, account := grantedSendProcess(t)
_, adminJetStream := adminJS(t, url)
if _, err := adminJetStream.CreateKeyValue(ctxFor(t), jetstream.KeyValueConfig{Bucket: "mesh-mail_unrelated"}); err != nil {
t.Fatalf("the admin could not make the bucket: %v", err)
}
conn, err := Connect(account)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
s := conn.State("unrelated")
_, err = s.Create("key", map[string]any{"v": 1})
if err == nil {
t.Fatal("writing to an ungranted bucket was not refused")
}
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("the refusal %v does not wrap nats.ErrPermissionViolation", err)
}
if !strings.Contains(err.Error(), "mesh-mail_unrelated") {
t.Errorf("the refusal %q does not name the bucket mesh-mail_unrelated", err.Error())
}
}
+105
View File
@@ -0,0 +1,105 @@
package process
import (
"context"
"errors"
"sync"
"testing"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/asker"
)
// fakeEntry is a fakeKV's KeyValueEntry: only Key, Value and Revision are read by State.
type fakeEntry struct {
jetstream.KeyValueEntry
key string
value []byte
rev uint64
}
func (e fakeEntry) Key() string { return e.key }
func (e fakeEntry) Value() []byte { return e.value }
func (e fakeEntry) Revision() uint64 { return e.rev }
// fakeKV is an in-memory jetstream.KeyValue: embedding the nil interface satisfies every method this test
// never calls, so only Get, Create and Update need a body (1188/AC2's "a fake KV").
type fakeKV struct {
jetstream.KeyValue
mu sync.Mutex
values map[string][]byte
revs map[string]uint64
next uint64
}
func newFakeKV() *fakeKV { return &fakeKV{values: map[string][]byte{}, revs: map[string]uint64{}} }
func (f *fakeKV) Get(_ context.Context, key string) (jetstream.KeyValueEntry, error) {
f.mu.Lock()
defer f.mu.Unlock()
v, ok := f.values[key]
if !ok {
return nil, jetstream.ErrKeyNotFound
}
return fakeEntry{key: key, value: v, rev: f.revs[key]}, nil
}
func (f *fakeKV) Create(_ context.Context, key string, value []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
f.mu.Lock()
defer f.mu.Unlock()
if _, ok := f.values[key]; ok {
return 0, jetstream.ErrKeyExists
}
f.next++
f.values[key], f.revs[key] = value, f.next
return f.next, nil
}
func (f *fakeKV) Update(_ context.Context, key string, value []byte, revision uint64) (uint64, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.revs[key] != revision {
return 0, jetstream.ErrKeyRevisionMismatch
}
f.next++
f.values[key], f.revs[key] = value, f.next
return f.next, nil
}
func stateOn(kv jetstream.KeyValue) State {
return State{Module: "mesh-mail", Name: "sends", kv: func(context.Context) (jetstream.KeyValue, error) { return kv, nil }}
}
// 1188/AC2: given the process connection's state type, when used as go/asker's Store, then it compiles and
// behaves as the interface says, against a fake KV.
func TestStateSatisfiesAskerStore(t *testing.T) {
var _ asker.Store = State{}
s := stateOn(newFakeKV())
if e, err := s.Get("ask.1"); err != nil || e != nil {
t.Fatalf("Get of an absent key: %v, %v", e, err)
}
rev, err := s.Create("ask.1", map[string]any{"a": 1})
if err != nil {
t.Fatalf("Create: %v", err)
}
e, err := s.Get("ask.1")
if err != nil || e == nil || e.Revision != rev {
t.Fatalf("Get after Create: %+v, %v", e, err)
}
if _, err := s.Create("ask.1", map[string]any{"a": 2}); !errors.Is(err, stdio.ErrChanged) {
t.Fatalf("Create of an existing key: %v, want stdio.ErrChanged", err)
}
rev2, err := s.Update("ask.1", map[string]any{"a": 3}, rev)
if err != nil {
t.Fatalf("Update at the revision read: %v", err)
}
if _, err := s.Update("ask.1", map[string]any{"a": 4}, rev); !errors.Is(err, stdio.ErrChanged) {
t.Fatalf("Update at a stale revision: %v, want stdio.ErrChanged", err)
}
if rev2 == rev {
t.Fatal("Update did not advance the revision")
}
}
+72
View File
@@ -0,0 +1,72 @@
package provider
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
)
// The provider's consumers' credentials, from the file its machine's provisioner client fills (novox/hq ADR 0373,
// issue 679).
//
// **The provider never asks the vault.** A provider in the tool runner has no account of its own, and the vault
// hands a secret only to a node's provisioner client or to a module under an account of its own (ADR 0349 clause
// 11, ADR 0370 clause 2). So the provider's template names `${consumers:<provision>}`, and its machine's client
// fills that file with one entry per consumer bound to it for that provision, each credential asked of the vault in
// the client's own name; the file is the provider's alone (mode 0600, owned). ConsumersFile is that file as the
// helper's Vault: it reads the file on each ask, so a consumer that binds or a credential the vault remakes reaches
// the provider at its next pass.
// ConsumerCredential is one entry of the file: the consumer, the vault's name of its credential, and its value.
type ConsumerCredential struct {
// Consumer and Module are the consumer's machine and module, Local its local name for the provision, if any.
Consumer string `json:"consumer"`
Module string `json:"module"`
Local string `json:"local,omitempty"`
// Credential is the vault's name of the consumer's credential, PairName's.
Credential string `json:"credential"`
// Value is the credential itself.
Value string `json:"value"`
}
// ConsumersFile answers a credential from the files the provider machine's client fills: one per provision whose
// template names `${consumers:<provision>}`, each a JSON list of ConsumerCredential.
type ConsumersFile struct {
Paths []string
}
// Value is the credential named name, from the first file carrying it. A file not there yet is passed over; one not
// in the client's form is an error, never read as no consumers.
func (f ConsumersFile) Value(_ context.Context, name string) ([]byte, error) {
var missing []string
for _, at := range f.Paths {
raw, err := os.ReadFile(at)
if errors.Is(err, fs.ErrNotExist) {
missing = append(missing, at)
continue
}
if err != nil {
return nil, fmt.Errorf("the consumers' file %s: %w", at, err)
}
var entries []ConsumerCredential
if err := json.Unmarshal(raw, &entries); err != nil {
return nil, fmt.Errorf("the consumers' file %s is not the provisioner client's list of consumers", at)
}
for _, e := range entries {
if e.Credential == name {
if e.Value == "" {
return nil, fmt.Errorf("the consumers' file %s carries %s with no value", at, name)
}
return []byte(e.Value), nil
}
}
}
if len(missing) == len(f.Paths) {
return nil, fmt.Errorf("%s: no consumers' file is filled yet (%v)", name, f.Paths)
}
return nil, fmt.Errorf("%s is in no consumers' file: the vault holds none for it yet, or its machine's client "+
"has not filled it", name)
}
+69
View File
@@ -0,0 +1,69 @@
package provider
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
)
// The provider reads its consumers' credentials only from the file its machine's provisioner client fills (novox/hq
// ADR 0373 clause 3, issue 679): one entry per consumer, naming the consumer and the vault's name for its credential,
// and carrying the value. The helper is wired to it as its Vault and asks nothing of the vault itself.
func TestTheConsumersFileAnswersEachConsumersCredential(t *testing.T) {
dir := t.TempDir()
at := filepath.Join(dir, "consumers.json")
file := `[{"consumer":"ace","module":"letta","credential":"pair/novox/postgres-database/ace/letta","value":"s3cret"},
{"consumer":"g14","module":"gitea","local":"cache","credential":"pair/novox/postgres-database/g14/gitea/cache","value":"other"}]`
if err := os.WriteFile(at, []byte(file), 0o600); err != nil {
t.Fatal(err)
}
v := ConsumersFile{Paths: []string{filepath.Join(dir, "absent.json"), at}}
got, err := v.Value(context.Background(), PairName("novox", "postgres-database", "ace", "letta", ""))
if err != nil || string(got) != "s3cret" {
t.Fatalf("ace's letta: %q, %v; want s3cret", got, err)
}
got, err = v.Value(context.Background(), PairName("novox", "postgres-database", "g14", "gitea", "cache"))
if err != nil || string(got) != "other" {
t.Fatalf("g14's gitea under cache: %q, %v; want other", got, err)
}
// A consumer the file does not carry is waited for, and never told of a login its provider has not made.
_, err = v.Value(context.Background(), PairName("novox", "postgres-database", "shanks", "photos", ""))
if err == nil || !strings.Contains(err.Error(), "pair/novox/postgres-database/shanks/photos") {
t.Fatalf("a consumer the file does not carry: %v", err)
}
if strings.Contains(err.Error(), "s3cret") || strings.Contains(err.Error(), "other") {
t.Fatalf("an error names a credential: %v", err)
}
// Wired as the helper's Vault, the file is what makes each consumer's login.
bus := newStandIn()
bus.Put(context.Background(), AssignmentsBucket, "ace.letta", []byte(`{"node":"ace","module":"letta",
"provisioning":{"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres"}]}}`))
var made []string
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v,
Offers: map[string]Offer{"postgres-database": {Serves: map[string]any{"port": 5432}}},
Make: func(_ context.Context, c Consumer) error {
made = append(made, c.Entry.User+"="+string(c.Credential))
return nil
}, Logf: func(string, ...any) {}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
if len(made) != 1 || made[0] != "mesh_ace_letta=s3cret" {
t.Fatalf("made %v, want ace's letta under the file's credential", made)
}
}
// A file that is not the client's form is said, never read as no consumers.
func TestAConsumersFileNotInItsFormIsRefused(t *testing.T) {
at := filepath.Join(t.TempDir(), "consumers.json")
if err := os.WriteFile(at, []byte(`{"not": "a list"}`), 0o600); err != nil {
t.Fatal(err)
}
_, err := ConsumersFile{Paths: []string{at}}.Value(context.Background(), "pair/novox/p/ace/m")
if err == nil || !strings.Contains(err.Error(), at) {
t.Fatalf("a file not in the client's form: %v", err)
}
}
+5
View File
@@ -0,0 +1,5 @@
package harness
import "context"
var ctx = context.Background()
+62
View File
@@ -0,0 +1,62 @@
package harness
// The moved path started before its runtime answers asks again until it does (issue 943).
import (
"context"
"errors"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// earlyBus refuses its first watches as the runtime does before Serve runs, then answers as the bucket bus does.
type earlyBus struct {
*bucketBus
refuse int
}
func (b *earlyBus) Watch(ctx context.Context, bucket string, handle func(string, []byte)) error {
if b.refuse > 0 {
b.refuse--
return errors.New("before Serve: a bundle reaches the bus through the runtime that launched it")
}
return b.bucketBus.Watch(ctx, bucket, handle)
}
func TestAMovedPathStartedTooEarlyAsksAgainUntilTheRuntimeAnswers(t *testing.T) {
w := newMovedWorld(t, map[[2]string]string{})
w.run.FirstRetry = time.Millisecond
w.run.loaded = false
bus := &earlyBus{bucketBus: w.bus, refuse: 3}
p := provider.New(provider.Options{Node: "novox", Module: "postgres", Bus: w.bus,
Vault: provider.ConsumersFile{Paths: []string{w.file}}, Make: w.run.Make, Gone: w.run.Gone,
Every: time.Hour, Logf: func(string, ...any) {}})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- w.run.Run(ctx, p, bus) }()
deadline := time.After(time.Second)
for {
w.run.mu.Lock()
loaded := w.run.loaded
w.run.mu.Unlock()
if loaded {
break
}
select {
case err := <-done:
cancel()
t.Fatalf("Run stopped before the runtime answered: %v", err)
case <-deadline:
cancel()
t.Fatal("the plans were never watched")
case <-time.After(time.Millisecond):
}
}
cancel()
<-done
if bus.refuse != 0 {
t.Fatalf("%d refusals left", bus.refuse)
}
}
+123
View File
@@ -0,0 +1,123 @@
package harness
// postgres handed to the provisioner still serves the consumers the controller's grant file gives (issue 937): a
// consumer not moved has no plan naming postgres, only its grant, until R1 (issue 682).
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"testing"
"time"
)
// granting gives the moved world a grant file, as the controller writes it for the consumers not moved.
func (w *movedWorld) granting(given ...map[string]any) {
if w.h.Receives == "" {
w.h.Receives = filepath.Join(w.t.TempDir(), "mesh.json")
}
dir := filepath.Dir(w.h.Receives)
for _, g := range given {
secret := filepath.Join(dir, g["as"].(string)+".secret")
if err := os.WriteFile(secret, []byte("pw-"+g["as"].(string)+"\n"), 0o600); err != nil {
w.t.Fatal(err)
}
g["secret"] = secret
}
if given == nil {
given = []map[string]any{}
}
raw, _ := json.Marshal(map[string]any{"requirement": ProvisionName, "given": given})
if err := os.WriteFile(w.h.Receives, raw, 0o600); err != nil {
w.t.Fatal(err)
}
}
// settled is enough passes and time for a retirement to be stable.
func (w *movedWorld) settled() {
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
w.pass()
}
// The logins the grant file gives are asked for: made as the grant path makes them, never offered for retirement,
// beside a moved consumer the helper makes from its plan.
func TestAConsumerOnlyTheGrantFileGivesIsAskedForAndMade(t *testing.T) {
bound, logins := bindings(1)
w := newMovedWorld(t, bound)
w.a.inv.Active = []string{"mesh_ace_baserow", "mesh_ace_letta", logins[0]}
w.bus.bind("ace", "app1", nil)
w.granting(
map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}},
map[string]any{"as": "mesh_ace_letta", "node": "ace",
"values": map[string]any{"name": "letta", "extensions": []any{"vector"}}},
)
w.settled()
if len(w.a.removed) != 0 || w.h.r.waiting != nil {
t.Fatalf("retired %v, waiting %+v: the grant file's logins were counted unasked", w.a.removed, w.h.r.waiting)
}
made := map[string]Provision{}
for _, c := range w.a.created {
made[c.As] = c
}
letta := made["mesh_ace_letta"]
if letta.Password != "pw-mesh_ace_letta" || letta.Consumer != "ace" ||
!reflect.DeepEqual(letta.Values, map[string]any{"name": "letta", "extensions": []any{"vector"}}) {
t.Fatalf("made %+v for letta, as its grant gives it", letta)
}
if _, ok := made["mesh_ace_baserow"]; !ok {
t.Fatalf("baserow not made: %+v", w.a.created)
}
if _, ok := made[logins[0]]; !ok {
t.Fatalf("the moved consumer not made: %+v", w.a.created)
}
}
// A login both a plan and the grant file name is the plan's: made once, with the plan's credential.
func TestALoginAPlanNamesIsNotMadeFromTheGrantFile(t *testing.T) {
bound, logins := bindings(1)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.granting(map[string]any{"as": logins[0], "node": "ace", "values": map[string]any{"name": "app1"}})
w.pass()
w.pass()
for _, c := range w.a.created {
if c.As == logins[0] && c.Password != "pw-app1" {
t.Fatalf("made %s with the grant's password %q, not the plan's", c.As, c.Password)
}
}
}
// An unreadable grant file is not nobody asking (issue 241): nothing is retired while it cannot be read.
func TestAnUnreadableGrantFileRetiresNothing(t *testing.T) {
bound, logins := bindings(1)
w := newMovedWorld(t, bound)
w.a.inv.Active = []string{"mesh_ace_baserow", logins[0]}
w.bus.bind("ace", "app1", nil)
w.h.Receives = filepath.Join(t.TempDir(), "absent.json")
w.settled()
if len(w.a.removed) != 0 || w.h.r.waiting != nil {
t.Fatalf("retired %v, waiting %+v with the grant file unreadable", w.a.removed, w.h.r.waiting)
}
}
// A login the grant file stops giving is retired by the usual rules: stable, then retired, never dropped.
func TestALoginTheGrantFileStopsGivingIsRetired(t *testing.T) {
bound, _ := bindings(1)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.granting(
map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}},
map[string]any{"as": "mesh_ace_letta", "node": "ace", "values": map[string]any{"name": "letta"}},
)
w.pass()
w.granting(map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}})
w.settled()
if !reflect.DeepEqual(w.a.removed, []string{"mesh_ace_letta"}) || len(w.a.deleted) != 0 {
t.Fatalf("retired %v, deleted %v", w.a.removed, w.a.deleted)
}
}
+572
View File
@@ -0,0 +1,572 @@
// Package harness is the reconcile loop and the retirement of a provider that keeps a resource per consumer; the
// module supplies an Adapter.
package harness
// The reconcile loop every provider shares, as the TypeScript SDK's runProvisioner runs it
// (@novox/mesh-sdk/provisioner, 0.1.12), line for line in behaviour. It moved here from the postgres and keycloak
// providers, which each carried an identical copy (novox/hq ADR 0039: the loop is the SDK's, the adapter is the
// module's; issue 849).
//
// Read the contributions the mesh delivered; bring each consumer's resource into being through the
// adapter, under the login and password the mesh minted; retire what the mesh no longer asks for.
// **A provider creates the credential the mesh minted, and seals nothing (novox/hq ADR 0048).**
//
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
// whose create, check or secret has failed without one success in between for FailingAfter is
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
// again every SayAgainEvery while it lasts; the first success after that is `provisioner.recovered`.
// The controller keeps the newest per provider and consumer and `status` names it. On 2026-10-05 the
// identity provider failed every consumer 31,000 times in a day and said so only in its journal
// (novox/hq issue 179).
//
// **A consumer the mesh stops asking for is retired, not withdrawn, and deleted only by a person
// (novox/hq ADR 0230, the operator's model of 2026-10-06).** Retiring disables its access — reversibly —
// and marks its login and data "to delete" with when and why; nothing is deleted. Asked for again, the
// ordinary create re-enables it as it was. It is retired only once the same set has gone unasked in
// StablePasses consecutive passes and for StableFor, and a set larger than the bound waits for a
// person. retirement.go.
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
"strings"
"sync"
"time"
)
// Provision is one consumer's resource to bring into being — everything the mesh derived and delivered.
type Provision struct {
// As is the login the mesh derived and gave the consumer to present.
As string
// Password is the one the mesh minted, read from the file the host unsealed.
Password string
// Values are what the consumer contributed (e.g. {"name": "letta", "extensions": ["vector"]}).
Values map[string]any
// Derived is what this provider's own definition derives for the consumer (novox/hq ADR 0201).
Derived map[string]any
// At is where the consumer is; Consumer is its node.
At string
Consumer string
}
// Adapter is the per-service half.
type Adapter interface {
// Create brings the consumer into being under the mesh's login and password — and, for one that
// was retired, re-enables it as it was and clears its mark to delete.
Create(ctx context.Context, p Provision) error
// Retire disables the consumer's access, reversibly, and marks its login and data to delete with
// when and why. It deletes nothing (novox/hq ADR 0230). derived is what the mesh last derived,
// remembered here; nil for a consumer this process did not make.
Retire(ctx context.Context, as string, derived map[string]any, why string, at time.Time) error
// Holds says whether the backend still holds the consumer exactly as p says. Read-only.
Holds(ctx context.Context, p Provision) (bool, error)
// Inventory is what the backend holds that the mesh made: consumers active and retired, read from
// the backend itself so a restart forgets nothing. Read-only.
Inventory(ctx context.Context) (Inventory, error)
// Delete removes one retired consumer — its login and its data — for good. Only ever asked by a
// person, through `cleanup delete`; the harness has checked it is retired and not asked for.
Delete(ctx context.Context, r Retired) (freedBytes int64, err error)
}
// Harness is the loop's settings and memory.
type Harness struct {
Resource string
Receives string
Adapter Adapter
Every time.Duration // 5s
VerifyEvery time.Duration // 60s
HoldsTimeout time.Duration // 30s
Log func(format string, args ...any)
Now func() time.Time
// Announce publishes one of the provider's standing events; nil announces nothing. Node is the
// machine this provider runs on, said in each.
Announce func(event string, body map[string]any)
Node string
// FailingAfter is how long a consumer fails without a success before it is announced (5m);
// SayAgainEvery is how often it is announced again while it lasts (15m), so a controller that
// missed the first hears the next, and a standing nobody repeats can be told from one that holds.
FailingAfter time.Duration
SayAgainEvery time.Duration
// StablePasses (5) is how many consecutive passes must see the same set no longer asked for, and
// StableFor (10m) how long it must have held since the first of them, before it is retired. RetireAtOnce (3) and RetireFraction (0.5) bound what is retired without a person:
// more consumers than RetireAtOnce, or — where more than one is held — a larger share of those held
// than RetireFraction, waits for `retire approve` (novox/hq ADR 0230).
StablePasses int
StableFor time.Duration
RetireAtOnce int
RetireFraction float64
// mu is held by a pass and by every tool a person asks, so the two never interleave.
mu sync.Mutex
verifiedAt time.Time
r retirement
applied map[string]appliedEntry
lost map[string]brake
waiting map[string]int
failing map[string]failure
trouble map[string]*standing
cleared map[string]bool
lastWarning string
}
// standing is one consumer's unbroken run of failures: since when, how often, and the last error.
type standing struct {
node string
since time.Time
attempts int
class string
text string
saidAt time.Time
}
// The events a provider's standing is announced as (novox/hq ADR 0224). The controller derives the
// permission to emit them for every module that receives contributions; no manifest lists them.
const (
EventFailing = "provisioner.failing"
EventRecovered = "provisioner.recovered"
)
// The classes of error a standing is announced with: what a person reading `status` needs to know
// before reading the journal. An adapter may say better (Classifier).
const (
ClassCredentials = "credentials-rejected"
ClassUnreachable = "unreachable"
ClassSecret = "secret-unreadable"
ClassRefused = "refused"
)
// Classifier is an adapter that can say what class an error of its own is.
type Classifier interface {
Class(err error) string
}
type appliedEntry struct {
hash string
derived map[string]any
node string
}
type brake struct {
times int
nextAt time.Time
}
type failure struct {
text string
times int
}
// The longest a consumer whose create keeps failing to satisfy holds waits between checks.
const maxBackoff = time.Hour
// How many passes a secret may be unreadable before it stops being called a race (issue 225), and
// once said loudly, how often it is repeated. The same cadence quiets a create that keeps failing
// the same way.
const (
patiently = 12
loudlyEvery = 240
)
type contribution struct {
As string `json:"as"`
Secret string `json:"secret"`
Node string `json:"node"`
At string `json:"at"`
Values map[string]any `json:"values"`
Derived map[string]any `json:"derived"`
}
func (h *Harness) init() {
if h.Every == 0 {
h.Every = 5 * time.Second
}
if h.VerifyEvery == 0 {
h.VerifyEvery = time.Minute
}
if h.HoldsTimeout == 0 {
h.HoldsTimeout = 30 * time.Second
}
if h.Now == nil {
h.Now = time.Now
}
if h.FailingAfter == 0 {
h.FailingAfter = 5 * time.Minute
}
if h.SayAgainEvery == 0 {
h.SayAgainEvery = 15 * time.Minute
}
if h.StablePasses == 0 {
h.StablePasses = 5
}
if h.StableFor == 0 {
h.StableFor = StableFor
}
if h.RetireAtOnce == 0 {
h.RetireAtOnce = 3
}
if h.RetireFraction == 0 {
h.RetireFraction = 0.5
}
if h.Log == nil {
h.Log = func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) }
}
if h.applied == nil {
h.applied = map[string]appliedEntry{}
h.lost = map[string]brake{}
h.waiting = map[string]int{}
h.failing = map[string]failure{}
h.trouble = map[string]*standing{}
h.cleared = map[string]bool{}
h.r.retired = map[string]retiredEntry{}
}
}
// Run reconciles until ctx ends. One consumer's failure never stops the others'.
func (h *Harness) Run(ctx context.Context) {
h.init()
for {
h.Reconcile(ctx)
select {
case <-ctx.Done():
return
case <-time.After(h.Every):
}
}
}
func (h *Harness) say(format string, args ...any) {
h.Log("[provisioner:"+h.Resource+"] "+format, args...)
}
func (h *Harness) warn(why string) {
if why == h.lastWarning {
return
}
if why != "" {
h.say("%s; nothing applied or removed until it can be read", why)
} else {
h.say("contributions file readable again")
}
h.lastWarning = why
}
// readContributions answers the consumers asked for, or nil when the file says nothing usable.
// **Nothing read is not nobody asking** (novox/hq issue 241): only a file that was read can retire anything.
func (h *Harness) readContributions() []contribution {
raw, err := os.ReadFile(h.Receives)
if err != nil {
h.warn(fmt.Sprintf("contributions file unreadable (%s): %v", h.Receives, err))
return nil
}
var doc struct {
Requirement string `json:"requirement"`
Given json.RawMessage `json:"given"`
}
if err := json.Unmarshal(raw, &doc); err != nil {
h.warn(fmt.Sprintf("contributions file is not JSON (%s): %v", h.Receives, err))
return nil
}
if doc.Requirement != "" && doc.Requirement != h.Resource {
h.warn(fmt.Sprintf("%s is for %s, not %s", h.Receives, doc.Requirement, h.Resource))
return nil
}
var given []contribution
if len(doc.Given) == 0 || string(doc.Given) == "null" || json.Unmarshal(doc.Given, &given) != nil {
h.warn(fmt.Sprintf("%s has no given list", h.Receives))
return nil
}
h.warn("")
out := []contribution{}
for _, g := range given {
// No `as` is not a credential grant: nothing to create for it.
if g.As != "" && g.Secret != "" {
out = append(out, g)
}
}
return out
}
func hashOf(as, password string, values, derived map[string]any) string {
// Derived is in the hash: a provider that renames what it derives gave a different resource.
b, _ := json.Marshal([]any{as, password, orEmpty(values), orEmpty(derived)})
return string(b)
}
func orEmpty(m map[string]any) map[string]any {
if m == nil {
return map[string]any{}
}
return m
}
// Reconcile is one pass.
func (h *Harness) Reconcile(ctx context.Context) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
given := h.readContributions()
if given == nil {
// Not a result: the passes that must agree before anything is retired start again.
h.r.key, h.r.count = "", 0
return
}
want := map[string]bool{}
for _, g := range given {
want[g.As] = true
}
h.seed(ctx, want)
h.apply(ctx, given)
// Retire what the mesh has stably stopped asking for — within the bound, or with a person.
h.retireUnasked(ctx, want)
h.r.lastWant = want
for as := range h.failing {
if !want[as] {
delete(h.failing, as)
}
}
// A consumer the mesh stopped asking for that this provider holds nothing for is no longer failed by
// anyone: said, so a standing the controller keeps for it is cleared rather than left naming a
// consumer that is gone. One still held is said recovered when it is retired.
for as := range h.trouble {
if _, held := h.applied[as]; !want[as] && !held {
h.recovered(as, "no longer asked for")
}
}
}
// apply makes each consumer the grant file gives, and checks every VerifyEvery that the backend still holds it. The
// caller holds h.mu. One consumer's failure never stops the others'.
func (h *Harness) apply(ctx context.Context, given []contribution) {
verifying := h.Now().Sub(h.verifiedAt) >= h.VerifyEvery
if verifying {
h.verifiedAt = h.Now()
}
for _, g := range given {
raw, err := os.ReadFile(g.Secret)
if err != nil {
// A secret the host has not written yet is a race on the first pass; past a minute it is
// a person's to look at, and said so (novox/hq issue 225).
n := h.waiting[g.As] + 1
h.waiting[g.As] = n
if n <= patiently {
h.say("%s: secret not readable yet (%s): %v", g.As, g.Secret, err)
} else if n == patiently+1 || n%loudlyEvery == 0 {
h.say("%s: CANNOT READ the secret after %d attempts (%s): %v. This is not a race any more — "+
"nothing has been provisioned for this consumer and nothing will be until somebody looks. "+
"Check who owns the file and who this process runs as (novox/hq issue 225)", g.As, n, g.Secret, err)
}
h.failed(g.As, g.Node, ClassSecret, fmt.Sprintf("secret not readable (%s): %v", g.Secret, err))
continue
}
delete(h.waiting, g.As)
password := strings.TrimSuffix(string(raw), "\n")
p := Provision{As: g.As, Password: password, Values: orEmpty(g.Values), Derived: orEmpty(g.Derived), At: g.At, Consumer: g.Node}
hash := hashOf(g.As, password, g.Values, g.Derived)
reapplying := 0
if was, ok := h.applied[g.As]; ok && was.hash == hash {
if !verifying {
continue
}
b, braked := h.lost[g.As]
if braked && h.Now().Before(b.nextAt) {
continue
}
hctx, cancel := context.WithTimeout(ctx, h.HoldsTimeout)
held, err := h.Adapter.Holds(hctx, p)
timedOut := errors.Is(hctx.Err(), context.DeadlineExceeded)
cancel()
if err != nil {
// Unable to ask is not evidence of loss. A backend that timed out will time out for
// the next consumer too, so the rest of this pass is not asked.
text := scrub(err, password)
h.say("%s: could not check the backend, will ask again: %s", g.As, text)
h.failed(g.As, g.Node, h.classOf(err, text), text)
if timedOut {
verifying = false
}
continue
}
if held {
delete(h.lost, g.As)
h.succeeded(g.As)
continue
}
reapplying = b.times + 1
if reapplying == 1 {
h.say("%s: the backend no longer holds it; applying again", g.As)
} else {
h.say("%s: still not held after being applied again (%d times in a row) — create does not "+
"produce what holds checks; applying again", g.As, reapplying)
}
}
if err := h.Adapter.Create(ctx, p); err != nil {
text := scrub(err, password)
f := h.failing[g.As]
if f.text != text {
f = failure{text: text}
}
f.times++
h.failing[g.As] = f
// Said each time it changes, and while it stays the same, as rarely as a lost secret.
if f.times == 1 || f.times%loudlyEvery == 0 {
h.say("%s: create failed, will retry: %s", g.As, text)
}
h.failed(g.As, g.Node, h.classOf(err, text), text)
if reapplying > 0 {
h.lost[g.As] = brake{times: reapplying - 1}
}
continue
}
if f, was := h.failing[g.As]; was {
h.say("%s: created, after %d failed attempt(s)", g.As, f.times)
delete(h.failing, g.As)
}
h.succeeded(g.As)
h.applied[g.As] = appliedEntry{hash: hash, derived: p.Derived, node: g.Node}
h.reenabled(g.As, g.Node)
if reapplying == 0 {
delete(h.lost, g.As)
} else {
wait := h.VerifyEvery << (reapplying - 1)
if wait > maxBackoff || wait <= 0 {
wait = maxBackoff
}
h.lost[g.As] = brake{times: reapplying, nextAt: h.Now().Add(wait)}
if reapplying > 1 {
h.say("%s: next check in %s", g.As, wait.Round(time.Second))
}
}
}
}
// failed counts one more failure in a consumer's unbroken run, and announces the run once it has
// lasted FailingAfter — then again every SayAgainEvery while it lasts.
func (h *Harness) failed(as, node, class, text string) {
now := h.Now()
s := h.trouble[as]
if s == nil {
s = &standing{since: now}
h.trouble[as] = s
}
s.node, s.class, s.text = node, class, text
s.attempts++
if now.Sub(s.since) < h.FailingAfter {
return
}
if !s.saidAt.IsZero() && now.Sub(s.saidAt) < h.SayAgainEvery {
return
}
first := s.saidAt.IsZero()
s.saidAt = now
if first {
h.say("%s: FAILING for %s (%d attempts, %s): %s. Announced as %s; `status` names it until it "+
"succeeds (novox/hq ADR 0224)", as, now.Sub(s.since).Round(time.Second), s.attempts, class, text, EventFailing)
}
h.announce(EventFailing, map[string]any{
"provider": h.Resource, "provider-node": h.Node,
"consumer": as, "node": node,
"class": class, "error": clip(text),
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
})
}
// succeeded ends a consumer's run of failures; one that was announced is announced recovered.
//
// **And the first success for a consumer since this process started is announced too**, failing or
// not: a provider that announced a failure and was restarted has forgotten it, and without this the
// controller would name the consumer failing for ever after it recovered unheard.
func (h *Harness) succeeded(as string) {
if h.trouble[as] == nil && !h.cleared[as] {
h.cleared[as] = true
h.announce(EventRecovered, map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "why": "first-success",
})
return
}
h.cleared[as] = true
h.recovered(as, "")
}
func (h *Harness) recovered(as, why string) {
s := h.trouble[as]
if s == nil {
return
}
delete(h.trouble, as)
if s.saidAt.IsZero() {
return // never announced, so there is nothing to take back
}
if why == "" {
h.say("%s: recovered after %s and %d failed attempt(s)", as, h.Now().Sub(s.since).Round(time.Second), s.attempts)
}
body := map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "node": s.node,
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
}
if why != "" {
body["why"] = why
}
h.announce(EventRecovered, body)
}
func (h *Harness) announce(event string, body map[string]any) {
if h.Announce != nil {
h.Announce(event, body)
}
}
// classOf is an error's class: the adapter's word when it has one, else read from the text.
func (h *Harness) classOf(err error, text string) string {
if c, ok := h.Adapter.(Classifier); ok {
if class := c.Class(err); class != "" {
return class
}
}
return ClassOf(text)
}
// ClassOf reads an error's class from its text — the words the backends the mesh runs use.
func ClassOf(text string) string {
t := strings.ToLower(text)
for _, w := range []string{"invalid_grant", "invalid user credentials", "password authentication failed",
"authentication failed", "unauthorized", " 401"} {
if strings.Contains(t, w) {
return ClassCredentials
}
}
for _, w := range []string{"connection refused", "no such host", "i/o timeout", "deadline exceeded",
"connection reset", "network is unreachable", "no route to host", "eof"} {
if strings.Contains(t, w) {
return ClassUnreachable
}
}
return ClassRefused
}
// clip keeps an announced error to what belongs in a status line.
func clip(text string) string {
const most = 300
if len(text) <= most {
return text
}
return text[:most] + "…"
}
// scrub is an error's text with the consumer's password removed, raw and URL-encoded.
func scrub(err error, password string) string {
text := err.Error()
if password == "" {
return text
}
for _, form := range []string{password, url.QueryEscape(password), url.PathEscape(password)} {
text = strings.ReplaceAll(text, form, "***")
}
return text
}
+154
View File
@@ -0,0 +1,154 @@
package harness
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
type world struct {
t *testing.T
dir string
receives string
now time.Time
h *Harness
a *recorder
said []string
}
func newWorld(t *testing.T) *world {
w := &world{t: t, dir: t.TempDir(), now: time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC), a: &recorder{held: true}}
w.receives = filepath.Join(w.dir, "mesh.json")
w.h = &Harness{Resource: "postgres-database", Receives: w.receives, Adapter: w.a,
Now: func() time.Time { return w.now },
Log: func(f string, a ...any) { w.said = append(w.said, f) }}
return w
}
func (w *world) give(given ...map[string]any) {
for _, g := range given {
secret := filepath.Join(w.dir, g["as"].(string)+".secret")
if err := os.WriteFile(secret, []byte("pw-"+g["as"].(string)+"\n"), 0o600); err != nil {
w.t.Fatal(err)
}
g["secret"] = secret
}
if given == nil {
given = []map[string]any{}
}
raw, _ := json.Marshal(map[string]any{"requirement": "postgres-database", "given": given})
if err := os.WriteFile(w.receives, raw, 0o600); err != nil {
w.t.Fatal(err)
}
}
func TestAConsumerIsCreatedOnceUnderTheMeshsLoginAndPassword(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "mesh_ace_letta", "node": "ace", "values": map[string]any{"name": "letta"}})
w.h.Reconcile(ctx)
w.h.Reconcile(ctx)
if len(w.a.created) != 1 {
t.Fatalf("created %d times", len(w.a.created))
}
p := w.a.created[0]
if p.As != "mesh_ace_letta" || p.Password != "pw-mesh_ace_letta" || p.Consumer != "ace" {
t.Fatalf("%+v", p)
}
}
func TestAConsumerNoLongerAskedForIsRetiredOnceStable(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
w.h.Reconcile(ctx)
w.give(map[string]any{"as": "a"})
w.settle() // five passes and ten minutes
if strings.Join(w.a.removed, ",") != "b" {
t.Fatal(w.a.removed)
}
// Only a file that says nobody asks retires the last one.
w.give()
w.settle()
if strings.Join(w.a.removed, ",") != "b,a" {
t.Fatal(w.a.removed)
}
}
func TestNothingReadIsNotNobodyAsking(t *testing.T) {
for name, content := range map[string]string{
"unreadable": "",
"not JSON": "{",
"no given": `{"requirement": "postgres-database"}`,
"another": `{"requirement": "mssql-database", "given": []}`,
} {
t.Run(name, func(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
if content == "" {
os.Remove(w.receives)
} else {
os.WriteFile(w.receives, []byte(content), 0o600)
}
w.h.Reconcile(ctx)
if len(w.a.removed) != 0 {
t.Fatalf("retired %v on a file it could not use", w.a.removed)
}
})
}
}
func TestALostConsumerIsAppliedAgainAndBraked(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
w.a.held = false
w.now = w.now.Add(2 * time.Minute)
w.h.Reconcile(ctx)
if len(w.a.created) != 2 {
t.Fatalf("created %d times", len(w.a.created))
}
// Still not held a minute later: applied again, then braked for two minutes.
w.now = w.now.Add(61 * time.Second)
w.h.Reconcile(ctx)
w.now = w.now.Add(61 * time.Second)
w.h.Reconcile(ctx)
if len(w.a.created) != 3 {
t.Fatalf("not braked: created %d times", len(w.a.created))
}
}
func TestAFailingCreateIsRetriedAndSaidOnce(t *testing.T) {
w := newWorld(t)
w.a.failing = &pgErr{"extension \"nope\" refused, password pw-a"}
w.give(map[string]any{"as": "a"})
for i := 0; i < 5; i++ {
w.h.Reconcile(ctx)
}
n := 0
for _, s := range w.said {
if strings.Contains(s, "create failed") {
n++
}
}
if n != 1 {
t.Fatalf("said %d times", n)
}
w.a.failing = nil
w.h.Reconcile(ctx)
if len(w.a.created) != 1 {
t.Fatal("not retried")
}
}
type pgErr struct{ s string }
func (e *pgErr) Error() string { return e.s }
func TestScrubRemovesThePassword(t *testing.T) {
if got := scrub(&pgErr{"bad pw a/b c and a%2Fb+c"}, "a/b c"); strings.Contains(got, "a/b c") || strings.Contains(got, "a%2Fb+c") {
t.Fatal(got)
}
}
+339
View File
@@ -0,0 +1,339 @@
package harness
// A provider handed to the provisioner (novox/hq ADR 0373, ADR 0349 clause 12, issues 780 and 805): where its
// consumers file is set, it runs on the provider helper (package provider) instead of the harness reading the
// controller's grant files. Moved is that path, shared by every provider that keeps a resource per consumer; a module
// says only what differs, in a MovedSpec.
//
// The helper learns who binds the provider from the plans, composes each consumer's entry and reads its credential
// through ConsumersFile, the file this machine's provisioner client fills. It leaves the rest to the two hooks
// here: Make brings the consumer's resource into being exactly as the harness's adapter does, and Gone leaves
// retirement to the harness's guards (retirement.go): the resource is disabled and marked to delete, never removed.
//
// **A consumer not moved is still the grant file's** (issue 937, ADR 0349 clause 17), for a provider whose spec says
// WithGrants: only a moved consumer's plan names its provider, so the controller still gives every other consumer
// through the grant file (MESH_RECEIVES) until R1 (issue 682). Each login it gives is asked for, and one no plan
// names is made and checked by the harness's grant pass exactly as on the grant path; a login a plan names is the
// helper's alone. A grant file that cannot be read is not nobody asking (issue 241): nothing is retired until it is
// read again.
import (
"context"
"encoding/json"
"errors"
"fmt"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// MovedSpec is what differs per provider on the moved path.
type MovedSpec struct {
// Module and Provision are the provider module's name and what it provides, as its manifest says.
Module, Provision string
// Offer is what it serves: its credential form and served values.
Offer provider.Offer
// Login is the identity a consumer is made under at the provider's end (postgres: the entry's User; keycloak: its
// As); an entry with none is not made.
Login func(e provider.Entry) string
// Values are the values the adapter is handed for a consumer; the entry's contribution when not set.
Values func(e provider.Entry) map[string]any
// Noun names what Login is in words ("login", "client id"), Made what is made ("login", "client"), and Backend
// the service in "no longer holds it" ("the backend", "Keycloak").
Noun, Made, Backend string
// WithGrants makes the logins the grant file gives, that no plan names, asked for and made by the harness's grant
// pass (issue 937).
WithGrants bool
}
// Moved is a provider on the provider helper, with the harness's own memory and retirement behind it: what is held,
// what is counted unasked, what waits for a person (retirement.go). The harness is never Run here; the helper's Make
// and Gone and the loop below drive the same state, so the guards are the harness's, not a copy.
type Moved struct {
h *Harness
spec MovedSpec
// FirstRetry is the first wait before a watch asked too early is asked again (issue 943); a second when zero.
FirstRetry time.Duration
mu sync.Mutex // guards assignments and loaded
assignments map[string][]byte
loaded bool
verified map[string]time.Time
}
func newMoved(h *Harness, spec MovedSpec) *Moved {
h.mu.Lock()
h.init()
h.mu.Unlock()
return &Moved{h: h, spec: spec, assignments: map[string][]byte{}, verified: map[string]time.Time{}}
}
// Make brings one consumer's resource into being under its login and the vault's credential, as the harness does:
// at once when it is new or changed, and otherwise only when the backend no longer holds it, which is asked every
// VerifyEvery.
func (m *Moved) Make(ctx context.Context, c provider.Consumer) error {
h, e := m.h, c.Entry
as := m.spec.Login(e)
if as == "" {
return fmt.Errorf("%s has no %s to make", e.Key(), m.spec.Noun)
}
if len(c.Credential) == 0 {
return fmt.Errorf("%s has no credential to make its %s with", e.Key(), m.spec.Made)
}
h.mu.Lock()
defer h.mu.Unlock()
password := string(c.Credential)
values := orEmpty(e.Contributes)
if m.spec.Values != nil {
values = m.spec.Values(e)
}
p := Provision{As: as, Password: password, Values: values, Consumer: e.Consumer}
hash := hashOf(as, password, p.Values, nil)
if was, ok := h.applied[as]; ok && was.hash == hash {
if h.Now().Sub(m.verified[as]) < h.VerifyEvery {
return nil
}
m.verified[as] = h.Now()
hctx, cancel := context.WithTimeout(ctx, h.HoldsTimeout)
held, err := h.Adapter.Holds(hctx, p)
cancel()
if err != nil {
text := scrub(err, password)
h.failed(as, e.Consumer, h.classOf(err, text), text)
return errors.New(text)
}
if held {
h.succeeded(as)
return nil
}
h.say("%s: %s no longer holds it; applying again", as, m.spec.Backend)
}
if err := h.Adapter.Create(ctx, p); err != nil {
text := scrub(err, password)
h.failed(as, e.Consumer, h.classOf(err, text), text)
return errors.New(text)
}
h.succeeded(as)
h.applied[as] = appliedEntry{hash: hash, node: e.Consumer}
m.verified[as] = h.Now()
h.reenabled(as, e.Consumer)
return nil
}
// Gone is the helper asking whether it may forget a consumer that no longer binds. **It never retires**: retiring is
// the harness's rule, the same unasked set over StablePasses passes and StableFor, at most RetireAtOnce at a time
// (retirement.go, ADR 0230), run by retire below. So a consumer missing from one read of the plans loses nothing:
// the helper is told no, and asks again at its next pass, until the login is retired, or asked for again.
func (m *Moved) Gone(_ context.Context, e provider.Entry) error {
h := m.h
as := m.spec.Login(e)
if as == "" {
return nil // refused: nothing was made for it
}
// The plans and the grant file as they are now, not as the last retirement pass read them: with either not
// read, nothing is let go.
want, read := m.asked(h.Node)
h.mu.Lock()
defer h.mu.Unlock()
asked := want[as]
if m.spec.WithGrants {
_, granted, ok := m.granted(want)
read = read && ok
asked = asked || granted[as]
}
_, held := h.applied[as]
_, retired := h.r.retired[as]
switch {
case !read:
case asked:
return nil // another holder of the same login still binds: nothing to retire
case retired, !held && h.r.seeded:
return nil
}
return fmt.Errorf("%s is kept until its retirement is stable (%d passes and %s) and within the bound, or a "+
"person approves it", as, h.StablePasses, h.StableFor)
}
// plan is as much of an assignment as is read to learn who binds the provider.
type plan struct {
Node string `json:"node"`
Module string `json:"module"`
Provisioning *struct {
Provisions []struct {
Provision string `json:"provision"`
Provider string `json:"provider"`
ProviderModule string `json:"provider-module"`
// As is who the consumer is at its provider's end, its slug first (issue 788).
As string `json:"as"`
} `json:"provisions"`
} `json:"provisioning"`
}
// asked is the logins the plans bind the provider for, and whether they could all be read: a plan that is not the
// controller's, or no read yet, is not an answer, so nothing is counted unasked from it (novox/hq issue 241).
func (m *Moved) asked(node string) (map[string]bool, bool) {
m.mu.Lock()
defer m.mu.Unlock()
if !m.loaded || len(m.assignments) == 0 {
return nil, false
}
want := map[string]bool{}
for key, raw := range m.assignments {
var a plan
if err := json.Unmarshal(raw, &a); err != nil || a.Node+"."+a.Module != key {
return nil, false
}
if a.Provisioning == nil {
continue
}
for _, b := range a.Provisioning.Provisions {
if b.Provider == node && b.ProviderModule == m.spec.Module && b.Provision == m.spec.Provision {
as := b.As
if as == "" {
as = provider.ConsumerIdentity(a.Node, a.Module) // a plan that names none
}
want[as] = true
}
}
}
return want, true
}
// granted is what the grant file gives that no plan names (issue 937), and every login it gives; and whether it
// could be read. The caller holds h.mu. With no grant file named there is nothing given, and that is a read.
func (m *Moved) granted(planned map[string]bool) ([]contribution, map[string]bool, bool) {
h := m.h
if h.Receives == "" {
return nil, map[string]bool{}, true
}
given := h.readContributions()
if given == nil {
return nil, nil, false
}
var own []contribution
logins := map[string]bool{}
for _, g := range given {
logins[g.As] = true
if !planned[g.As] {
own = append(own, g)
}
}
return own, logins, true
}
// retire is one pass of the harness's retirement over what the plans bind and the grant file gives now; the logins
// only the grant file gives are made and checked first, as the grant path does.
func (m *Moved) retire(ctx context.Context) {
h := m.h
want, ok := m.asked(h.Node)
h.mu.Lock()
defer h.mu.Unlock()
var given []contribution
if ok && m.spec.WithGrants {
var granted map[string]bool
given, granted, ok = m.granted(want)
for as := range granted {
want[as] = true
}
}
if !ok {
// Not a result: the passes that must agree before anything is retired start again.
h.r.key, h.r.count = "", 0
return
}
h.seed(ctx, want)
h.apply(ctx, given)
h.retireUnasked(ctx, want)
h.r.lastWant = want
for as := range h.failing {
if !want[as] {
delete(h.failing, as)
}
}
for as := range h.trouble {
if _, held := h.applied[as]; !want[as] && !held {
h.recovered(as, "no longer asked for")
}
}
}
// watch keeps the plans as the bus holds them, from the first read on.
func (m *Moved) watch(ctx context.Context, bus provider.Bus) error {
err := bus.Watch(ctx, provider.AssignmentsBucket, func(key string, value []byte) {
m.mu.Lock()
defer m.mu.Unlock()
if value == nil {
delete(m.assignments, key)
} else {
m.assignments[key] = append([]byte(nil), value...)
}
})
if err == nil {
m.mu.Lock()
m.loaded = true
m.mu.Unlock()
}
return err
}
// Run answers the consumers through the helper and retires through the harness's rules, until ctx ends.
//
// The bus is reached through the runtime that launched this bundle, which answers only once Serve runs, and Run starts
// beside Serve: a watch asked too early is asked again, after a wait that doubles from FirstRetry up to a minute,
// until it is answered or ctx ends, so a start that loses the race never leaves the consumers unanswered (issue 943).
func (m *Moved) Run(ctx context.Context, p *provider.Provider, bus provider.Bus) error {
wait := m.FirstRetry
if wait <= 0 {
wait = time.Second
}
for tries := 1; ; tries++ {
err := m.watch(ctx, bus)
if err == nil {
break
}
if tries == 4 || tries%20 == 0 {
m.h.say("the plans cannot be watched yet (%d tries), asking again: %v", tries, err)
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
wait = min(wait*2, time.Minute)
}
go func() {
for {
m.retire(ctx)
select {
case <-ctx.Done():
return
case <-time.After(m.h.Every):
}
}
}()
return p.Run(ctx)
}
// NewMoved is a provider handed to the provisioner: the helper answering its consumers, and the harness (never run)
// whose retirement and tools it shares. consumersFile is where its machine's client fills its consumers. Run the
// result with Moved.Run, passing the provider and the bus returned with it.
func NewMoved(consumersFile string, h *Harness, spec MovedSpec) (*Moved, *provider.Provider, provider.Bus, error) {
if consumersFile == "" {
return nil, nil, nil, errors.New("no consumers' file")
}
run := newMoved(h, spec)
bus := provider.StateBus(spec.Module)
return run, provider.New(provider.Options{
Node: h.Node,
Module: spec.Module,
Offers: map[string]provider.Offer{spec.Provision: spec.Offer},
Bus: bus,
Vault: provider.ConsumersFile{Paths: []string{consumersFile}},
Make: run.Make,
Gone: run.Gone,
Every: h.Every,
Logf: func(f string, a ...any) { h.say(f, a...) },
}), bus, nil
}
+480
View File
@@ -0,0 +1,480 @@
package harness
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
)
const (
ProvisionName = "postgres-database"
servedPort = 5432
)
// postgresSpec is what a postgres-like provider says: a login per consumer, and the grant file's logins.
var postgresSpec = MovedSpec{Module: "postgres", Provision: ProvisionName,
Offer: provider.Offer{Credential: provider.PerConsumer, Serves: map[string]any{"port": servedPort}},
Login: func(e provider.Entry) string { return e.User },
Noun: "login", Made: "login", Backend: "the backend", WithGrants: true}
// bucketBus is the helper's bus over in-memory buckets.
type bucketBus struct {
mu sync.Mutex
buckets map[string]map[string][]byte
}
func (b *bucketBus) Watch(_ context.Context, bucket string, handle func(string, []byte)) error {
b.mu.Lock()
defer b.mu.Unlock()
for k, v := range b.buckets[bucket] {
handle(k, v)
}
return nil
}
func (b *bucketBus) Put(_ context.Context, bucket, key string, value []byte) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.buckets[bucket] == nil {
b.buckets[bucket] = map[string][]byte{}
}
b.buckets[bucket][key] = append([]byte(nil), value...)
return nil
}
func (b *bucketBus) Delete(_ context.Context, bucket, key string) error {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.buckets[bucket], key)
return nil
}
// bind is one consumer's assignment, whose plan names this machine's postgres as the provider and carries what the
// consumer contributes (issue 776).
func (b *bucketBus) bind(node, module string, contributes map[string]any) {
b.bindNamed("postgres", node, module, contributes, nil)
}
// bindNamed is bind to a provider module, with the names the controller composed for the consumer (issue 792).
func (b *bucketBus) bindNamed(provides, node, module string, contributes map[string]any, names map[string]string) {
plan, _ := json.Marshal(map[string]any{"node": node, "module": module, "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox", "provider-module": provides,
"contributes": contributes, "names": names}}}})
if err := b.Put(context.Background(), provider.AssignmentsBucket, node+"."+module, plan); err != nil {
panic(err)
}
}
// consumersFile writes the file the machine's client fills: one credential per consumer.
func consumersFile(t *testing.T, bound map[[2]string]string) string {
var list []provider.ConsumerCredential
for k, password := range bound {
list = append(list, provider.ConsumerCredential{Consumer: k[0], Module: k[1],
Credential: provider.PairName("novox", ProvisionName, k[0], k[1], ""), Value: password})
}
raw, _ := json.Marshal(list)
at := filepath.Join(t.TempDir(), "consumers.json")
if err := os.WriteFile(at, raw, 0o600); err != nil {
t.Fatal(err)
}
return at
}
// movedWorld is postgres on the helper over an in-memory bus, with a recorder for a server and a clock to move.
type movedWorld struct {
t *testing.T
a *recorder
bus *bucketBus
h *Harness
spec MovedSpec
run *Moved
p *provider.Provider
now time.Time
file string
}
func newMovedWorld(t *testing.T, bound map[[2]string]string) *movedWorld {
return newMovedWorldOf(t, bound, postgresSpec)
}
func newMovedWorldOf(t *testing.T, bound map[[2]string]string, spec MovedSpec) *movedWorld {
w := &movedWorld{t: t, spec: spec, a: &recorder{}, bus: &bucketBus{buckets: map[string]map[string][]byte{}},
now: time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)}
w.file = consumersFile(t, bound)
w.h = &Harness{Resource: ProvisionName, Adapter: w.a, Node: "novox", Now: func() time.Time { return w.now },
Log: func(string, ...any) {}}
var bus provider.Bus
w.run, w.p, bus, _ = NewMoved(w.file, w.h, spec)
_ = bus
if err := w.run.watch(ctx, w.bus); err != nil {
t.Fatal(err)
}
return w
}
// pass is one helper pass and one retirement pass, five seconds after the last.
func (w *movedWorld) pass() []provider.Outcome {
w.now = w.now.Add(5 * time.Second)
// the plans as the bus holds them now, as the watch would hand them
w.run.assignments = map[string][]byte{}
for k, v := range w.bus.buckets[provider.AssignmentsBucket] {
w.run.assignments[k] = v
}
// the helper reads its buckets from the in-memory bus
w.p = provider.New(provider.Options{Node: "novox", Module: w.spec.Module, Bus: w.bus,
Vault: provider.ConsumersFile{Paths: []string{w.file}},
Offers: map[string]provider.Offer{ProvisionName: w.spec.Offer},
Make: w.run.Make, Gone: w.run.Gone, Logf: func(string, ...any) {}})
out, err := w.p.Pass(ctx)
if err != nil {
w.t.Fatal(err)
}
w.run.retire(ctx)
return out
}
func bindings(n int) (map[[2]string]string, []string) {
bound := map[[2]string]string{}
var logins []string
for i := 1; i <= n; i++ {
module := fmt.Sprintf("app%d", i)
bound[[2]string{"ace", module}] = "pw-" + module
logins = append(logins, provider.ConsumerIdentity("ace", module))
}
return bound, logins
}
// The consumers' file makes the same calls on the adapter as the equivalent grant files do (the harness).
func TestTheConsumersFileMakesWhatTheGrantFilesMake(t *testing.T) {
letta := provider.ConsumerIdentity("ace", "letta")
plain := provider.ConsumerIdentity("ace", "plain")
// Today: the grant files, through the harness.
g := newWorld(t)
g.give(
map[string]any{"as": letta, "node": "ace", "values": map[string]any{"name": "letta", "extensions": []any{"vector"}}},
map[string]any{"as": plain, "node": "ace", "values": map[string]any{"name": "plain"}},
)
g.h.Reconcile(ctx)
// The consumers' file, through the provider helper.
w := newMovedWorld(t, map[[2]string]string{{"ace", "letta"}: "pw-" + letta, {"ace", "plain"}: "pw-" + plain})
w.bus.bind("ace", "letta", map[string]any{"name": "letta", "extensions": []any{"vector"}})
w.bus.bind("ace", "plain", map[string]any{"name": "plain"})
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
byAs := map[string]Provision{}
for _, c := range w.a.created {
byAs[c.As] = c
}
if len(w.a.created) != 2 {
t.Fatalf("created %d, want 2", len(w.a.created))
}
for _, want := range g.a.created {
c := byAs[want.As]
if c.As != want.As || c.Password != want.Password || c.Consumer != want.Consumer ||
!reflect.DeepEqual(orEmpty(c.Values), orEmpty(want.Values)) {
t.Fatalf("made %+v, the grant files make %+v", c, want)
}
}
// Unchanged, and held: nothing is made again, within a minute or after it.
w.a.held = true
w.pass()
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 2 {
t.Fatalf("an unchanged pass created again: %d", len(w.a.created))
}
// No longer held: made again.
w.a.held = false
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 4 {
t.Fatalf("not made again: %d", len(w.a.created))
}
}
// One read without a consumer retires nothing: it takes the harness's stable passes and time, and then the entry
// goes and the login is retired, never dropped.
func TestAConsumerMissingFromOneReadRetiresNothing(t *testing.T) {
bound, logins := bindings(2)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.bus.bind("ace", "app2", nil)
w.pass()
// One read without app2.
saved := w.bus.buckets[provider.AssignmentsBucket]["ace.app2"]
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app2")
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v on one read", w.a.removed)
}
if _, kept := w.bus.buckets[provider.BucketOf("postgres")]["ace.app2.postgres-database"]; !kept {
t.Fatal("the entry went on one read")
}
// It is back: nothing was lost, and the entry never left the bucket.
w.bus.Put(ctx, provider.AssignmentsBucket, "ace.app2", saved)
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
if len(w.a.removed) != 0 || len(w.bus.buckets[provider.BucketOf("postgres")]) != 2 {
t.Fatalf("retired %v, entries %d", w.a.removed, len(w.bus.buckets[provider.BucketOf("postgres")]))
}
// Gone for good: not retired before StablePasses and StableFor, then retired and its entry removed.
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app2")
for i := 0; i < 6; i++ {
w.pass()
}
if len(w.a.removed) != 0 {
t.Fatalf("retired %v before the time was out", w.a.removed)
}
w.now = w.now.Add(11 * time.Minute)
w.pass()
w.pass()
if !reflect.DeepEqual(w.a.removed, []string{logins[1]}) || len(w.a.deleted) != 0 {
t.Fatalf("retired %v, deleted %v", w.a.removed, w.a.deleted)
}
if _, kept := w.bus.buckets[provider.BucketOf("postgres")]["ace.app2.postgres-database"]; kept {
t.Fatal("the entry stayed after the login was retired")
}
}
// More than three at once waits for a person, as the harness's bound says.
func TestMoreThanTheBoundWaitsForAPerson(t *testing.T) {
bound, _ := bindings(5)
w := newMovedWorld(t, bound)
for i := 1; i <= 5; i++ {
w.bus.bind("ace", fmt.Sprintf("app%d", i), nil)
}
w.pass()
for i := 1; i <= 4; i++ {
w.bus.Delete(ctx, provider.AssignmentsBucket, fmt.Sprintf("ace.app%d", i))
}
for i := 0; i < 6; i++ {
w.pass()
}
w.now = w.now.Add(11 * time.Minute)
w.pass()
w.pass()
if len(w.a.removed) != 0 || w.h.r.waiting == nil {
t.Fatalf("retired %v, waiting %v", w.a.removed, w.h.r.waiting)
}
}
// A plan list that was never read is not nobody asking.
func TestNoPlansReadRetiresNothing(t *testing.T) {
bound, _ := bindings(1)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.pass()
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app1")
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(20 * time.Minute)
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v with no plans read", w.a.removed)
}
}
// A consumer whose credential is not in the file yet is waited for: nothing is made for it.
func TestAConsumerWithoutACredentialIsWaitedFor(t *testing.T) {
w := newMovedWorld(t, map[[2]string]string{})
w.bus.bind("ace", "letta", map[string]any{"name": "letta"})
outcomes := w.pass()
if len(outcomes) != 1 || outcomes[0].Status != provider.Waiting || len(w.a.created) != 0 {
t.Fatalf("%+v, created %v", outcomes, w.a.created)
}
}
// Where the file is not there, postgres takes the grant directory, as before; where it is, the helper.
// The grant path restarts itself when the consumers file is first filled.
// A consumer whose module has a slug keeps the login its plan names, the slug first (issue 788): the adapter makes it
// under that login, and the retirement counts it as asked for, so a pass retires nothing.
func TestASluggedConsumerKeepsTheLoginItsPlanNames(t *testing.T) {
agent := provider.ConsumerIdentity("ace", "agent")
w := newMovedWorld(t, map[[2]string]string{{"ace", "claude-code"}: "pw-agent"})
plan, _ := json.Marshal(map[string]any{"node": "ace", "module": "claude-code", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
"provider-module": "postgres", "as": agent}}}})
if err := w.bus.Put(ctx, provider.AssignmentsBucket, "ace.claude-code", plan); err != nil {
t.Fatal(err)
}
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
if len(w.a.created) != 1 || w.a.created[0].As != agent {
t.Fatalf("made %+v, want one under %s", w.a.created, agent)
}
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v: the plan's login was counted unasked", w.a.removed)
}
}
// A provider whose spec names the plan's `as` as the login and no grant file (the keycloak shape) makes the consumer
// under that name with the values its spec composes, and a grant file that cannot be read does not hold up its
// retirement pass.
func TestASpecWithoutGrantsMakesUnderThePlansAsAndIgnoresTheGrantFile(t *testing.T) {
a := &recorder{}
bus := &bucketBus{buckets: map[string]map[string][]byte{}}
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
h := &Harness{Resource: ProvisionName, Adapter: a, Node: "novox", Now: func() time.Time { return now },
Receives: filepath.Join(t.TempDir(), "unreadable-or-missing.json"), Log: func(string, ...any) {}}
spec := MovedSpec{Module: "keycloak", Provision: ProvisionName,
Offer: provider.Offer{Credential: provider.PerConsumer},
Login: func(e provider.Entry) string { return e.As }, Noun: "client id", Made: "client", Backend: "Keycloak",
Values: func(e provider.Entry) map[string]any { return map[string]any{"label": e.Contributes["label"]} }}
file := consumersFile(t, map[[2]string]string{{"ace", "wiki"}: "pw-wiki"})
run, _, _, err := NewMoved(file, h, spec)
if err != nil {
t.Fatal(err)
}
plan, _ := json.Marshal(map[string]any{"node": "ace", "module": "wiki", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
"provider-module": "keycloak", "as": "wiki-client", "contributes": map[string]any{"label": "Wiki"}}}}})
_ = bus.Put(ctx, provider.AssignmentsBucket, "ace.wiki", plan)
if err := run.watch(ctx, bus); err != nil {
t.Fatal(err)
}
p := provider.New(provider.Options{Node: "novox", Module: "keycloak", Bus: bus,
Vault: provider.ConsumersFile{Paths: []string{file}},
Offers: map[string]provider.Offer{ProvisionName: spec.Offer}, Make: run.Make, Gone: run.Gone,
Logf: func(string, ...any) {}})
out, err := p.Pass(ctx)
if err != nil || len(out) != 1 || out[0].Status != provider.Answered {
t.Fatalf("%v %+v", err, out)
}
run.retire(ctx)
if len(a.created) != 1 || a.created[0].As != "wiki-client" || a.created[0].Password != "pw-wiki" ||
a.created[0].Values["label"] != "Wiki" {
t.Fatalf("made %+v", a.created)
}
}
const testIssuer = "https://id.example.org/realms/Novox"
var (
grafanaContributes = map[string]any{"label": "grafana", "endpoint": "web", "callback": "/login/generic_oauth"}
grafanaNames = map[string]string{"name": "grafana.example.org", "internal-name": "grafana.home.internal"}
)
// keycloakSpec is what a keycloak-like provider says: the client id is the plan's `as`, the values are what the
// consumer contributes with the controller's names over them, and no grant file is consulted.
var keycloakSpec = MovedSpec{Module: "keycloak", Provision: ProvisionName,
Offer: provider.Offer{Credential: provider.PerConsumer, Serves: map[string]any{"issuer": testIssuer,
"token-path": "/protocol/openid-connect/token"}},
Login: func(e provider.Entry) string { return e.As },
Values: func(e provider.Entry) map[string]any {
values := map[string]any{}
for k, v := range e.Contributes {
values[k] = v
}
for k, name := range e.Names {
values[k] = name
}
return values
},
Noun: "client id", Made: "client", Backend: "Keycloak"}
// The consumers' file makes the same call on the adapter as the equivalent grant file does (the harness), for a
// provider of the keycloak shape: the same client id, secret and values; and the entry tells the consumer what the
// provider serves.
func TestTheConsumersFileMakesWhatTheGrantFileMakesForTheKeycloakShape(t *testing.T) {
grafana := provider.ConsumerIdentity("home", "grafana")
values := map[string]any{"name": grafanaNames["name"], "internal-name": grafanaNames["internal-name"]}
for k, v := range grafanaContributes {
values[k] = v
}
g := newWorld(t)
g.give(map[string]any{"as": grafana, "node": "home", "values": values})
g.h.Reconcile(ctx)
w := newMovedWorldOf(t, map[[2]string]string{{"home", "grafana"}: "pw-" + grafana}, keycloakSpec)
w.bus.bindNamed("keycloak", "home", "grafana", grafanaContributes, grafanaNames)
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
if len(w.a.created) != 1 || len(g.a.created) != 1 {
t.Fatalf("created %d through the helper, %d through the harness, want 1 each", len(w.a.created), len(g.a.created))
}
c, want := w.a.created[0], g.a.created[0]
if c.As != want.As || c.Password != want.Password || c.Consumer != want.Consumer ||
!reflect.DeepEqual(orEmpty(c.Values), orEmpty(want.Values)) {
t.Fatalf("made %+v, the grant file makes %+v", c, want)
}
var entry provider.Entry
if err := json.Unmarshal(w.bus.buckets[provider.BucketOf("keycloak")]["home.grafana."+ProvisionName], &entry); err != nil {
t.Fatal(err)
}
if entry.Values["issuer"] != testIssuer || entry.Values["token-path"] != "/protocol/openid-connect/token" {
t.Fatalf("served %v", entry.Values)
}
w.a.held = true
w.pass()
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 1 {
t.Fatalf("an unchanged pass created again: %d", len(w.a.created))
}
w.a.held = false
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 2 {
t.Fatalf("not made again: %d", len(w.a.created))
}
}
// A consumer whose plan names its client id (a slugged module) keeps it, and the retirement counts it as asked for.
func TestAConsumerNamedByItsPlanKeepsItsClientId(t *testing.T) {
agent := provider.ConsumerIdentity("home", "dashboard")
w := newMovedWorldOf(t, map[[2]string]string{{"home", "grafana-2"}: "pw"}, keycloakSpec)
plan, _ := json.Marshal(map[string]any{"node": "home", "module": "grafana-2", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox", "provider-module": "keycloak",
"as": agent, "contributes": grafanaContributes, "names": grafanaNames}}}})
if err := w.bus.Put(ctx, provider.AssignmentsBucket, "home.grafana-2", plan); err != nil {
t.Fatal(err)
}
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
if len(w.a.created) != 1 || w.a.created[0].As != agent {
t.Fatalf("made %+v, want one under %s", w.a.created, agent)
}
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v: the plan's client id was counted unasked", w.a.removed)
}
}
+623
View File
@@ -0,0 +1,623 @@
package harness
// What becomes of a consumer the mesh no longer asks for (novox/hq ADR 0230 — the operator's model,
// decided 2026-10-06; it replaces ADR 0229's withdrawal brake, which let one consumer go every hour).
//
// A consumer — a login, a client, a key, and the data behind it — is in one of three states:
//
// 1. ACTIVE.
// 2. RETIRED. The mesh stopped asking for it: its access is disabled, reversibly, and its login and
// data are marked "to delete" with when and why. Nothing is deleted. Asked for again, the ordinary
// create re-enables it at once, as it was.
// 3. DELETED, only through `cleanup delete`, a person's act, which this provider executes because it
// owns its backend.
//
// **Stable removals.** A consumer is retired only once the same set of consumers has gone unasked BOTH
// in StablePasses (5) consecutive passes that read the file AND for at least StableFor (10 minutes)
// since the first pass that saw it. Five passes alone are twenty-five seconds — shorter than a
// controller restart, a store reconnecting or a file half written. A pass that could not read the file
// is not a result and starts both again; so does a different set. Additions and changes are never
// delayed.
//
// **Too many is a person.** A stable set of more than RetireAtOnce (3), or — where more than one is
// held — of more than RetireFraction (half) of those held, retires nothing: it WAITS, said in the
// journal and announced `provisioner.retirement` `waiting` (again every SayAgainEvery), which the
// controller raises as an urgent condition, until `retire approve <node> <module>` or `retire reject`.
// An unassignment the controller made itself is no exception: many changes at once means a person is
// at work, and a person confirms once. On 2026-10-04 one misread file withdrew seven consumers at once
// (issue 241); under this rule that is a question, not an act.
//
// **The backend remembers, not this process.** What is retired, since when and why is read from the
// backend's own mark (Adapter.Inventory), so a restart forgets nothing; a consumer the backend holds
// active that the mesh no longer asks for is retired by the same rules after a restart as before one.
// A consumer found disabled without the mark — withdrawn before this record — is ADOPTED as retired:
// marked, said, announced `adopted`, and its clock starts then.
//
// **A rejection lives as long as this process.** Rejected, the set is kept active and not asked about
// again while it stays the same set; a restart asks again — loudly, never silently.
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// EventRetirement is the one event a provider says about retirement, its `change` saying what
// happened. The controller derives the permission to emit it for every module that receives
// contributions, as it does the standing events (novox/hq ADR 0224, 0230).
const EventRetirement = "provisioner.retirement"
// The changes EventRetirement carries.
const (
ChangeWaiting = "waiting" // a stable set over the bound waits for a person
ChangeSettled = "settled" // a waiting or rejected set ended without being retired
ChangeApproved = "approved" // a person approved the waiting (or rejected) set
ChangeRejected = "rejected" // a person kept the waiting set active
ChangeRetired = "retired" // consumers disabled and marked to delete
ChangeReenabled = "reenabled" // a retired consumer asked for again, enabled as it was
ChangeDeleted = "deleted" // a retired consumer deleted, by a person
ChangeAdopted = "adopted" // found disabled without the mark, now retired on record
)
// StableFor is the least time the same unasked set must hold before it is retired (novox/hq ADR 0230):
// longer than a controller restart, a store reconnecting or a file half written.
const StableFor = 10 * time.Minute
// KindConsumer is a retired consumer. A backend may list other things set aside for deletion under a
// word of its own (postgres: a database renamed aside).
const KindConsumer = "consumer"
// Retired is one thing a provider holds retired, as its backend's mark says.
type Retired struct {
Consumer string
// Node is the consumer's machine, where the mark records it.
Node string
// RetiredAt is when it was retired; zero for one found disabled without the mesh's mark.
RetiredAt time.Time
Why string
// SizeBytes is what deleting it would free; -1 when the backend cannot say.
SizeBytes int64
Kind string
}
// Inventory is what a backend holds that the mesh made.
type Inventory struct {
Active []string
Retired []Retired
// Sizes is what each active consumer keeps on the backend, in bytes, where the backend can say: what
// the controller tells an empty replacement of a consumer's data by (novox/hq ADR 0233).
Sizes map[string]int64
}
// retirement is the loop's memory of the sets it is counting, waiting on and was told to keep.
type retirement struct {
key string // the unasked set, joined
count int // consecutive passes that saw it
firstSeen time.Time
waiting *waitingSet
rejected *rejectedSet
retired map[string]retiredEntry
lastWant map[string]bool
seeded bool
seedSaid string
// noneSaid is whether this process has said that nothing waits: an earlier process may have left a set
// waiting that this one never knew (issue 937).
noneSaid bool
}
type waitingSet struct {
set []string
since time.Time
saidAt time.Time
held int
}
type rejectedSet struct {
set []string
by, why string
at time.Time
}
type retiredEntry struct {
node string
at time.Time
why string
}
// seed reads what the backend holds, once per process: an active consumer the mesh no longer asks
// for is held, so it is retired by the same rules as one this process made; one found disabled
// without the mark is adopted as retired.
func (h *Harness) seed(ctx context.Context, want map[string]bool) {
if h.r.seeded {
return
}
inv, err := h.Adapter.Inventory(ctx)
if err != nil {
if said := err.Error(); said != h.r.seedSaid {
h.say("cannot list what the backend holds (%v); a consumer the mesh stopped asking for while this "+
"provider was down is not retired until it can", err)
h.r.seedSaid = said
}
return
}
h.r.seeded = true
for _, as := range inv.Active {
if _, held := h.applied[as]; !held && !want[as] {
// No hash: asked for again, it is applied again, which is harmless and marks it.
h.applied[as] = appliedEntry{}
h.say("%s: held by the backend and no longer asked for; retired once that holds for %d passes "+
"and %s (novox/hq ADR 0230)", as, h.StablePasses, h.StableFor)
}
}
now := h.Now()
for _, r := range inv.Retired {
if r.Kind != "" && r.Kind != KindConsumer {
continue
}
if !r.RetiredAt.IsZero() {
h.r.retired[r.Consumer] = retiredEntry{node: r.Node, at: r.RetiredAt, why: r.Why}
continue
}
if want[r.Consumer] {
continue // asked for: this pass's create enables it
}
why := "found disabled without the mesh's mark — withdrawn before retirement was recorded " +
"(novox/hq ADR 0230); retired as found"
if err := h.Adapter.Retire(ctx, r.Consumer, nil, why, now); err != nil {
h.say("%s: found disabled and could not be marked retired, will try at the next start: %v", r.Consumer, err)
continue
}
h.r.retired[r.Consumer] = retiredEntry{node: r.Node, at: now, why: why}
h.say("%s: ADOPTED as retired — %s", r.Consumer, why)
h.announce(EventRetirement, h.retirementBody(ChangeAdopted, []map[string]any{
{"consumer": r.Consumer, "node": r.Node, "retired_at": stamp(now), "why": why, "size_bytes": r.SizeBytes},
}, map[string]any{"why": why}))
}
}
// retireUnasked counts the passes that saw the same set no longer asked for and, once it is stable,
// retires it — or, past the bound, waits for a person.
func (h *Harness) retireUnasked(ctx context.Context, want map[string]bool) {
var unasked []string
for as := range h.applied {
if !want[as] {
unasked = append(unasked, as)
}
}
sort.Strings(unasked)
key := strings.Join(unasked, "\x00")
now := h.Now()
switch {
case len(unasked) == 0:
h.settle("every consumer held is asked for again")
h.r.key, h.r.count = "", 0
return
case key != h.r.key:
if h.r.waiting != nil || h.r.rejected != nil {
h.settle("the set the mesh no longer asks for changed")
}
h.r.key, h.r.count, h.r.firstSeen = key, 1, now
h.say("no longer asked for: %s; retired once the same set holds for %d passes and %s",
strings.Join(unasked, ", "), h.StablePasses, h.StableFor)
default:
h.r.count++
}
if h.r.rejected != nil || h.r.count < h.StablePasses || now.Sub(h.r.firstSeen) < h.StableFor {
return
}
if h.overTheBound(len(unasked), len(h.applied)) {
h.wait(unasked)
return
}
why := fmt.Sprintf("the mesh stopped asking for it: the same result in %d consecutive passes over %s, from %s",
h.r.count, now.Sub(h.r.firstSeen).Round(time.Second), stamp(h.r.firstSeen))
h.retire(ctx, unasked, why, nil)
}
// overTheBound says retiring n of the held consumers at once needs a person.
func (h *Harness) overTheBound(n, held int) bool {
if n == 0 {
return false
}
return n > h.RetireAtOnce || (held > 1 && float64(n) > h.RetireFraction*float64(held))
}
func (h *Harness) bound(held int) string {
return fmt.Sprintf("more than %d at once, or more than %.0f%% of the %d held", h.RetireAtOnce,
h.RetireFraction*100, held)
}
// wait holds a stable set over the bound for a person, said once and announced again every
// SayAgainEvery so a controller that missed the first hears the next.
func (h *Harness) wait(set []string) {
now := h.Now()
w := h.r.waiting
if w == nil {
w = &waitingSet{set: set, since: now, held: len(h.applied)}
h.r.waiting = w
h.say("RETIREMENT WAITS FOR A PERSON: the mesh no longer asks for %d of the %d consumer(s) this provider "+
"holds (%s), %s. Nothing is retired; each stays active until `retire approve %s <module>` or "+
"`retire reject` (novox/hq ADR 0230)", len(set), w.held, strings.Join(set, ", "), h.bound(w.held), h.Node)
} else if now.Sub(w.saidAt) < h.SayAgainEvery {
return
}
w.saidAt = now
h.announce(EventRetirement, h.retirementBody(ChangeWaiting, h.named(set), map[string]any{
"held": w.held, "bound": h.bound(w.held), "since": stamp(w.since),
}))
}
// settle ends a waiting or rejected set that the mesh's own answer made moot.
func (h *Harness) settle(why string) {
var set []string
switch {
case h.r.waiting != nil:
set = h.r.waiting.set
case h.r.rejected != nil:
set = h.r.rejected.set
case h.r.noneSaid:
return
default:
// A restart forgets a set an earlier process waited on, and so would never say it ended: the
// controller's condition would ask a person to approve what nobody waits on any more (issue 937).
// The first time this process finds every consumer held asked for, it says so once, with no set.
h.r.noneSaid = true
h.say("nothing waits for a person: %s", why)
h.announce(EventRetirement, h.retirementBody(ChangeSettled, []map[string]any{}, map[string]any{"why": why}))
return
}
h.r.waiting, h.r.rejected = nil, nil
h.say("retirement of %s settled without retiring: %s", strings.Join(set, ", "), why)
h.announce(EventRetirement, h.retirementBody(ChangeSettled, h.named(set), map[string]any{"why": why}))
}
// retire disables and marks each consumer of set; one that fails stays held and is tried again once
// its set is stable again.
func (h *Harness) retire(ctx context.Context, set []string, why string, extra map[string]any) []string {
now := h.Now()
held := len(h.applied)
var done []string
var said []map[string]any
for _, as := range set {
was, ok := h.applied[as]
if !ok {
continue
}
if err := h.Adapter.Retire(ctx, as, was.derived, why, now); err != nil {
h.say("%s: retiring failed, will try again: %v", as, err)
continue
}
delete(h.applied, as)
delete(h.lost, as)
delete(h.failing, as)
h.r.retired[as] = retiredEntry{node: was.node, at: now, why: why}
h.recovered(as, "retired")
h.say("%s: RETIRED — its access disabled and its data kept, marked to delete (%s). Asked for again "+
"it is re-enabled as it was; it is deleted only by `cleanup delete` (novox/hq ADR 0230)", as, why)
done = append(done, as)
said = append(said, map[string]any{"consumer": as, "node": was.node, "retired_at": stamp(now), "why": why})
}
h.r.key, h.r.count, h.r.waiting, h.r.rejected = "", 0, nil, nil
if len(done) > 0 {
body := map[string]any{"held": held, "why": why}
for k, v := range extra {
body[k] = v
}
h.announce(EventRetirement, h.retirementBody(ChangeRetired, said, body))
}
return done
}
// reenabled says a retired consumer was asked for again and its create enabled it.
func (h *Harness) reenabled(as, node string) {
e, was := h.r.retired[as]
if !was {
return
}
delete(h.r.retired, as)
h.say("%s: asked for again — re-enabled as it was, its mark to delete cleared (retired %s: %s)", as,
stamp(e.at), e.why)
h.announce(EventRetirement, h.retirementBody(ChangeReenabled, []map[string]any{
{"consumer": as, "node": node, "retired_at": stamp(e.at), "why": e.why},
}, nil))
}
// Approve retires exactly the set waiting (or the set rejected) — a person's confirmation.
func (h *Harness) Approve(ctx context.Context, consumers []string, why, by, via string) ([]string, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
if strings.TrimSpace(why) == "" {
return nil, errors.New("approve: say why")
}
set := sorted(consumers)
var held []string
switch {
case h.r.waiting != nil && same(set, h.r.waiting.set):
held = h.r.waiting.set
case h.r.rejected != nil && same(set, h.r.rejected.set):
held = h.r.rejected.set
default:
return nil, fmt.Errorf("approve: %s is not the set waiting here — %s", orNone(set), h.waitingWords())
}
h.say("retirement of %s APPROVED by %s: %s", strings.Join(held, ", "), orSomebody(by), why)
h.announce(EventRetirement, h.retirementBody(ChangeApproved, h.named(held),
map[string]any{"why": why, "by": by, "via": via}))
reason := fmt.Sprintf("the mesh stopped asking for it; approved by %s: %s", orSomebody(by), why)
return h.retire(ctx, held, reason, map[string]any{"by": by, "via": via}), nil
}
// Reject keeps the waiting set active; it is not asked about again while it stays the same set.
func (h *Harness) Reject(consumers []string, why, by, via string) ([]string, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
if strings.TrimSpace(why) == "" {
return nil, errors.New("reject: say why")
}
set := sorted(consumers)
if h.r.waiting == nil || !same(set, h.r.waiting.set) {
return nil, fmt.Errorf("reject: %s is not the set waiting here — %s", orNone(set), h.waitingWords())
}
h.r.rejected = &rejectedSet{set: h.r.waiting.set, by: by, why: why, at: h.Now()}
h.r.waiting = nil
h.say("retirement of %s REJECTED by %s: %s. Kept active, and not asked about again while the mesh goes on "+
"not asking for exactly these (until this provider restarts)", strings.Join(set, ", "), orSomebody(by), why)
h.announce(EventRetirement, h.retirementBody(ChangeRejected, h.named(set),
map[string]any{"why": why, "by": by, "via": via}))
return set, nil
}
// DeleteRetired deletes one retired consumer, by a person's word: never an active one, never one the
// mesh asks for.
func (h *Harness) DeleteRetired(ctx context.Context, consumer, why, by, via string) (int64, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
if strings.TrimSpace(why) == "" {
return 0, errors.New("delete: say why")
}
if h.r.lastWant[consumer] {
return 0, fmt.Errorf("delete: the mesh asks for %s — it is not retired", consumer)
}
if _, held := h.applied[consumer]; held {
return 0, fmt.Errorf("delete: %s is active here — only a retired consumer is deleted", consumer)
}
inv, err := h.Adapter.Inventory(ctx)
if err != nil {
return 0, fmt.Errorf("delete: what the backend holds cannot be read, so nothing is deleted: %w", err)
}
var found *Retired
for i := range inv.Retired {
if inv.Retired[i].Consumer == consumer {
found = &inv.Retired[i]
}
}
if found == nil {
return 0, fmt.Errorf("delete: %s is not retired here — only a retired consumer is deleted", consumer)
}
freed, err := h.Adapter.Delete(ctx, *found)
if err != nil {
return 0, err
}
delete(h.r.retired, consumer)
h.say("%s: DELETED by %s: %s (retired %s: %s; %d bytes freed)", consumer, orSomebody(by), why,
stampOr(found.RetiredAt), found.Why, freed)
h.announce(EventRetirement, h.retirementBody(ChangeDeleted, []map[string]any{{
"consumer": consumer, "node": found.Node, "retired_at": stampOr(found.RetiredAt), "why": found.Why,
"size_bytes": found.SizeBytes, "kind": kindOf(*found),
}}, map[string]any{"why": why, "by": by, "via": via, "freed_bytes": freed}))
return freed, nil
}
// Retirement is what a person (and the controller's `cleanup list` and its probe) asks: what is
// held, waiting, rejected and retired here.
func (h *Harness) Retirement(ctx context.Context) (map[string]any, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
inv, err := h.Adapter.Inventory(ctx)
if err != nil {
return nil, err
}
var held []string
for as := range h.applied {
held = append(held, as)
}
sort.Strings(held)
retired := []map[string]any{}
for _, r := range inv.Retired {
retired = append(retired, map[string]any{"consumer": r.Consumer, "node": r.Node,
"retired_at": stampOr(r.RetiredAt), "why": r.Why, "size_bytes": r.SizeBytes, "kind": kindOf(r)})
}
sizes := map[string]int64{}
for _, as := range held {
if size, ok := inv.Sizes[as]; ok && size >= 0 {
sizes[as] = size
}
}
out := map[string]any{"resource": h.Resource, "node": h.Node, "held": orEmptyList(held), "held_sizes": sizes,
"stable_passes": h.StablePasses, "stable_for_seconds": int(h.StableFor.Seconds()), "bound": h.bound(len(h.applied)), "waiting": nil, "rejected": nil,
"retired": retired}
if w := h.r.waiting; w != nil {
out["waiting"] = map[string]any{"consumers": h.named(w.set), "since": stamp(w.since), "held": w.held}
}
if r := h.r.rejected; r != nil {
out["rejected"] = map[string]any{"consumers": h.named(r.set), "by": r.by, "why": r.why, "at": stamp(r.at)}
}
return out, nil
}
func (h *Harness) waitingWords() string {
switch {
case h.r.waiting != nil:
return "waiting: " + strings.Join(h.r.waiting.set, ", ")
case h.r.rejected != nil:
return "nothing waits; rejected and kept: " + strings.Join(h.r.rejected.set, ", ")
}
return "nothing waits for a person here"
}
// named is a set with each consumer's machine, as the events and the tools say it.
func (h *Harness) named(set []string) []map[string]any {
out := make([]map[string]any, 0, len(set))
for _, as := range set {
out = append(out, map[string]any{"consumer": as, "node": h.applied[as].node})
}
return out
}
func (h *Harness) retirementBody(change string, consumers []map[string]any, extra map[string]any) map[string]any {
body := map[string]any{"provider": h.Resource, "provider-node": h.Node, "change": change,
"consumers": consumers, "at": stamp(h.Now())}
for k, v := range extra {
body[k] = v
}
return body
}
// RetirementTools are the four tools every provider serves for retirement, the same names in every
// module: the controller's `retire` and `cleanup` verbs ask them on the provider's machine.
func RetirementTools(h *Harness) []stdio.Tool {
if h == nil {
return nil
}
ask := func() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), 2*time.Minute)
}
who := map[string]any{
"why": map[string]any{"type": "string", "description": "why — required, kept in the hand-act log"},
"by": map[string]any{"type": "string", "description": "who decided"},
"via": map[string]any{"type": "string", "description": "mesh-controller when asked through its verbs"},
}
withSet := map[string]any{"consumers": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "the set, exactly as provisioner_retirement names it"}}
for k, v := range who {
withSet[k] = v
}
withOne := map[string]any{
"consumer": map[string]any{"type": "string", "description": "the retired consumer to delete"},
"confirm": map[string]any{"type": "string", "description": "the consumer's name again, to say this is meant"},
}
for k, v := range who {
withOne[k] = v
}
return []stdio.Tool{
{Name: "provisioner_retirement",
Description: "What this provider holds, what waits for a person to approve its retirement, what was rejected, " +
"and every retired consumer with when, why and its size (novox/hq ADR 0230).",
Run: func(map[string]any) (any, error) {
ctx, cancel := ask()
defer cancel()
return h.Retirement(ctx)
}},
{Name: "provisioner_retire_approve",
Description: "Retire exactly the set waiting for a person (or the set rejected earlier): access disabled, data " +
"kept and marked to delete. Use the controller's `retire approve`, which records the hand act.",
Input: withSet,
Run: func(args map[string]any) (any, error) {
ctx, cancel := ask()
defer cancel()
done, err := h.Approve(ctx, strs(args["consumers"]), argString(args, "why"), argString(args, "by"), argString(args, "via"))
if err != nil {
return nil, err
}
return map[string]any{"retired": orEmptyList(done)}, nil
}},
{Name: "provisioner_retire_reject",
Description: "Keep the set waiting for a person active. Use the controller's `retire reject`.",
Input: withSet,
Run: func(args map[string]any) (any, error) {
kept, err := h.Reject(strs(args["consumers"]), argString(args, "why"), argString(args, "by"), argString(args, "via"))
if err != nil {
return nil, err
}
return map[string]any{"kept": kept}, nil
}},
{Name: "provisioner_delete",
Description: "Delete one RETIRED consumer — its login and its data — for good. Refused for anything active or " +
"asked for. Use the controller's `cleanup delete`, which records the hand act.",
Input: withOne,
Run: func(args map[string]any) (any, error) {
consumer := argString(args, "consumer")
if consumer == "" || argString(args, "confirm") != consumer {
return nil, errors.New("delete: name the consumer, and repeat it in confirm")
}
ctx, cancel := ask()
defer cancel()
freed, err := h.DeleteRetired(ctx, consumer, argString(args, "why"), argString(args, "by"), argString(args, "via"))
if err != nil {
return nil, err
}
return map[string]any{"deleted": consumer, "freed_bytes": freed}, nil
}},
}
}
func strs(v any) []string {
list, _ := v.([]any)
out := []string{}
for _, x := range list {
if s, ok := x.(string); ok && s != "" {
out = append(out, s)
}
}
return out
}
func sorted(list []string) []string {
out := append([]string(nil), list...)
sort.Strings(out)
return out
}
func same(a, b []string) bool {
return strings.Join(sorted(a), "\x00") == strings.Join(sorted(b), "\x00")
}
func orNone(set []string) string {
if len(set) == 0 {
return "an empty set"
}
return strings.Join(set, ", ")
}
func orSomebody(by string) string {
if by == "" {
return "a person"
}
return by
}
func orEmptyList(list []string) []string {
if list == nil {
return []string{}
}
return list
}
func kindOf(r Retired) string {
if r.Kind == "" {
return KindConsumer
}
return r.Kind
}
func stamp(t time.Time) string { return t.UTC().Format(time.RFC3339) }
func stampOr(t time.Time) string {
if t.IsZero() {
return ""
}
return stamp(t)
}
func argString(args map[string]any, key string) string {
s, _ := args[key].(string)
return s
}
+571
View File
@@ -0,0 +1,571 @@
package harness
// Retirement (novox/hq ADR 0230), as the shared loop does it: a consumer the mesh stops asking for is
// retired only after the same result in five consecutive passes, too many at once wait for a person,
// asked for again it is re-enabled, and only a person deletes. The same file in every Go provider.
import (
"context"
"errors"
"fmt"
"os"
"strings"
"testing"
"time"
)
// recorder is a backend that remembers what it holds, active and retired, as a real one's mark does.
type recorder struct {
created []Provision
removed []string // retired, in order
deleted []string
held bool
failing error
holdsErr error
retErr error
inv Inventory
invErr error
}
func (r *recorder) Create(_ context.Context, p Provision) error {
if r.failing != nil {
return r.failing
}
r.created = append(r.created, p)
r.inv.Retired = withoutRetired(r.inv.Retired, p.As)
if !listHas(r.inv.Active, p.As) {
r.inv.Active = append(r.inv.Active, p.As)
}
return nil
}
func (r *recorder) Retire(_ context.Context, as string, _ map[string]any, why string, at time.Time) error {
if r.retErr != nil {
return r.retErr
}
r.removed = append(r.removed, as)
r.inv.Active = without(r.inv.Active, as)
r.inv.Retired = append(withoutRetired(r.inv.Retired, as),
Retired{Consumer: as, RetiredAt: at, Why: why, SizeBytes: 42, Kind: KindConsumer})
return nil
}
func (r *recorder) Holds(context.Context, Provision) (bool, error) {
if r.holdsErr != nil {
return false, r.holdsErr
}
return r.held, nil
}
func (r *recorder) Inventory(context.Context) (Inventory, error) { return r.inv, r.invErr }
func (r *recorder) Delete(_ context.Context, x Retired) (int64, error) {
r.deleted = append(r.deleted, x.Consumer)
r.inv.Retired = withoutRetired(r.inv.Retired, x.Consumer)
return x.SizeBytes, nil
}
func listHas(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
func without(list []string, s string) []string {
var out []string
for _, x := range list {
if x != s {
out = append(out, x)
}
}
return out
}
func withoutRetired(list []Retired, s string) []Retired {
var out []Retired
for _, x := range list {
if x.Consumer != s {
out = append(out, x)
}
}
return out
}
// holding gives the provider n consumers c1…cn and applies them.
func holding(w *world, n int) []map[string]any {
var given []map[string]any
for i := 1; i <= n; i++ {
given = append(given, map[string]any{"as": fmt.Sprintf("c%d", i), "node": "anchor"})
}
w.give(given...)
w.h.Reconcile(ctx)
return given
}
// pass is one reconcile five seconds after the last.
func (w *world) pass() {
w.now = w.now.Add(5 * time.Second)
w.h.Reconcile(ctx)
}
func retirements(said []announced) []string {
var out []string
for _, a := range said {
if a.event == EventRetirement {
out = append(out, a.body["change"].(string))
}
}
return out
}
func noneWaits(a announced) bool {
c, _ := a.body["consumers"].([]map[string]any)
return a.body["change"] == ChangeSettled && len(c) == 0
}
func lastRetirement(t *testing.T, said []announced) map[string]any {
t.Helper()
for i := len(said) - 1; i >= 0; i-- {
if said[i].event == EventRetirement {
return said[i].body
}
}
t.Fatal("nothing about retirement was announced")
return nil
}
func namesOf(body map[string]any) string {
var out []string
for _, c := range body["consumers"].([]map[string]any) {
out = append(out, c["consumer"].(string))
}
return strings.Join(out, ",")
}
// settle passes every five seconds until the same answer has held for StableFor, and one pass more.
func (w *world) settle() { w.passes(StableFor + 5*time.Second) }
func TestATransientEmptyListForFourPassesRetiresNothing(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 1)
w.give()
for i := 0; i < 4; i++ {
w.pass()
}
w.give(given...)
w.pass()
if len(w.a.removed) != 0 || len(retirements(*said)) != 0 {
t.Fatalf("four passes retired %v and said %v", w.a.removed, retirements(*said))
}
}
// Five identical passes are twenty-five seconds — shorter than a controller restart. Both must hold:
// five passes AND ten minutes of the same answer (novox/hq ADR 0230).
func TestFivePassesInTwentyFiveSecondsRetireNothingTenMinutesDo(t *testing.T) {
w, said := standingWorld(t)
holding(w, 1)
w.give()
for i := 0; i < 5; i++ {
w.pass()
}
if len(w.a.removed) != 0 || len(retirements(*said)) != 0 {
t.Fatalf("five passes in 25 s retired %v", w.a.removed)
}
w.passes(StableFor - 30*time.Second)
if len(w.a.removed) != 0 {
t.Fatalf("retired before the set held %s: %v", StableFor, w.a.removed)
}
w.passes(time.Minute)
if strings.Join(w.a.removed, ",") != "c1" {
t.Fatalf("the same set held %s and was not retired: %v", StableFor, w.a.removed)
}
// Ten minutes in one slow pass are not five passes.
w2 := newWorld(t)
holding(w2, 1)
w2.give()
w2.pass()
w2.now = w2.now.Add(StableFor)
w2.pass()
if len(w2.a.removed) != 0 {
t.Fatalf("two passes ten minutes apart retired %v", w2.a.removed)
}
w2.passes(15 * time.Second)
if len(w2.a.removed) != 1 {
t.Fatalf("five passes over ten minutes did not retire: %v", w2.a.removed)
}
}
func TestAStableSetIsRetiredAndAskedAgainReEnables(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 3)
w.give(given[:2]...)
w.settle()
if strings.Join(w.a.removed, ",") != "c3" {
t.Fatalf("retired %v", w.a.removed)
}
body := lastRetirement(t, *said)
if body["change"] != ChangeRetired || namesOf(body) != "c3" || body["held"] != 3 || body["provider-node"] != "anchor" ||
!strings.Contains(body["why"].(string), "consecutive passes over 10m") {
t.Fatalf("%v", body)
}
if len(w.a.inv.Retired) != 1 || w.a.inv.Retired[0].Consumer != "c3" {
t.Fatalf("the backend does not hold it retired: %+v", w.a.inv)
}
// Asked for again: the ordinary create, on the first pass, and said.
created := len(w.a.created)
w.give(given...)
w.pass()
if len(w.a.created) != created+1 || w.a.created[created].As != "c3" {
t.Fatalf("not created again: %v", w.a.created)
}
if body := lastRetirement(t, *said); body["change"] != ChangeReenabled || namesOf(body) != "c3" {
t.Fatalf("%v", body)
}
if len(w.a.inv.Retired) != 0 {
t.Fatalf("still marked retired: %+v", w.a.inv.Retired)
}
}
func TestAnUnreadablePassStartsTheCountAgain(t *testing.T) {
w := newWorld(t)
holding(w, 1)
w.give()
w.passes(StableFor - time.Minute)
os.WriteFile(w.receives, []byte("{"), 0o600)
w.pass()
w.give()
w.passes(StableFor - time.Minute)
if len(w.a.removed) != 0 {
t.Fatalf("an unreadable pass counted: %v", w.a.removed)
}
w.passes(2 * time.Minute)
if len(w.a.removed) != 1 {
t.Fatalf("not retired after ten minutes of readable passes: %v", w.a.removed)
}
}
func TestADifferentSetStartsTheCountAgain(t *testing.T) {
w := newWorld(t)
given := holding(w, 5)
w.give(given[:4]...)
w.passes(StableFor - time.Minute)
w.give(given[:3]...)
w.passes(StableFor - time.Minute)
if len(w.a.removed) != 0 {
t.Fatalf("a changed set kept its count: %v", w.a.removed)
}
w.passes(2 * time.Minute)
if strings.Join(w.a.removed, ",") != "c4,c5" {
t.Fatalf("%v", w.a.removed)
}
}
func TestAdditionsAndChangesAreNeverDelayed(t *testing.T) {
w := newWorld(t)
holding(w, 1)
w.give(map[string]any{"as": "c1", "node": "anchor"}, map[string]any{"as": "c2"})
w.pass()
if len(w.a.created) != 2 || w.a.created[1].As != "c2" {
t.Fatalf("an addition waited: %v", w.a.created)
}
w.give(map[string]any{"as": "c1", "node": "anchor", "values": map[string]any{"name": "rotated"}}, map[string]any{"as": "c2"})
w.pass()
if len(w.a.created) != 3 || w.a.created[2].As != "c1" {
t.Fatalf("a change waited: %v", w.a.created)
}
}
func TestTooManyWaitsForAPersonAndApproveRetiresExactlyThatSet(t *testing.T) {
w, said := standingWorld(t)
holding(w, 4)
w.give()
w.passes(15 * time.Minute)
if len(w.a.removed) != 0 {
t.Fatalf("four of four were retired without a person: %v", w.a.removed)
}
if got := strings.Join(retirements(*said), ","); got != ChangeWaiting {
t.Fatalf("said %q, want one waiting", got)
}
body := lastRetirement(t, *said)
if namesOf(body) != "c1,c2,c3,c4" || body["held"] != 4 || body["bound"] == "" {
t.Fatalf("%v", body)
}
if !strings.Contains(strings.Join(w.said, "\n"), "RETIREMENT WAITS FOR A PERSON") {
t.Fatal("the wait was not said")
}
// Said again every quarter of an hour while it waits.
w.passes(11 * time.Minute)
if got := strings.Join(retirements(*said), ","); got != "waiting,waiting" {
t.Fatalf("%q", got)
}
if _, err := w.h.Approve(ctx, []string{"c1", "c2"}, "tidy", "operator", "mesh-controller"); err == nil {
t.Fatal("approved a set that is not the one waiting")
}
if _, err := w.h.Approve(ctx, []string{"c4", "c3", "c2", "c1"}, "", "operator", ""); err == nil {
t.Fatal("approved without a why")
}
done, err := w.h.Approve(ctx, []string{"c4", "c3", "c2", "c1"}, "the old machine is gone", "operator", "mesh-controller")
if err != nil || strings.Join(done, ",") != "c1,c2,c3,c4" || strings.Join(w.a.removed, ",") != "c1,c2,c3,c4" {
t.Fatal(done, err, w.a.removed)
}
changes := retirements(*said)
if strings.Join(changes[len(changes)-2:], ",") != "approved,retired" {
t.Fatalf("%v", changes)
}
if b := lastRetirement(t, *said); b["by"] != "operator" || b["via"] != "mesh-controller" ||
!strings.Contains(b["why"].(string), "the old machine is gone") {
t.Fatalf("%v", b)
}
// Nothing more waits.
w.passes(time.Minute)
if r, _ := w.h.Retirement(ctx); r["waiting"] != nil || len(r["retired"].([]map[string]any)) != 4 {
t.Fatalf("%v", r)
}
}
func TestRejectedIsKeptAndNotAskedAgainUntilTheSetChanges(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 4)
w.give()
w.settle()
if _, err := w.h.Reject([]string{"c1"}, "no", "operator", ""); err == nil {
t.Fatal("rejected a set that is not the one waiting")
}
kept, err := w.h.Reject([]string{"c1", "c2", "c3", "c4"}, "a person is moving them", "operator", "mesh-controller")
if err != nil || len(kept) != 4 {
t.Fatal(kept, err)
}
w.passes(time.Hour)
if len(w.a.removed) != 0 {
t.Fatalf("a rejected set was retired: %v", w.a.removed)
}
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected" {
t.Fatalf("asked again after a rejection: %q", got)
}
r, _ := w.h.Retirement(ctx)
if r["rejected"] == nil || r["waiting"] != nil {
t.Fatalf("%v", r)
}
// One of them asked for again: a different answer, so the rejection is settled and counting
// starts over — three of four is over the bound again, and waits again.
w.give(given[0])
w.pass()
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected,settled" {
t.Fatalf("%q", got)
}
w.settle()
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected,settled,waiting" {
t.Fatalf("%q", got)
}
// A rejected set can still be approved later.
w2, _ := standingWorld(t)
holding(w2, 4)
w2.give()
w2.settle()
w2.h.Reject([]string{"c1", "c2", "c3", "c4"}, "wait", "operator", "")
if done, err := w2.h.Approve(ctx, []string{"c1", "c2", "c3", "c4"}, "now", "operator", ""); err != nil || len(done) != 4 {
t.Fatal(done, err)
}
}
func TestAWaitingSetAskedForAgainSettles(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 4)
w.give()
w.settle()
w.give(given...)
w.pass()
if got := strings.Join(retirements(*said), ","); got != "waiting,settled" || len(w.a.removed) != 0 {
t.Fatalf("%q %v", got, w.a.removed)
}
if _, err := w.h.Approve(ctx, []string{"c1", "c2", "c3", "c4"}, "late", "operator", ""); err == nil {
t.Fatal("approved a set that no longer waits")
}
}
func TestDeleteRemovesOnlyThatRetiredConsumer(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 5)
w.give(given[:3]...)
w.settle()
if strings.Join(w.a.removed, ",") != "c4,c5" {
t.Fatalf("%v", w.a.removed)
}
if _, err := w.h.DeleteRetired(ctx, "c1", "tidy", "operator", ""); err == nil {
t.Fatal("deleted an active consumer")
}
if _, err := w.h.DeleteRetired(ctx, "c5", "", "operator", ""); err == nil {
t.Fatal("deleted without a why")
}
if _, err := w.h.DeleteRetired(ctx, "nobody", "tidy", "operator", ""); err == nil {
t.Fatal("deleted something not retired")
}
freed, err := w.h.DeleteRetired(ctx, "c5", "the experiment is over", "operator", "mesh-controller")
if err != nil || freed != 42 || strings.Join(w.a.deleted, ",") != "c5" {
t.Fatal(freed, err, w.a.deleted)
}
if b := lastRetirement(t, *said); b["change"] != ChangeDeleted || namesOf(b) != "c5" || b["freed_bytes"] != int64(42) {
t.Fatalf("%v", b)
}
r, _ := w.h.Retirement(ctx)
if left := r["retired"].([]map[string]any); len(left) != 1 || left[0]["consumer"] != "c4" {
t.Fatalf("%v", r)
}
// Retired and asked for again before anybody deleted it: refused, it is the mesh's again.
w.give(given[:4]...)
w.pass()
if _, err := w.h.DeleteRetired(ctx, "c4", "tidy", "operator", ""); err == nil {
t.Fatal("deleted a consumer the mesh asks for")
}
}
func TestTheBound(t *testing.T) {
h := &Harness{}
h.init()
for _, c := range []struct{ n, held int }{{1, 1}, {1, 2}, {1, 3}, {3, 7}, {3, 6}, {2, 5}} {
if h.overTheBound(c.n, c.held) {
t.Errorf("%d of %d waits for a person", c.n, c.held)
}
}
for _, c := range []struct{ n, held int }{{2, 2}, {2, 3}, {4, 7}, {4, 10}, {7, 7}} {
if !h.overTheBound(c.n, c.held) {
t.Errorf("%d of %d is retired without a person", c.n, c.held)
}
}
}
func TestARestartRetiresWhatTheBackendHoldsUnaskedAndAdoptsWhatItFindsDisabled(t *testing.T) {
w, said := standingWorld(t)
w.a.inv = Inventory{Active: []string{"kept", "orphan"}, Retired: []Retired{
{Consumer: "locked-before", SizeBytes: 7, Kind: KindConsumer},
{Consumer: "old_deleted_20261005", RetiredAt: time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC), Kind: "set-aside-database"},
}}
w.give(map[string]any{"as": "kept"})
w.h.Reconcile(ctx)
if b := lastRetirement(t, *said); b["change"] != ChangeAdopted || namesOf(b) != "locked-before" {
t.Fatalf("%v", b)
}
if strings.Join(w.a.removed, ",") != "locked-before" {
t.Fatalf("adopting did not mark it: %v", w.a.removed)
}
w.settle()
if strings.Join(w.a.removed, ",") != "locked-before,orphan" {
t.Fatalf("an orphan the backend holds was not retired: %v", w.a.removed)
}
}
func TestABackendThatCannotBeListedIsSaidAndAskedAgain(t *testing.T) {
w := newWorld(t)
w.a.invErr = errors.New("connection refused")
holding(w, 1)
if !strings.Contains(strings.Join(w.said, "\n"), "cannot list what the backend holds") {
t.Fatal(w.said)
}
w.a.invErr = nil
w.a.inv = Inventory{Active: []string{"c1", "orphan"}}
w.pass()
w.settle()
if strings.Join(w.a.removed, ",") != "orphan" {
t.Fatalf("%v", w.a.removed)
}
}
func TestTheToolsAnswer(t *testing.T) {
w, _ := standingWorld(t)
holding(w, 4)
w.give()
w.settle()
tools := map[string]func(map[string]any) (any, error){}
for _, tool := range RetirementTools(w.h) {
tools[tool.Name] = tool.Run
}
if len(tools) != 4 {
t.Fatalf("%d tools", len(tools))
}
got, err := tools["provisioner_retirement"](nil)
if err != nil || got.(map[string]any)["waiting"] == nil {
t.Fatal(got, err)
}
set := []any{"c1", "c2", "c3", "c4"}
if _, err := tools["provisioner_retire_approve"](map[string]any{"consumers": set}); err == nil {
t.Fatal("approved without a why")
}
if _, err := tools["provisioner_retire_approve"](map[string]any{"consumers": set, "why": "gone", "by": "operator"}); err != nil {
t.Fatal(err)
}
if _, err := tools["provisioner_delete"](map[string]any{"consumer": "c1", "why": "gone"}); err == nil {
t.Fatal("deleted without confirm")
}
if _, err := tools["provisioner_delete"](map[string]any{"consumer": "c1", "confirm": "c1", "why": "gone"}); err != nil {
t.Fatal(err)
}
if strings.Join(w.a.deleted, ",") != "c1" {
t.Fatal(w.a.deleted)
}
}
func TestAFailingConsumerRetiredIsSaidRecovered(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 2)
w.a.held = false
w.a.failing = errors.New("boom")
w.passes(7 * time.Minute)
w.give(given[0])
w.settle()
recovered := false
for _, a := range *said {
if a.event == EventRecovered && a.body["consumer"] == "c2" && a.body["why"] == "retired" {
recovered = true
}
}
if !recovered {
t.Fatalf("%v", *said)
}
}
// everyWord is a world that keeps every word the provider announces.
func everyWord(t *testing.T) (*world, *[]announced) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) { said = append(said, announced{e, b}) }
w.h.Node = "anchor"
return w, &said
}
// A restarted provider that finds every consumer it holds asked for says once that nothing waits, so the controller
// clears a waiting condition an earlier process raised and this one never knew (issue 937); it says it no more.
func TestARestartedProviderSaysOnceThatNothingWaits(t *testing.T) {
w, said := everyWord(t)
holding(w, 2)
w.pass()
w.pass()
var words []announced
for _, a := range *said {
if a.event == EventRetirement {
words = append(words, a)
}
}
if len(words) != 1 || !noneWaits(words[0]) || words[0].body["provider-node"] != w.h.Node {
t.Fatalf("said %+v, want one settled with no set", words)
}
}
// A provider whose own set waits does not say that nothing waits.
func TestAProviderThatWaitsDoesNotSayNothingWaits(t *testing.T) {
w, said := everyWord(t)
w.a.inv.Active = []string{"c1", "c2", "c3", "c4", "c5"}
w.give(map[string]any{"as": "c1", "node": "anchor"})
w.settle()
for _, a := range *said {
if a.event == EventRetirement && noneWaits(a) {
t.Fatalf("said nothing waits while %v waited", w.h.r.waiting)
}
}
if w.h.r.waiting == nil {
t.Fatal("nothing waited: the test does not hold the case")
}
}
+193
View File
@@ -0,0 +1,193 @@
package harness
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224): not on the first
// failure, which may be a restart; after FailingAfter of failures with no success between; again
// every SayAgainEvery while it lasts; and recovered on the first success, or when the consumer goes.
import (
"errors"
"os"
"strings"
"testing"
"time"
)
type announced struct {
event string
body map[string]any
}
func standingWorld(t *testing.T) (*world, *[]announced) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) {
// The first success since start, and the first word that nothing waits (issue 937), are their own
// tests'; every other test reads past them.
if c, _ := b["consumers"].([]map[string]any); b["why"] != "first-success" &&
!(e == EventRetirement && b["change"] == ChangeSettled && len(c) == 0) {
said = append(said, announced{e, b})
}
}
w.h.Node = "anchor"
return w, &said
}
// passes reconciles every five seconds for d, as Run would.
func (w *world) passes(d time.Duration) {
for end := w.now.Add(d); w.now.Before(end); w.now = w.now.Add(5 * time.Second) {
w.h.Reconcile(ctx)
}
}
func events(said []announced) string {
var out []string
for _, a := range said {
out = append(out, a.event)
}
return strings.Join(out, ",")
}
func TestAConsumerFailedForMinutesIsAnnouncedNamingItAndTheClass(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New(`token request failed: 401 {"error":"invalid_grant","error_description":"Invalid user credentials"}`)
w.give(map[string]any{"as": "mesh_home_grafana", "node": "home-server"})
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("announced before FailingAfter: %v", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("want one %s, got %q", EventFailing, events(*said))
}
b := (*said)[0].body
if b["consumer"] != "mesh_home_grafana" || b["node"] != "home-server" || b["class"] != ClassCredentials ||
b["provider"] != "postgres-database" || b["provider-node"] != "anchor" || b["attempts"].(int) < 60 {
t.Fatalf("%v", b)
}
// Said again while it lasts, not every pass.
w.passes(14 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("repeated too soon: %q", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("not repeated: %q", events(*said))
}
// The first success takes it back.
w.a.failing = nil
w.passes(5 * time.Second)
if events(*said) != EventFailing+","+EventFailing+","+EventRecovered {
t.Fatalf("no recovery: %q", events(*said))
}
if (*said)[2].body["consumer"] != "mesh_home_grafana" {
t.Fatal((*said)[2].body)
}
}
func TestOneSuccessBetweenFailuresStartsTheRunAgain(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("connection refused")
w.give(map[string]any{"as": "a"})
w.passes(4 * time.Minute)
w.a.failing = nil
w.passes(5 * time.Second)
w.give(map[string]any{"as": "a", "values": map[string]any{"name": "changed"}})
w.a.failing = errors.New("connection refused")
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("two runs of four minutes are not one of eight: %q", events(*said))
}
}
// The check that failed for a day on 2026-10-05: clients already made, every minute's check refused
// at the token. A check that cannot be asked is a failure too.
func TestACheckThatKeepsFailingIsAFailureToo(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
w.a.holdsErr = errors.New("401 invalid_grant")
w.passes(7 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassCredentials {
t.Fatalf("%q %v", events(*said), *said)
}
w.a.holdsErr = nil
w.passes(time.Minute + 5*time.Second)
if events(*said) != EventFailing+","+EventRecovered {
t.Fatalf("%q", events(*said))
}
}
func TestAnUnreadableSecretIsAnnouncedAsSuch(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
os.Remove(w.dir + "/a.secret")
w.passes(6 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassSecret {
t.Fatalf("%q %v", events(*said), *said)
}
}
func TestAConsumerNoLongerAskedForIsNoLongerFailing(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("boom")
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
w.passes(6 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("%q", events(*said))
}
w.give(map[string]any{"as": "a"})
w.passes(5 * time.Second)
last := (*said)[len(*said)-1]
if last.event != EventRecovered || last.body["consumer"] != "b" || last.body["why"] != "no longer asked for" {
t.Fatalf("%v", *said)
}
}
func TestAnErrorIsClassedByItsWords(t *testing.T) {
for text, want := range map[string]string{
`Keycloak token request failed: 401 {"error":"invalid_grant"}`: ClassCredentials,
`FATAL: password authentication failed for user "postgres"`: ClassCredentials,
`dial tcp 127.0.0.1:5432: connect: connection refused`: ClassUnreachable,
`context deadline exceeded`: ClassUnreachable,
`extension "nope" is not available`: ClassRefused,
} {
if got := ClassOf(text); got != want {
t.Errorf("%s: %s, want %s", text, got, want)
}
}
}
func TestAnAdapterThatClassesItsOwnErrorsIsBelieved(t *testing.T) {
w, said := standingWorld(t)
w.h.Adapter = classing{w.a}
w.a.failing = errors.New("anything")
w.give(map[string]any{"as": "a"})
w.passes(6 * time.Minute)
if (*said)[0].body["class"] != "its-own" {
t.Fatal((*said)[0].body)
}
}
type classing struct{ *recorder }
func (classing) Class(error) string { return "its-own" }
// A provider restarted after announcing a failure has forgotten it; its first success for each
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
func TestTheFirstSuccessSinceStartIsAnnouncedOnce(t *testing.T) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) {
if e != EventRetirement {
said = append(said, announced{e, b})
}
}
w.give(map[string]any{"as": "a"})
w.passes(3 * time.Minute)
if events(said) != EventRecovered || said[0].body["why"] != "first-success" || said[0].body["consumer"] != "a" {
t.Fatalf("%v", said)
}
}
+746
View File
@@ -0,0 +1,746 @@
// Package provider is a providing module's half of a binding: it answers the provider's consumers itself, over the
// bus (novox/hq ADR 0349 clause 12, issue 678): the provider module, not the controller, makes each bound
// consumer's entry.
//
// **Who binds the provider.** Each node's provisioner client records the needs of the modules handed to it from
// their plans, and the controller names in each plan, per required provision and local name, the provider that
// resolution chose: `provisioning.provisions` in `<node>.<module>` of the assignments mirror (issue 760). A recorded
// need names no provider (issue 677's note 7), so the plan's entry is where a bind is seen, as the vault sees it
// (mesh-vault pair.go). A consumer binds this provider when an entry names this machine and this module, and
// unbinds when the entry, or the assignment, goes. A consumer still filled the old way has no plan and is not seen
// here; the controller answers it as before (ADR 0349 clause 17).
//
// **One entry per consumer, in the provider's own bucket.** For each bind the helper composes the consumer's
// served values from the provider's offer (its config already composed into it, ADR 0202 §7) and the consumer's
// identity (`${consumer:as}`, `${consumer:as:dns}`, ADR 0202 §1), and writes one entry, `<node>.<module>.
// <provision>[.<local>]` of `<provider module>_bound`: the one statement both ends read. It never carries a
// secret. It names the credential the vault keeps for the pair, which the consumer's client asks the vault for.
//
// **The credential is the vault's** (ADR 0349 clause 8). The helper mints nothing and reads nothing of the
// provider's own secrets: it reads the value from the vault and hands it to the provider's adapter, which makes the
// login its software keeps. An offer whose software holds one credential (ADR 0158) is the vault's one value per
// provider and provision, `provision/<provider node>/<provision>`, with the provider's one user name; an offer with
// a login per consumer (ADR 0048, ADR 0158 §11, the default) is each consumer's own value,
// `pair/<provider node>/<provision>/<node>/<module>[/<local>]`, with the consumer's identity as its login (issue
// 677's notes 12 and 13). A credential the vault does not hold yet is waited for: nothing is made and no entry is
// written, so a consumer is never told of a login its provider has not made. A provider in the tool runner never asks
// the vault itself: its Vault is ConsumersFile, the file its machine's provisioner client fills (ADR 0373).
//
// **A multi-holder consumer gets no derived value** (ADR 0202 §10 as ADR 0349 clause 12 changes it): a consumer
// bound under a local name keeps several holders of the provision, each with its own login, while what the
// provider derives names one; its entry refuses it, naming both ends, and its client reports it.
//
// The helper speaks to the bus and the vault through the two interfaces below, so a provider wires it to its
// runtime and a test to a stand-in. Standard library only.
package provider
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// AssignmentsBucket is the controller's assignments mirror, `<node>.<module>`, whose value carries the plan of a
// module handed to the provisioner.
const AssignmentsBucket = "mesh-controller_assignments"
// BucketOf is a providing module's own bucket of entries, one per consumer: `<module>_bound`, the module's state
// `bound`.
func BucketOf(module string) string { return module + "_bound" }
// Bus is what the helper needs of the bus: a key-value watch and the provider's writes to its own bucket.
type Bus interface {
// Watch hands handle every current key and value of bucket, then each change as it comes, a removed key with a
// nil value; it returns once the current values are handed, and the watch lasts until ctx ends.
Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error
Put(ctx context.Context, bucket, key string, value []byte) error
Delete(ctx context.Context, bucket, key string) error
}
// ConditionalBus is a Bus that can also delete a key only while it still holds what the caller read. The helper uses
// it when the Bus offers it, so a pass that lets a consumer go never removes an entry another instance of the
// provider wrote after the pass read it; a Bus without it deletes unconditionally. DeleteIf returns an error that
// is stdio.ErrChanged when the key holds something else now.
type ConditionalBus interface {
Bus
DeleteIf(ctx context.Context, bucket, key string, held []byte) error
}
// Vault is how the provider reads a credential the vault keeps, by its name, opened.
type Vault interface {
Value(ctx context.Context, name string) ([]byte, error)
}
// CredentialForm is which credential an offer's consumers present.
type CredentialForm int
const (
// PerConsumer is a login of each consumer's own (ADR 0048, ADR 0158 §11): the default.
PerConsumer CredentialForm = iota
// Shared is the one credential the provider's software holds, shared with every consumer (ADR 0158).
Shared
// NoCredential is an offer whose consumers present none.
NoCredential
)
// Offer is what the provider serves one provision with.
type Offer struct {
// Serves are the values served to each consumer, the provider's config already composed into them; a string
// may name the consumer's identity, `${consumer:as}` or `${consumer:as:dns}`.
Serves map[string]any
// Credential is the form of the credential its consumers present. A consumer's plan entry that names a
// credential form of its own wins over it (the plan's `credential-form`).
Credential CredentialForm
// User is the provider's one user name, for a Shared credential.
User string
}
// Entry is one consumer's entry in the provider's bucket: the one statement both ends read. Never a secret.
type Entry struct {
Provider string `json:"provider"`
ProviderModule string `json:"provider-module"`
Provision string `json:"provision"`
Consumer string `json:"consumer"`
Module string `json:"module"`
Local string `json:"local,omitempty"`
// As is the consumer's identity, the mesh's (ADR 0202 §1).
As string `json:"as,omitempty"`
// User is the login the consumer presents: its identity, or the provider's one user name for a shared credential.
User string `json:"user,omitempty"`
// Values are the served values, composed for this consumer: what `${bound:<provision>:<key>}` reads.
Values map[string]any `json:"values,omitempty"`
// VaultName is the name under which the vault keeps this consumer's credential, which the consumer's client asks
// the vault for: a name, never the credential. Its key is not "credential" because the runtime's state guard
// refuses any key ending in that word, whatever the value (issue 945).
VaultName string `json:"vault-name,omitempty"`
// Refused, when set, is why the consumer is not served, naming both ends.
Refused string `json:"refused,omitempty"`
// Contributes is what the consumer's module tells its provider about this provision (a database's name, its
// extensions), from its plan's entry (issue 776). Handed to Make and Gone; never written to the bucket.
Contributes map[string]any `json:"-"`
// At and From are where the consumer's plan says the provider is and which machine it is (issue 788): the values
// today's binding file gives that consumer. Written into Values as `at` and `from`; never alone to the bucket.
At, From string `json:"-"`
// Names are the names the controller composed for this consumer on this provision (issue 792): its own name, its
// internal name, and one of each per local name it contributes under, keyed as its binding file names them.
// Written into Values over anything the provider serves under the key; never alone to the bucket.
Names map[string]string `json:"-"`
// CredentialForm and Name are from the consumer's plan entry (issue 825): the form of the credential the
// controller chose for this bind (`shared`, `pair`, `issued` or `given`) and the holder's name. When set, they
// win over the offer's form. Never written to the bucket.
CredentialForm, Name string `json:"-"`
}
// Key is the entry's key in the provider's bucket: `<node>.<module>.<provision>`, and `.<local>` for a bind under a
// local name.
func (e Entry) Key() string {
k := e.Consumer + "." + e.Module + "." + e.Provision
if e.Local != "" {
k += "." + e.Local
}
return k
}
// Consumer is what the provider's adapter makes: the consumer's entry and the credential the vault keeps for it.
type Consumer struct {
Entry Entry
Credential []byte
}
// Options are the provider's.
type Options struct {
// Node and Module are this provider: the machine it runs on and the module serving.
Node, Module string
// Offers are the provisions it serves, by name.
Offers map[string]Offer
Bus Bus
Vault Vault
// Make brings a consumer into being in the provider's software under the vault's credential. Asked on every
// pass for every consumer answered, so it must leave one that is already as asked alone; a rotated credential
// reaches the software this way.
Make func(ctx context.Context, c Consumer) error
// Gone, when set, hears that a consumer no longer binds the provider, before its entry is removed; an error
// keeps the entry, and it is heard again on the next pass. Retiring what was kept for it is the provider's
// (ADR 0230).
Gone func(ctx context.Context, e Entry) error
// Identity is the consumer's identity for an entry whose plan names none; ConsumerIdentity of the node and the
// module when not set.
Identity func(node, module string) string
// Every is how often a pass runs without a change watched; a minute when not set.
Every time.Duration
// Logf says what a pass did that differs from the last; the standard log when not set.
Logf func(format string, args ...any)
}
// Provider answers its consumers.
type Provider struct {
opts Options
mu sync.Mutex
assignments map[string][]byte
entries map[string][]byte
said map[string]string
poke chan struct{}
}
// New is a provider answering its consumers as opts say.
func New(opts Options) *Provider {
if opts.Identity == nil {
opts.Identity = ConsumerIdentity
}
if opts.Every <= 0 {
opts.Every = time.Minute
}
if opts.Logf == nil {
opts.Logf = func(f string, a ...any) { fmt.Printf("[provider] "+f+"\n", a...) }
}
return &Provider{opts: opts, assignments: map[string][]byte{}, entries: map[string][]byte{},
said: map[string]string{}, poke: make(chan struct{}, 1)}
}
// What a pass did for one bind.
const (
Answered = "answered"
Refused = "refused"
Waiting = "waiting"
Failed = "failed"
Removed = "removed"
// Superseded is an entry no bind asks for any more that another instance of the provider wrote after this pass
// read it: it is left standing.
Superseded = "superseded"
)
// Outcome is what a pass did for one consumer's entry.
type Outcome struct {
Key string `json:"key"`
Status string `json:"status"`
Why string `json:"why,omitempty"`
}
// Run watches the binds and the provider's own entries, and passes once at start, on every change and every
// Every, until ctx ends.
func (p *Provider) Run(ctx context.Context) error {
if err := p.watch(ctx, BucketOf(p.opts.Module), p.entries); err != nil {
return fmt.Errorf("watching %s: %w", BucketOf(p.opts.Module), err)
}
if err := p.watch(ctx, AssignmentsBucket, p.assignments); err != nil {
return fmt.Errorf("watching %s: %w", AssignmentsBucket, err)
}
tick := time.NewTicker(p.opts.Every)
defer tick.Stop()
for {
p.mu.Lock()
assignments, entries := clone(p.assignments), clone(p.entries)
p.mu.Unlock()
if _, err := p.pass(ctx, assignments, entries); err != nil {
p.say("pass", err.Error())
}
select {
case <-ctx.Done():
return ctx.Err()
case <-p.poke:
case <-tick.C:
}
}
}
func (p *Provider) watch(ctx context.Context, bucket string, into map[string][]byte) error {
return p.opts.Bus.Watch(ctx, bucket, func(key string, value []byte) {
p.mu.Lock()
if value == nil {
delete(into, key)
} else {
into[key] = append([]byte(nil), value...)
}
p.mu.Unlock()
if bucket == AssignmentsBucket {
select {
case p.poke <- struct{}{}:
default:
}
}
})
}
// Pass reads the binds and the provider's entries from the bus and answers once: each bind of this provider gets
// its entry, and each entry of this provider no bind asks for is removed. An assignment that cannot be read, or that
// is not the one its key names, is skipped and said (an outcome of status Failed under the assignment's key, and a
// line of the log), and the entries of its consumer are left as they are, since a bind missed would read as an
// unbind; the pass goes on for the rest. Run passes by itself; Pass is for a provider that passes when it chooses.
//
// Over StateBus each call to Pass leaves two watches open for the life of the process, silenced once the call has
// ended (see StateBus): a provider that passes often should use Run, which watches once.
func (p *Provider) Pass(ctx context.Context) ([]Outcome, error) {
watchCtx, cancel := context.WithCancel(ctx)
defer cancel()
var mu sync.Mutex
read := func(bucket string) (map[string][]byte, error) {
got := map[string][]byte{}
err := p.opts.Bus.Watch(watchCtx, bucket, func(key string, value []byte) {
mu.Lock()
defer mu.Unlock()
if value == nil {
delete(got, key)
} else {
got[key] = append([]byte(nil), value...)
}
})
mu.Lock()
defer mu.Unlock()
return clone(got), err
}
entries, err := read(BucketOf(p.opts.Module))
if err != nil {
return nil, fmt.Errorf("reading %s: %w", BucketOf(p.opts.Module), err)
}
assignments, err := read(AssignmentsBucket)
if err != nil {
return nil, fmt.Errorf("reading %s: %w", AssignmentsBucket, err)
}
return p.pass(ctx, assignments, entries)
}
// planned is as much of an assignment as the helper reads: lenient, since the plan is the client's to act on.
type planned struct {
Node string `json:"node"`
Module string `json:"module"`
Provisioning *struct {
Provisions []struct {
Provision string `json:"provision"`
Local string `json:"local"`
Provider string `json:"provider"`
ProviderModule string `json:"provider-module"`
Contributes map[string]any `json:"contributes"`
At string `json:"at"`
From string `json:"from"`
As string `json:"as"`
Names map[string]string `json:"names"`
CredentialForm string `json:"credential-form"`
Name string `json:"name"`
} `json:"provisions"`
} `json:"provisioning"`
}
func (p *Provider) pass(ctx context.Context, assignments, entries map[string][]byte) ([]Outcome, error) {
var binds []Entry
var out []Outcome
unknown := map[string]bool{} // consumers whose assignment was skipped: their entries are not judged
for _, key := range sortedKeys(assignments) {
var a planned
if err := json.Unmarshal(assignments[key], &a); err != nil || a.Node+"."+a.Module != key {
o := Outcome{Key: key, Status: Failed, Why: fmt.Sprintf("the assignment %s is not the controller's, so "+
"who it binds is not known; it is skipped, and its entries are left as they are", key)}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
unknown[key] = true
continue
}
if a.Provisioning == nil {
continue
}
for _, b := range a.Provisioning.Provisions {
if b.Provider != p.opts.Node || b.ProviderModule != p.opts.Module {
continue
}
binds = append(binds, Entry{Provider: p.opts.Node, ProviderModule: p.opts.Module,
Provision: b.Provision, Consumer: a.Node, Module: a.Module, Local: b.Local,
Contributes: b.Contributes, At: b.At, From: b.From, As: b.As, Names: b.Names,
CredentialForm: b.CredentialForm, Name: b.Name})
}
}
want := map[string]bool{}
for _, b := range binds {
key := b.Key()
want[key] = true
o := p.answer(ctx, b, entries[key])
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
}
for _, key := range sortedKeys(entries) {
if want[key] || unknown[consumerOf(key)] {
continue
}
var e Entry
if err := json.Unmarshal(entries[key], &e); err != nil || e.Provider != p.opts.Node ||
e.ProviderModule != p.opts.Module {
continue // another machine's instance of this provider wrote it, or nothing of this helper's
}
if p.opts.Gone != nil {
if err := p.opts.Gone(ctx, e); err != nil {
o := Outcome{Key: key, Status: Failed, Why: "no longer bound, and the provider could not let it go: " +
err.Error()}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
continue
}
}
if err := p.remove(ctx, key, entries[key]); err != nil {
if errors.Is(err, stdio.ErrChanged) {
o := Outcome{Key: key, Status: Superseded, Why: "another instance of the provider wrote its entry " +
"since this pass read it, so it stands"}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
continue
}
o := Outcome{Key: key, Status: Failed, Why: "no longer bound, and its entry could not be removed: " +
err.Error()}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
continue
}
out = append(out, Outcome{Key: key, Status: Removed})
p.say(key, Removed)
p.mu.Lock()
delete(p.said, key)
p.mu.Unlock()
}
return out, nil
}
// consumerOf is the `<node>.<module>` an entry's key starts with: its first two parts.
func consumerOf(key string) string {
parts := strings.SplitN(key, ".", 3)
if len(parts) < 3 {
return key
}
return parts[0] + "." + parts[1]
}
// remove deletes an entry this instance read as held, and only while it is still as read when the Bus can say so.
func (p *Provider) remove(ctx context.Context, key string, held []byte) error {
if c, ok := p.opts.Bus.(ConditionalBus); ok {
return c.DeleteIf(ctx, BucketOf(p.opts.Module), key, held)
}
return p.opts.Bus.Delete(ctx, BucketOf(p.opts.Module), key)
}
// answer makes one bind's entry and writes it when it differs from what the bucket holds.
func (p *Provider) answer(ctx context.Context, e Entry, have []byte) Outcome {
key := e.Key()
offer, offered := p.opts.Offers[e.Provision]
switch {
case !offered:
e.Refused = fmt.Sprintf("%s on %s is bound to %s on %s for %s, which %s does not serve", e.Module,
e.Consumer, e.ProviderModule, e.Provider, e.Provision, e.ProviderModule)
case e.Local != "" && derives(offer.Serves):
e.Refused = fmt.Sprintf("%s on %s keeps several holders of %s (this one is %q), and %s on %s derives %s for "+
"each consumer from its login: each holder has a login of its own and a consumer is told one value per "+
"requirement, so the two ends would name different things (novox/hq ADR 0202 §10, ADR 0349 clause 12)",
e.Module, e.Consumer, e.Provision, e.Local, e.ProviderModule, e.Provider,
strings.Join(derivedKeys(offer.Serves), ", "))
}
if e.Refused != "" {
return p.write(ctx, e, have, Outcome{Key: key, Status: Refused, Why: e.Refused})
}
// The plan's identity is the controller's, slug first; Identity only answers for an entry without one.
if e.As == "" {
e.As = p.opts.Identity(e.Consumer, e.Module)
}
values, err := ServedTo(offer.Serves, e.As)
if err != nil {
e.Refused = fmt.Sprintf("%s on %s serving %s to %s on %s: %v", e.ProviderModule, e.Provider, e.Provision,
e.Module, e.Consumer, err)
e.As = ""
return p.write(ctx, e, have, Outcome{Key: key, Status: Refused, Why: e.Refused})
}
if e.At != "" || e.From != "" {
if values == nil {
values = map[string]any{}
}
if e.At != "" {
values["at"] = e.At
}
if e.From != "" {
values["from"] = e.From
}
}
if len(e.Names) > 0 {
if values == nil {
values = map[string]any{}
}
for key, name := range e.Names {
values[key] = name
}
}
e.Values = values
var credential []byte
// The plan's form wins; the offer's is the default.
// An offer that declares no credential has none whatever the plan says: the controller writes "pair" for it.
form := e.CredentialForm
if offer.Credential == NoCredential {
form = ""
} else if form == "" {
form = map[CredentialForm]string{Shared: "shared", PerConsumer: "pair"}[offer.Credential]
}
switch form {
case "shared":
e.User = offer.User
e.VaultName = SharedName(e.Provider, e.Provision)
case "pair":
e.User = e.As
e.VaultName = PairName(e.Provider, e.Provision, e.Consumer, e.Module, e.Local)
case "issued", "given":
// A per-holder value, as mesh-vault names it (perholder.go).
e.User = e.As
e.VaultName = HolderName(e.Provider, e.Provision, e.Consumer, e.Module)
}
if e.VaultName != "" {
credential, err = p.opts.Vault.Value(ctx, e.VaultName)
if err != nil {
return Outcome{Key: key, Status: Waiting, Why: fmt.Sprintf("the vault's %s: %v", e.VaultName, err)}
}
}
if p.opts.Make != nil {
if err := p.opts.Make(ctx, Consumer{Entry: e, Credential: credential}); err != nil {
why := err.Error()
why = redact(why, credential)
return Outcome{Key: key, Status: Failed, Why: "the provider could not make it: " + why}
}
}
return p.write(ctx, e, have, Outcome{Key: key, Status: Answered})
}
// redact takes a credential out of text in the forms it can come back in: as it is, in base64 (standard and URL,
// padded and not) and as a JSON string says it (escapes included).
func redact(text string, credential []byte) string {
if len(credential) == 0 {
return text
}
quoted, _ := json.Marshal(string(credential))
forms := []string{string(credential), base64.StdEncoding.EncodeToString(credential),
base64.RawStdEncoding.EncodeToString(credential), base64.URLEncoding.EncodeToString(credential),
base64.RawURLEncoding.EncodeToString(credential), string(quoted[1 : len(quoted)-1])}
sort.SliceStable(forms, func(i, j int) bool { return len(forms[i]) > len(forms[j]) })
for _, f := range forms {
text = strings.ReplaceAll(text, f, "<credential>")
}
return text
}
func (p *Provider) write(ctx context.Context, e Entry, have []byte, o Outcome) Outcome {
body, err := json.Marshal(e)
if err != nil {
return Outcome{Key: o.Key, Status: Failed, Why: err.Error()}
}
if sameJSON(have, body) {
return o
}
if err := p.opts.Bus.Put(ctx, BucketOf(p.opts.Module), o.Key, body); err != nil {
return Outcome{Key: o.Key, Status: Failed, Why: "its entry could not be written: " + err.Error()}
}
return o
}
// say logs what a key's pass said when it differs from the last.
func (p *Provider) say(key, what string) {
p.mu.Lock()
same := p.said[key] == what
p.said[key] = what
p.mu.Unlock()
if !same {
p.opts.Logf("%s: %s", key, strings.TrimSpace(what))
}
}
// SharedName is the vault's one value per provider and provision (ADR 0158, issue 677).
func SharedName(provider, provision string) string { return "provision/" + provider + "/" + provision }
// HolderName is the vault's value of one holder's own credential, issued or given for it (mesh-vault perholder.go).
func HolderName(provider, provision, node, module string) string {
return "provision/" + provider + "/" + provision + "/" + node + "/" + module
}
// PairName is the vault's value of one consumer's own login (ADR 0048, ADR 0158 §11, issue 677's note 12).
func PairName(provider, provision, node, module, local string) string {
n := "pair/" + provider + "/" + provision + "/" + node + "/" + module
if local != "" {
n += "/" + local
}
return n
}
// identityUnusable is every character a consumer's identity does not hold (mesh-controller internal/catalogue
// identity.go, the one derivation).
var identityUnusable = regexp.MustCompile(`[^a-z0-9_]+`)
// ConsumerIdentity is what one module on one machine is called wherever it authenticates: `mesh_<node>_<module>`,
// lower case, anything but letters, digits and `_` written `_` (mesh-controller ConsumerIdentity). module is the
// identity source: the module's slug when it declares one, otherwise its name (ADR 0049).
func ConsumerIdentity(node, module string) string {
clean := func(s string) string {
return strings.Trim(identityUnusable.ReplaceAllString(strings.ToLower(s), "_"), "_")
}
return "mesh_" + clean(node) + "_" + clean(module)
}
// consumerFact is `${consumer:<fact>}` or `${consumer:<fact>:<alphabet>}`.
var consumerFact = regexp.MustCompile(`\$\{consumer:([a-z][a-z0-9-]*)(?::([a-z][a-z0-9-]*))?\}`)
// ServedTo fills an offer's served values for one consumer whose identity is as (ADR 0202 §1): `${consumer:as}` as
// it is, `${consumer:as:dns}` with `-` for `_`. Only strings carry placeholders; any other fact or alphabet is
// refused, never left standing.
func ServedTo(serves map[string]any, as string) (map[string]any, error) {
if serves == nil {
return nil, nil
}
out := make(map[string]any, len(serves))
for _, key := range sortedKeys(serves) {
text, ok := serves[key].(string)
if !ok || !strings.Contains(text, "${consumer:") {
out[key] = serves[key]
continue
}
var failed error
out[key] = consumerFact.ReplaceAllStringFunc(text, func(match string) string {
parts := consumerFact.FindStringSubmatch(match)
switch {
case parts[1] == "as" && parts[2] == "":
return as
case parts[1] == "as" && parts[2] == "dns":
return strings.ReplaceAll(as, "_", "-")
}
if failed == nil {
failed = fmt.Errorf("the value served as %q says %s, and the mesh states only ${consumer:as} and "+
"${consumer:as:dns}", key, match)
}
return match
})
if failed != nil {
return nil, failed
}
}
return out, nil
}
func derives(serves map[string]any) bool { return len(derivedKeys(serves)) > 0 }
func derivedKeys(serves map[string]any) []string {
var out []string
for _, k := range sortedKeys(serves) {
if text, ok := serves[k].(string); ok && strings.Contains(text, "${consumer:") {
out = append(out, k)
}
}
return out
}
func clone(m map[string][]byte) map[string][]byte {
out := make(map[string][]byte, len(m))
for k, v := range m {
out[k] = v
}
return out
}
func sameJSON(a, b []byte) bool {
if a == nil || b == nil {
return false
}
var x, y any
if json.Unmarshal(a, &x) != nil || json.Unmarshal(b, &y) != nil {
return false
}
ja, _ := json.Marshal(x)
jb, _ := json.Marshal(y)
return string(ja) == string(jb)
}
func sortedKeys[V any](m map[string]V) []string {
if len(m) == 0 {
return nil
}
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// StateBus is the Bus of a provider that runs as a module bundle over the runtime's stdio: the provider's own bucket,
// BucketOf(module), is the module's state `bound`, and AssignmentsBucket is the state it reads,
// `mesh-controller.assignments` (both declared in the module's manifest). Any other bucket is refused.
//
// A stdio state watch lasts as long as the process and cannot be ended (stdio.KeptState.Watch), so Watch honours the
// Bus contract by silencing instead: once ctx has ended the handler is never called again, though the runtime keeps
// sending changes that are dropped. A Pass, which watches twice per call, is therefore correct, only the dropped
// changes of its two spent watches remain.
func StateBus(module string) Bus { return stateBus{module: module} }
type stateBus struct{ module string }
func (b stateBus) state(bucket string) (stdio.KeptState, error) {
switch bucket {
case BucketOf(b.module):
return stdio.State("bound"), nil
case AssignmentsBucket:
return stdio.State("mesh-controller.assignments"), nil
}
return stdio.KeptState{}, fmt.Errorf("%s is not a bucket this provider keeps or reads", bucket)
}
func (b stateBus) Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error {
s, err := b.state(bucket)
if err != nil {
return err
}
return s.Watch("", func(ch stdio.StateChange) error {
if ctx.Err() != nil {
return nil
}
if ch.Op == "delete" {
handle(ch.Key, nil)
} else {
handle(ch.Key, ch.Value)
}
return nil
})
}
func (b stateBus) Put(_ context.Context, bucket, key string, value []byte) error {
s, err := b.state(bucket)
if err != nil {
return err
}
_, err = s.Put(key, json.RawMessage(value))
return err
}
// DeleteIf removes the key only while it still holds what the caller read; it reads the key first, so another
// writer's change in that short while can still be lost.
func (b stateBus) DeleteIf(_ context.Context, bucket, key string, held []byte) error {
s, err := b.state(bucket)
if err != nil {
return err
}
cur, err := s.Get(key)
if err != nil {
return err
}
if cur == nil {
return nil
}
if !sameJSON(cur.Value, held) {
return fmt.Errorf("%s is not as it was read: %w", key, stdio.ErrChanged)
}
return s.Delete(key)
}
func (b stateBus) Delete(_ context.Context, bucket, key string) error {
s, err := b.state(bucket)
if err != nil {
return err
}
return s.Delete(key)
}
+803
View File
@@ -0,0 +1,803 @@
package provider
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"reflect"
"slices"
"strings"
"sync"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// standIn is a stand-in bus: buckets of keys and values, and watches that hand the current values first and then
// every change, as the bus's key-value watch does.
type standIn struct {
mu sync.Mutex
buckets map[string]map[string][]byte
watchers map[string][]func(string, []byte)
}
func newStandIn() *standIn {
return &standIn{buckets: map[string]map[string][]byte{}, watchers: map[string][]func(string, []byte){}}
}
func (s *standIn) Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error {
s.mu.Lock()
current := map[string][]byte{}
for k, v := range s.buckets[bucket] {
current[k] = v
}
s.watchers[bucket] = append(s.watchers[bucket], handle)
s.mu.Unlock()
for _, k := range sortedKeys(current) {
handle(k, current[k])
}
return nil
}
func (s *standIn) Put(_ context.Context, bucket, key string, value []byte) error {
s.mu.Lock()
if s.buckets[bucket] == nil {
s.buckets[bucket] = map[string][]byte{}
}
s.buckets[bucket][key] = append([]byte(nil), value...)
ws := slices.Clone(s.watchers[bucket])
s.mu.Unlock()
for _, w := range ws {
w(key, value)
}
return nil
}
func (s *standIn) Delete(_ context.Context, bucket, key string) error {
s.mu.Lock()
delete(s.buckets[bucket], key)
ws := slices.Clone(s.watchers[bucket])
s.mu.Unlock()
for _, w := range ws {
w(key, nil)
}
return nil
}
// DeleteIf removes the key only while it still holds what the caller read.
func (s *standIn) DeleteIf(ctx context.Context, bucket, key string, held []byte) error {
s.mu.Lock()
cur := s.buckets[bucket][key]
s.mu.Unlock()
if !bytes.Equal(cur, held) {
return fmt.Errorf("%s: %w", key, stdio.ErrChanged)
}
return s.Delete(ctx, bucket, key)
}
func (s *standIn) get(bucket, key string) []byte {
s.mu.Lock()
defer s.mu.Unlock()
return s.buckets[bucket][key]
}
func (s *standIn) keys(bucket string) []string {
s.mu.Lock()
defer s.mu.Unlock()
return sortedKeys(s.buckets[bucket])
}
// everything is every value the bus holds, for a test that no secret is in a bucket.
func (s *standIn) everything() []byte {
s.mu.Lock()
defer s.mu.Unlock()
var all []byte
for _, b := range s.buckets {
for k, v := range b {
all = append(all, k...)
all = append(all, v...)
}
}
return all
}
// vault is a stand-in vault: a value per name, or nothing yet.
type vault struct {
mu sync.Mutex
values map[string]string
asked []string
}
func (v *vault) Value(_ context.Context, name string) ([]byte, error) {
v.mu.Lock()
defer v.mu.Unlock()
v.asked = append(v.asked, name)
val, ok := v.values[name]
if !ok {
return nil, fmt.Errorf("the vault holds no %s yet", name)
}
return []byte(val), nil
}
// maker records what the provider's adapter was asked to make, with the credential it was handed.
type maker struct {
mu sync.Mutex
made map[string]string
// contributed is what each consumer's entry carried of its plan's contribution.
contributed map[string]map[string]any
// as is the identity each consumer's entry was made under.
as map[string]string
gone []string
fail error
calls int
}
func (m *maker) Make(_ context.Context, c Consumer) error {
m.mu.Lock()
defer m.mu.Unlock()
m.calls++
if m.fail != nil {
return m.fail
}
if m.made == nil {
m.made = map[string]string{}
}
m.made[c.Entry.Key()] = string(c.Credential)
if m.contributed == nil {
m.contributed = map[string]map[string]any{}
}
m.contributed[c.Entry.Key()] = c.Entry.Contributes
if m.as == nil {
m.as = map[string]string{}
}
m.as[c.Entry.Key()] = c.Entry.As
return nil
}
func (m *maker) Gone(_ context.Context, e Entry) error {
m.mu.Lock()
defer m.mu.Unlock()
m.gone = append(m.gone, e.Key())
return nil
}
// assign puts a consumer's assignment with its plan's provisions into the stand-in mirror.
func assign(t *testing.T, bus *standIn, node, module string, provisions ...map[string]string) {
t.Helper()
ps := []map[string]string{}
ps = append(ps, provisions...)
raw, err := json.Marshal(map[string]any{"node": node, "module": module, "assigned": "2026-10-15T00:00:00Z",
"provisioning": map[string]any{"templates": map[string]string{"source": "x", "digest": "y"},
"provisions": ps}})
if err != nil {
t.Fatal(err)
}
if err := bus.Put(context.Background(), AssignmentsBucket, node+"."+module, raw); err != nil {
t.Fatal(err)
}
}
func bind(provision, local, provider, providerModule string) map[string]string {
b := map[string]string{"provision": provision, "provider": provider, "provider-module": providerModule}
if local != "" {
b["local"] = local
}
return b
}
func entryOf(t *testing.T, bus *standIn, key string) Entry {
t.Helper()
raw := bus.get(BucketOf("postgres"), key)
if raw == nil {
t.Fatalf("no entry %s in %s; it holds %v", key, BucketOf("postgres"), bus.keys(BucketOf("postgres")))
}
var e Entry
if err := json.Unmarshal(raw, &e); err != nil {
t.Fatalf("entry %s: %v", key, err)
}
return e
}
// postgres is a provider of postgres-database on novox with a login per consumer (ADR 0048, ADR 0158 §11), its
// served values derived from each consumer's identity (ADR 0202 §1).
func postgres(bus *standIn, v *vault, m *maker) *Provider {
return New(Options{
Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Gone: m.Gone,
Offers: map[string]Offer{
"postgres-database": {Credential: PerConsumer, Serves: map[string]any{
"host": "postgres.novox.mesh", "port": 5432, "database": "${consumer:as}"}},
},
Logf: func(string, ...any) {},
})
}
// A provider answers each consumer that binds it with an entry of its own in its own bucket: composed from its
// served values and the consumer's identity, naming the credential the vault keeps for that consumer, and never
// the credential itself; its adapter is handed each consumer's own value read from the vault.
func TestEachConsumerGetsAnEntryOfItsOwnAndTheVaultsValue(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/letta": "letta-secret-1",
"pair/novox/postgres-database/g14/umami": "umami-secret-2",
}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.postgres-database",
"g14.umami.postgres-database"}) {
t.Fatalf("entries %v", got)
}
e := entryOf(t, bus, "ace.letta.postgres-database")
if e.Provider != "novox" || e.ProviderModule != "postgres" || e.Consumer != "ace" || e.Module != "letta" ||
e.As != "mesh_ace_letta" || e.User != "mesh_ace_letta" ||
e.VaultName != "pair/novox/postgres-database/ace/letta" || e.Refused != "" {
t.Fatalf("entry %+v", e)
}
if e.Values["database"] != "mesh_ace_letta" || e.Values["host"] != "postgres.novox.mesh" ||
e.Values["port"] != float64(5432) {
t.Fatalf("values %v", e.Values)
}
if m.made["ace.letta.postgres-database"] != "letta-secret-1" ||
m.made["g14.umami.postgres-database"] != "umami-secret-2" {
t.Fatalf("made %v", m.made)
}
if bytes.Contains(bus.everything(), []byte("secret-")) {
t.Fatal("a credential is in a bucket")
}
for _, o := range out {
if o.Status != Answered {
t.Fatalf("outcome %+v", o)
}
}
}
// A provider whose offer shares one credential (ADR 0158) names the vault's one value for the provision, and its
// one user name, to every consumer.
func TestASharedCredentialIsTheVaultsOneValueAndTheProvidersOneUser(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"provision/novox/sonarr-api": "one-value"}}
assign(t, bus, "ace", "overseerr", bind("sonarr-api", "", "novox", "sonarr"))
assign(t, bus, "g14", "homepage", bind("sonarr-api", "", "novox", "sonarr"))
p := New(Options{Node: "novox", Module: "sonarr", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"sonarr-api": {Credential: Shared, User: "sonarr",
Serves: map[string]any{"url": "http://sonarr.novox.mesh:8989"}}}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
for _, key := range []string{"ace.overseerr.sonarr-api", "g14.homepage.sonarr-api"} {
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("sonarr"), key), &e); err != nil {
t.Fatalf("%s: %v", key, err)
}
if e.VaultName != "provision/novox/sonarr-api" || e.User != "sonarr" || e.Values["url"] == nil {
t.Fatalf("%s: %+v", key, e)
}
if m.made[key] != "one-value" {
t.Fatalf("%s made with %q", key, m.made[key])
}
}
if bytes.Contains(bus.everything(), []byte("one-value")) {
t.Fatal("the shared credential is in a bucket")
}
}
// A consumer that keeps several holders of one provision is served no derived value: its entry refuses it, naming
// both ends, so its client can report it (ADR 0349 clause 12, ADR 0202 §10); nothing is made for it.
func TestAMultiHolderConsumerIsRefusedADerivedValueNamingBothEnds(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta/archive": "x-secret"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "archive", "novox", "postgres"))
out, err := postgres(bus, v, m).Pass(context.Background())
if err != nil {
t.Fatal(err)
}
e := entryOf(t, bus, "ace.letta.postgres-database.archive")
for _, says := range []string{"letta on ace", "postgres on novox", "postgres-database", "archive"} {
if !strings.Contains(e.Refused, says) {
t.Fatalf("the refusal does not name %q: %s", says, e.Refused)
}
}
if e.Values != nil || e.VaultName != "" {
t.Fatalf("a refused entry carries values: %+v", e)
}
if m.calls != 0 {
t.Fatal("the adapter made a refused consumer")
}
if len(out) != 1 || out[0].Status != Refused {
t.Fatalf("outcomes %+v", out)
}
}
// A consumer bound under a local name is served when the offer derives nothing from it: its credential is its own
// pair row under that local name.
func TestALocalNameIsServedWhenNothingIsDerived(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/redis-cache/ace/letta/queue": "q-secret"}}
assign(t, bus, "ace", "letta", bind("redis-cache", "queue", "novox", "redis"))
p := New(Options{Node: "novox", Module: "redis", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"redis-cache": {Credential: PerConsumer, Serves: map[string]any{"port": 6379}}}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("redis"), "ace.letta.redis-cache.queue"), &e); err != nil {
t.Fatal(err)
}
if e.Refused != "" || e.VaultName != "pair/novox/redis-cache/ace/letta/queue" || e.Local != "queue" {
t.Fatalf("entry %+v", e)
}
if m.made["ace.letta.redis-cache.queue"] != "q-secret" {
t.Fatalf("made %v", m.made)
}
}
// A consumer that unbinds loses its entry and the adapter hears it is gone; an entry another machine's instance of
// the same provider wrote is not this one's to remove.
func TestAnUnbindRemovesTheEntryAndLeavesAnotherProvidersAlone(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
other, _ := json.Marshal(Entry{Provider: "shanks", ProviderModule: "postgres", Provision: "postgres-database",
Consumer: "g14", Module: "gitea"})
_ = bus.Put(context.Background(), BucketOf("postgres"), "g14.gitea.postgres-database", other)
p := postgres(bus, v, m)
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"g14.gitea.postgres-database"}) {
t.Fatalf("entries left %v", got)
}
if !slices.Equal(m.gone, []string{"ace.letta.postgres-database"}) {
t.Fatalf("gone %v", m.gone)
}
if len(out) != 1 || out[0].Status != Removed {
t.Fatalf("outcomes %+v", out)
}
}
// A bind whose credential the vault does not hold yet is waited for: nothing is made and no entry is written, so a
// consumer is never told of a login its provider has not made; once the vault holds it, it is answered.
func TestACredentialTheVaultDoesNotHoldYetIsWaitedFor(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if len(out) != 1 || out[0].Status != Waiting || !strings.Contains(out[0].Why, "pair/novox/postgres-database/ace/letta") {
t.Fatalf("outcomes %+v", out)
}
if bus.keys(BucketOf("postgres")) != nil || m.calls != 0 {
t.Fatal("a consumer was answered before its credential was there")
}
v.values["pair/novox/postgres-database/ace/letta"] = "late"
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
if entryOf(t, bus, "ace.letta.postgres-database").Refused != "" || m.made["ace.letta.postgres-database"] != "late" {
t.Fatal("not answered once the vault held the value")
}
}
// Only the binds naming this provider, on this machine, are its own; a bind naming it for a provision it does not
// offer is refused, naming both ends.
func TestOnlyBindsNamingThisProviderAreAnswered(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "shanks", "postgres"),
bind("mongodb-database", "", "novox", "postgres"))
assign(t, bus, "ace", "other", bind("postgres-database", "", "novox", "mssql"))
out, err := postgres(bus, v, m).Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.mongodb-database"}) {
t.Fatalf("entries %v", got)
}
e := entryOf(t, bus, "ace.letta.mongodb-database")
if !strings.Contains(e.Refused, "postgres on novox") || !strings.Contains(e.Refused, "letta on ace") {
t.Fatalf("refusal %q", e.Refused)
}
if len(out) != 1 || m.calls != 0 {
t.Fatalf("outcomes %+v, %d made", out, m.calls)
}
}
// An assignment that cannot be read is skipped and said, by its key and why; the rest of the pass goes on (issue 825,
// finding 3). Its consumer's own entry is left alone, since a bind missed would read as an unbind.
func TestAnUnreadableAssignmentIsSkippedAndSaidAndTheRestGoesOn(t *testing.T) {
for name, bad := range map[string]string{"not json": "{not json",
"another key": `{"node":"g14","module":"umami"}`} {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s",
"pair/novox/postgres-database/g14/umami": "u"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
_ = bus.Put(context.Background(), AssignmentsBucket, "ace.letta", []byte(bad))
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
var logged []string
p.opts.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
out, err := p.Pass(context.Background())
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if bus.get(BucketOf("postgres"), "g14.umami.postgres-database") == nil {
t.Fatalf("%s: the good consumer was not answered", name)
}
if bus.get(BucketOf("postgres"), "ace.letta.postgres-database") == nil || len(m.gone) != 0 {
t.Fatalf("%s: the entry of the unreadable assignment's consumer was removed", name)
}
said := false
for _, o := range out {
if o.Key == "ace.letta" && o.Status == Failed && o.Why != "" {
said = true
}
}
if !said || !strings.Contains(strings.Join(logged, "\n"), "ace.letta") {
t.Fatalf("%s: the skipped assignment is not said: %+v, %q", name, out, logged)
}
}
}
// An adapter that fails is said with the credential taken out of its words, and the entry is not written.
func TestAFailedMakeIsSaidWithoutTheCredential(t *testing.T) {
bus, m := newStandIn(), &maker{fail: errors.New(`password "hunter2-secret" rejected`)}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "hunter2-secret"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
var logged []string
p := postgres(bus, v, m)
p.opts.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if len(out) != 1 || out[0].Status != Failed || strings.Contains(out[0].Why, "hunter2") {
t.Fatalf("outcomes %+v", out)
}
if strings.Contains(strings.Join(logged, "\n"), "hunter2") || len(logged) == 0 {
t.Fatalf("logged %q", logged)
}
if bus.keys(BucketOf("postgres")) != nil {
t.Fatal("an entry was written for a consumer the adapter could not make")
}
}
// Run watches the binds on the stand-in bus: a consumer that binds is answered, and one that unbinds is removed,
// without a pass asked by hand.
func TestRunAnswersBindsAndUnbindsAsTheyAreWatched(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
p := postgres(bus, v, m)
p.opts.Every = time.Hour
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- p.Run(ctx) }()
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
waitFor(t, func() bool { return bus.get(BucketOf("postgres"), "ace.letta.postgres-database") != nil })
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
waitFor(t, func() bool { return bus.get(BucketOf("postgres"), "ace.letta.postgres-database") == nil })
cancel()
if err := <-done; err != nil && !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
}
func waitFor(t *testing.T, ok func() bool) {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal("not within 3s")
}
time.Sleep(5 * time.Millisecond)
}
}
// The consumer's identity is the mesh's (ADR 0202 §1): mesh_<node>_<module>, lower case, `_` for anything else,
// and its DNS form writes `-` for `_`.
func TestTheIdentityIsTheMeshsAndItsDNSForm(t *testing.T) {
if got := ConsumerIdentity("Home.Server", "my-app"); got != "mesh_home_server_my_app" {
t.Fatal(got)
}
got, err := ServedTo(map[string]any{"bucket": "b-${consumer:as:dns}", "n": 3}, "mesh_ace_letta")
if err != nil || got["bucket"] != "b-mesh-ace-letta" || got["n"] != 3 {
t.Fatalf("%v %v", got, err)
}
if _, err := ServedTo(map[string]any{"x": "${consumer:node}"}, "mesh_ace_letta"); err == nil {
t.Fatal("a fact the mesh does not state was filled")
}
}
// A consumer's plan entry may carry what its module contributes to the provider (issue 776): the helper hands it to
// the provider's adapter with the consumer, and does not write it to the bucket.
func TestAConsumersContributionIsHandedToTheProviderPerConsumer(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/nextcloud": "a", "pair/novox/postgres-database/g14/umami": "b"}}
raw := `{"node":"ace","module":"nextcloud","provisioning":{"templates":{"source":"x","digest":"y"},` +
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
`"contributes":{"name":"nextcloud","extensions":["pg_trgm"]}}]}}`
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.nextcloud", []byte(raw)); err != nil {
t.Fatal(err)
}
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
t.Fatal(err)
}
want := map[string]any{"name": "nextcloud", "extensions": []any{"pg_trgm"}}
if got := m.contributed["ace.nextcloud.postgres-database"]; !reflect.DeepEqual(got, want) {
t.Fatalf("the adapter was handed the contribution %v, want %v", got, want)
}
if got := m.contributed["g14.umami.postgres-database"]; got != nil {
t.Fatalf("a consumer contributing nothing was handed %v", got)
}
if bytes.Contains(bus.get(BucketOf("postgres"), "ace.nextcloud.postgres-database"), []byte("pg_trgm")) {
t.Fatal("a contribution is in the provider's bucket")
}
}
// A consumer's plan entry carries where its provider is, which machine, and who the consumer is (issue 788): the
// entry's values carry `at` and `from` as the binding file gives them, and its `as` is the plan's, so a module with a
// slug keeps the login it has today. An entry of a plan without them is as it was.
func TestAConsumersEntryCarriesAtFromAndTheAsOfItsPlan(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/claude-code": "a", "pair/novox/postgres-database/g14/umami": "b"}}
raw := `{"node":"ace","module":"claude-code","provisioning":{"templates":{"source":"x","digest":"y"},` +
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
`"at":"10.7.0.1","from":"novox","as":"mesh_ace_agent"}]}}`
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.claude-code", []byte(raw)); err != nil {
t.Fatal(err)
}
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
t.Fatal(err)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.claude-code.postgres-database"), &e); err != nil {
t.Fatal(err)
}
if e.Values["at"] != "10.7.0.1" || e.Values["from"] != "novox" || e.Values["port"] == nil {
t.Fatalf("the entry's values are %v", e.Values)
}
if e.As != "mesh_ace_agent" || e.User != "mesh_ace_agent" || m.as["ace.claude-code.postgres-database"] != "mesh_ace_agent" {
t.Fatalf("the entry is %+v, made as %q", e, m.as["ace.claude-code.postgres-database"])
}
var plain Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "g14.umami.postgres-database"), &plain); err != nil {
t.Fatal(err)
}
if _, has := plain.Values["at"]; has || plain.As != "mesh_g14_umami" {
t.Fatalf("an entry of a plan without them changed: %+v", plain)
}
}
// A consumer's plan entry may carry the names the controller composed for it (issue 792): each is copied into the
// entry's values under its key, over anything the provider serves under it. An entry without them is as it was.
func TestAConsumersEntryCarriesTheNamesOfItsPlan(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/gitea": "a", "pair/novox/postgres-database/g14/umami": "b"}}
raw := `{"node":"ace","module":"gitea","provisioning":{"templates":{"source":"x","digest":"y"},` +
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
`"names":{"name":"git.example.org","internal-name":"git.mesh","name-web":"web.example.org"}}]}}`
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.gitea", []byte(raw)); err != nil {
t.Fatal(err)
}
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
t.Fatal(err)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.gitea.postgres-database"), &e); err != nil {
t.Fatal(err)
}
if e.Values["name"] != "git.example.org" || e.Values["internal-name"] != "git.mesh" ||
e.Values["name-web"] != "web.example.org" || e.Values["port"] == nil {
t.Fatalf("the entry's values are %v", e.Values)
}
var plain Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "g14.umami.postgres-database"), &plain); err != nil {
t.Fatal(err)
}
if _, has := plain.Values["name"]; has {
t.Fatalf("an entry of a plan without names changed: %+v", plain)
}
}
// refusedByTheStateGuard mirrors the key rule of the runtime's state guard (mesh-tools node-tools internal/bus
// state.go): a JSON key whose lowercased name, with `-`, `_` and spaces taken out, ends in one of these is refused.
func refusedByTheStateGuard(key string) bool {
k := strings.NewReplacer("-", "", "_", "", " ", "").Replace(strings.ToLower(key))
for _, end := range []string{"password", "passwd", "secret", "token", "credential", "credentials",
"authorization", "apikey", "privatekey", "accesskey", "cookie"} {
if strings.HasSuffix(k, end) {
return true
}
}
return false
}
func keysOf(v any, into *[]string) {
switch x := v.(type) {
case map[string]any:
for k, c := range x {
*into = append(*into, k)
keysOf(c, into)
}
case []any:
for _, c := range x {
keysOf(c, into)
}
}
}
// Every entry the helper writes passes the state guard's key rule (issue 945), and names where the vault keeps the
// credential under `vault-name`, for a login per consumer and for a shared one.
func TestAnEntryHasNoKeyTheStateGuardRefuses(t *testing.T) {
for name, form := range map[string]CredentialForm{"pair": PerConsumer, "shared": Shared} {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1",
"provision/novox/postgres-database": "s2"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"postgres-database": {Credential: form, User: "shared-user"}}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
var doc map[string]any
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.letta.postgres-database"), &doc); err != nil {
t.Fatal(err)
}
var keys []string
keysOf(doc, &keys)
for _, k := range keys {
if refusedByTheStateGuard(k) {
t.Fatalf("%s: the entry carries the key %q, which the state guard refuses", name, k)
}
}
want := map[string]string{"pair": "pair/novox/postgres-database/ace/letta",
"shared": "provision/novox/postgres-database"}[name]
if doc["vault-name"] != want {
t.Fatalf("%s: vault-name is %v, want %s", name, doc["vault-name"], want)
}
}
}
// The plan's credential form and holder name win over the offer's; the offer's form is the default (issue 825,
// finding 4).
func TestThePlansCredentialFormWinsOverTheOffers(t *testing.T) {
run := func(offer CredentialForm, plan map[string]string) Entry {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "p",
"provision/novox/postgres-database": "s", "provision/novox/postgres-database/ace/letta": "h"}}
b := bind("postgres-database", "", "novox", "postgres")
for k, val := range plan {
b[k] = val
}
assign(t, bus, "ace", "letta", b)
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"postgres-database": {Credential: offer, User: "shared-user"}}})
out, err := p.Pass(context.Background())
if err != nil || len(out) != 1 || out[0].Status != Answered {
t.Fatalf("%v %+v", err, out)
}
return entryOf(t, bus, "ace.letta.postgres-database")
}
if e := run(PerConsumer, map[string]string{"credential-form": "shared"}); e.VaultName !=
"provision/novox/postgres-database" || e.User != "shared-user" {
t.Fatalf("shared over pair: %+v", e)
}
if e := run(Shared, map[string]string{"credential-form": "pair"}); e.VaultName !=
"pair/novox/postgres-database/ace/letta" || e.User != "mesh_ace_letta" {
t.Fatalf("pair over shared: %+v", e)
}
for _, form := range []string{"issued", "given"} {
if e := run(PerConsumer, map[string]string{"credential-form": form, "name": "main"}); e.VaultName !=
"provision/novox/postgres-database/ace/letta" || e.User != "mesh_ace_letta" {
t.Fatalf("%s holder: %+v", form, e)
}
}
if e := run(Shared, nil); e.VaultName != "provision/novox/postgres-database" {
t.Fatalf("the offer's form is the default: %+v", e)
}
}
// An instance deletes only an entry it wrote, and only while the entry is as it read it: one that another instance of
// the provider wrote after the consumer moved stands (issue 825, finding 5).
func TestAnEntryAnotherInstanceWroteAfterTheMoveStands(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
a := postgres(bus, v, m)
if _, err := a.Pass(context.Background()); err != nil {
t.Fatal(err)
}
// The consumer moves to the instance on shanks: A's pass reads its own entry and the unbind, and while it lets
// the consumer go, B writes the consumer's entry.
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
theirs, _ := json.Marshal(Entry{Provider: "shanks", ProviderModule: "postgres", Provision: "postgres-database",
Consumer: "ace", Module: "letta", VaultName: "pair/shanks/postgres-database/ace/letta"})
a.opts.Gone = func(ctx context.Context, e Entry) error {
return bus.Put(ctx, BucketOf("postgres"), e.Key(), theirs)
}
out, err := a.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.get(BucketOf("postgres"), "ace.letta.postgres-database"); !bytes.Equal(got, theirs) {
t.Fatalf("the other instance's entry was removed: %s", got)
}
for _, o := range out {
if o.Status == Removed {
t.Fatalf("outcomes %+v", out)
}
}
}
// A failed Make is said with the credential taken out in the forms it can come back in: as it is, in base64 (standard
// and URL, with and without padding) and as a JSON string says it (issue 825, finding 6).
func TestAFailedMakeIsSaidWithoutTheCredentialInAnyOfItsForms(t *testing.T) {
cred := "pa/ss+w\"rd?>>x"
b := []byte(cred)
jsonQuoted, _ := json.Marshal(cred)
forms := map[string]string{
"raw": cred, "std": base64.StdEncoding.EncodeToString(b), "std unpadded": base64.RawStdEncoding.EncodeToString(b),
"url": base64.URLEncoding.EncodeToString(b), "url unpadded": base64.RawURLEncoding.EncodeToString(b),
"json": string(jsonQuoted[1 : len(jsonQuoted)-1]),
}
if forms["std"] == forms["url"] || forms["json"] == forms["raw"] {
t.Fatal("the test credential does not tell the forms apart")
}
for name, form := range forms {
bus, m := newStandIn(), &maker{fail: errors.New("login failed for " + form + " at the server")}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": cred}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
out, err := postgres(bus, v, m).Pass(context.Background())
if err != nil || len(out) != 1 {
t.Fatalf("%v %+v", err, out)
}
if strings.Contains(out[0].Why, form) || !strings.Contains(out[0].Why, "<credential>") {
t.Fatalf("%s form: %q", name, out[0].Why)
}
}
}
// An offer that declares no credential is answered with none whatever form the plan says (the controller writes
// "pair" for such an offer): no vault name, and the Vault is never asked, so a provider with no Vault works (issue 825,
// finding 4; issue 923).
func TestAnOfferWithNoCredentialIgnoresThePlansForm(t *testing.T) {
bus := newStandIn()
b := bind("route", "", "novox", "route-proxy")
b["credential-form"] = "pair"
assign(t, bus, "ace", "letta", b)
p := New(Options{Node: "novox", Module: "route-proxy", Bus: bus, Logf: func(string, ...any) {},
Offers: map[string]Offer{"route": {Credential: NoCredential}}})
out, err := p.Pass(context.Background())
if err != nil || len(out) != 1 || out[0].Status != Answered {
t.Fatalf("%v %+v", err, out)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("route-proxy"), "ace.letta.route"), &e); err != nil || e.VaultName != "" ||
e.User != "" {
t.Fatalf("%v %+v", err, e)
}
}
+152
View File
@@ -0,0 +1,152 @@
package provider
import (
"bufio"
"context"
"encoding/json"
"io"
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// runtime drives a bundle over pipes as the runtime does: it reads what the bundle says and sends what the runtime would.
type runtime struct {
t *testing.T
lines *bufio.Scanner
in io.Writer
}
func (r *runtime) next() map[string]any {
done := make(chan map[string]any, 1)
go func() {
if r.lines.Scan() {
var m map[string]any
_ = json.Unmarshal(r.lines.Bytes(), &m)
done <- m
}
}()
select {
case m := <-done:
return m
case <-time.After(3 * time.Second):
r.t.Fatal("the bundle said nothing")
return nil
}
}
func (r *runtime) send(m map[string]any) {
b, _ := json.Marshal(m)
_, _ = r.in.Write(append(b, '\n'))
}
func (r *runtime) call(id int, tool string, args map[string]any) {
r.send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "tools/call", "params": map[string]any{"name": tool, "arguments": args}})
}
func serve(t *testing.T, tools []stdio.Tool) *runtime {
t.Setenv(stdio.ServedAs, "keeper")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() { _ = stdio.ServeOn("", tools, inR, outW) }()
t.Cleanup(func() { _ = inW.Close() })
return &runtime{t: t, lines: bufio.NewScanner(outR), in: inW}
}
// The bus of a moved provider maps its own bucket to the state `bound` and the assignments to the state
// `mesh-controller.assignments`, writes only the own bucket's state, and refuses any other bucket.
func TestStateBusMapsTheProvidersBucketsToTheRuntimesStates(t *testing.T) {
bus := StateBus("postgres")
rt := serve(t, []stdio.Tool{
{Name: "watch", Run: func(a map[string]any) (any, error) {
return nil, bus.Watch(context.Background(), a["bucket"].(string), func(string, []byte) {})
}},
{Name: "put", Run: func(map[string]any) (any, error) {
return "put", bus.Put(context.Background(), BucketOf("postgres"), "k", []byte(`{"a":1}`))
}},
{Name: "delete", Run: func(map[string]any) (any, error) {
return "deleted", bus.Delete(context.Background(), BucketOf("postgres"), "k")
}},
})
for i, c := range []struct{ bucket, state string }{
{BucketOf("postgres"), "bound"},
{AssignmentsBucket, "mesh-controller.assignments"},
} {
rt.call(i+1, "watch", map[string]any{"bucket": c.bucket})
w := rt.next()
if w["method"] != "mesh/state.watch" || w["params"].(map[string]any)["state"] != c.state {
t.Fatalf("bucket %s asked %v, want the state %s", c.bucket, w, c.state)
}
rt.send(map[string]any{"jsonrpc": "2.0", "id": w["id"], "result": map[string]any{}})
rt.next()
}
rt.call(3, "put", nil)
p := rt.next()
if p["method"] != "mesh/state.put" || p["params"].(map[string]any)["state"] != "bound" || p["params"].(map[string]any)["key"] != "k" {
t.Fatalf("put asked %v", p)
}
rt.send(map[string]any{"jsonrpc": "2.0", "id": p["id"], "result": map[string]any{"revision": 1}})
rt.next()
rt.call(4, "delete", nil)
d := rt.next()
if d["method"] != "mesh/state.delete" || d["params"].(map[string]any)["state"] != "bound" {
t.Fatalf("delete asked %v", d)
}
rt.send(map[string]any{"jsonrpc": "2.0", "id": d["id"], "result": map[string]any{}})
rt.next()
for _, bucket := range []string{"other_bound", "elsewhere"} {
if err := bus.Watch(context.Background(), bucket, func(string, []byte) {}); err == nil || !strings.Contains(err.Error(), "not a bucket") {
t.Fatalf("a watch of %s gave %v", bucket, err)
}
if err := bus.Put(context.Background(), bucket, "k", nil); err == nil {
t.Fatalf("a put to %s was allowed", bucket)
}
if err := bus.Delete(context.Background(), bucket, "k"); err == nil {
t.Fatalf("a delete in %s was allowed", bucket)
}
}
}
// A watch whose ctx has ended calls its handler no more, though the runtime keeps sending changes: the Bus contract
// says a watch lasts until ctx ends, and a stdio state watch lasts as long as the process.
func TestStateBusWatchCallsItsHandlerNoMoreOnceItsContextEnded(t *testing.T) {
bus := StateBus("postgres")
ctx, cancel := context.WithCancel(context.Background())
handled := make(chan string, 8)
rt := serve(t, []stdio.Tool{
{Name: "start", Run: func(map[string]any) (any, error) {
return "started", bus.Watch(ctx, AssignmentsBucket, func(key string, value []byte) {
handled <- key + "=" + string(value)
})
}},
{Name: "stop", Run: func(map[string]any) (any, error) { cancel(); return "stopped", nil }},
})
change := func(id int, watch any, key, op string) {
rt.send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "mesh/state", "params": map[string]any{"watch": watch,
"change": map[string]any{"state": "mesh-controller.assignments", "key": key, "op": op, "value": "v"}}})
if r := rt.next(); r["id"] != float64(id) || r["error"] != nil {
t.Fatalf("a change was answered %v", r)
}
}
rt.call(1, "start", nil)
w := rt.next()
rt.send(map[string]any{"jsonrpc": "2.0", "id": w["id"], "result": map[string]any{}})
rt.next()
change(50, w["id"], "before", "put")
if got := <-handled; got != `before="v"` {
t.Fatalf("handled %q", got)
}
rt.call(2, "stop", nil)
rt.next()
change(51, w["id"], "after", "put")
change(52, w["id"], "after", "delete")
select {
case got := <-handled:
t.Fatalf("the handler was called after its context ended: %q", got)
case <-time.After(100 * time.Millisecond):
}
}
+90
View File
@@ -0,0 +1,90 @@
package stdio
import (
"encoding/json"
"errors"
"time"
)
// A module's code reads its values at runtime (novox/hq ADR 0349 clause 14, ADR 0370, and the part of the feature 57 design that lets a running module read, rather than be handed in a file, its provisioned values): its config
// values and its bindings' values that are not secret, for what its manifest declares, and watches them. The
// runtime that launched this bundle asks its machine's provisioner client **as this bundle's module** — the module it
// launched, never a name this code passes — and the client answers only what that module declares, only while the
// mesh places it on that machine. A secret is refused while the module's code runs in the tool runner: the vault
// hands a secret only to a caller the bus proves, and the bus proves only the tool runner there, so a module's
// secrets reach it through its files until it runs under a bus account of its own (novox/hq issue 569).
// ProvisionChange is what a watch hands over: the value now, or — once it is no longer answered, the module taken
// off the machine or the key no longer declared — Refused, why.
type ProvisionChange struct {
Kind string `json:"kind"`
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
Refused string `json:"refused,omitempty"`
}
const provisionTimeout = 30 * time.Second
func provisionGet(kind, key string) (json.RawMessage, error) {
c, err := live()
if err != nil {
return nil, err
}
return c.ask("mesh/provision.get", map[string]any{"kind": kind, "key": key}, provisionTimeout)
}
func provisionWatch(kind, key string, handle func(ProvisionChange) error) error {
if handle == nil {
return errors.New("a watch needs a handler")
}
c, err := live()
if err != nil {
return err
}
id := c.newID()
c.mu.Lock()
c.provisioned[id] = handle
c.mu.Unlock()
if _, err := c.askAs(id, "mesh/provision.watch", map[string]any{"kind": kind, "key": key}, provisionTimeout); err != nil {
c.mu.Lock()
delete(c.provisioned, id)
c.mu.Unlock()
return err
}
return nil
}
// Config is one of this module's config values as mesh-config resolved it for this machine, as JSON: the key as
// its manifest declares it under `config`. A key it does not declare is refused, with the reason.
func Config(key string) (json.RawMessage, error) { return provisionGet("config", key) }
// WatchConfig hands handle the config value now and then every change, in order, and returns once the value now is
// handled. The watch lasts as long as the process.
func WatchConfig(key string, handle func(ProvisionChange) error) error {
return provisionWatch("config", key, handle)
}
// Bound is one value, not a secret, of what this module is bound to: the provision as its manifest requires it,
// and the key its provider answers (`${bound:<provision>:<key>}`).
func Bound(provision, key string) (json.RawMessage, error) {
return provisionGet("bound", provision+":"+key)
}
// WatchBound is WatchConfig for a bound value.
func WatchBound(provision, key string, handle func(ProvisionChange) error) error {
return provisionWatch("bound", provision+":"+key, handle)
}
// Secret is one of this module's secrets. Refused, with the reason, while the module's code runs in the tool
// runner (novox/hq ADR 0349 §11, ADR 0370 §2): its secrets reach it through its files.
func Secret(name string) ([]byte, error) {
raw, err := provisionGet("secret", name)
if err != nil {
return nil, err
}
var s string
if err := json.Unmarshal(raw, &s); err != nil {
return nil, errors.New("the runtime answered a secret that is not a string")
}
return []byte(s), nil
}
+176
View File
@@ -0,0 +1,176 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"strings"
"testing"
"time"
)
// A Go bundle reads its values at runtime through the runtime (novox/hq ADR 0349 §14, ADR 0370, issue 685): Config,
// Bound and their watches ask the runtime, which asks as the module; a watch's value now is handled before it
// returns and each change after is named by the watch's id; what the module does not declare and every secret come
// back refused, in the runtime's words; a change for no watch of this bundle is refused.
func TestABundleReadsAndWatchesItsValuesThroughTheRuntime(t *testing.T) {
t.Setenv(ServedAs, "claude-code")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
seen := make(chan string, 16)
watched := make(chan error, 1)
answers := make(chan string, 4)
go func() {
_ = ServeOn("", []Tool{
{Name: "start", Run: func(map[string]any) (any, error) {
go func() {
watched <- WatchConfig("managed_settings", func(c ProvisionChange) error {
seen <- c.Kind + " " + c.Key + " " + string(c.Value) + c.Refused
return nil
})
}()
return "started", nil
}},
{Name: "read", Run: func(a map[string]any) (any, error) {
var v json.RawMessage
var err error
switch a["what"] {
case "config":
v, err = Config(a["key"].(string))
case "bound":
v, err = Bound("mcp-endpoint", "url")
case "secret":
var s []byte
s, err = Secret("gitea-read")
v = json.RawMessage(s)
}
if err != nil {
answers <- "refused: " + err.Error()
} else {
answers <- string(v)
}
return "read", nil
}},
}, inR, outW)
}()
lines := bufio.NewScanner(outR)
next := func() map[string]any {
done := make(chan map[string]any, 1)
go func() {
if lines.Scan() {
var m map[string]any
_ = json.Unmarshal(lines.Bytes(), &m)
done <- m
}
}()
select {
case m := <-done:
return m
case <-time.After(3 * time.Second):
t.Fatal("the bundle said nothing")
return nil
}
}
send := func(m map[string]any) { b, _ := json.Marshal(m); _, _ = inW.Write(append(b, '\n')) }
// asked is the bundle's next ask of the runtime, beside its answers to this tool call and those before.
asked := func(callID int) map[string]any {
t.Helper()
var ask map[string]any
for ask == nil {
m := next()
if m["method"] != nil {
ask = m
} else if id, ok := m["id"].(float64); !ok || id > float64(callID) || m["error"] != nil {
t.Fatalf("said %v", m)
}
}
return ask
}
// The watch: asked of the runtime with the kind and the key and no module, the module being the runtime's word.
send(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": map[string]any{"name": "start"}})
watch := asked(1)
if watch["method"] != "mesh/provision.watch" {
t.Fatalf("asked %v", watch)
}
if p := watch["params"].(map[string]any); p["kind"] != "config" || p["key"] != "managed_settings" || len(p) != 2 {
t.Fatalf("asked to watch %v", p)
}
send(map[string]any{"jsonrpc": "2.0", "id": 50, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
"change": map[string]any{"kind": "config", "key": "managed_settings", "value": map[string]any{"spinnerTipsEnabled": false}}}})
for {
if r := next(); r["id"] == float64(50) {
if r["error"] != nil {
t.Fatalf("the value now was answered %v", r)
}
break
}
}
if s := <-seen; s != `config managed_settings {"spinnerTipsEnabled":false}` {
t.Fatalf("handled %q", s)
}
select {
case err := <-watched:
t.Fatalf("the watch returned before it was answered: %v", err)
default:
}
send(map[string]any{"jsonrpc": "2.0", "id": watch["id"], "result": map[string]any{}})
if err := <-watched; err != nil {
t.Fatal(err)
}
// A change, and then the value no longer answered.
send(map[string]any{"jsonrpc": "2.0", "id": 51, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
"change": map[string]any{"kind": "config", "key": "managed_settings", "value": map[string]any{"spinnerTipsEnabled": true}}}})
if r := next(); r["id"] != float64(51) || r["error"] != nil {
t.Fatalf("a change was answered %v", r)
}
if s := <-seen; s != `config managed_settings {"spinnerTipsEnabled":true}` {
t.Fatalf("handled %q", s)
}
send(map[string]any{"jsonrpc": "2.0", "id": 52, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
"change": map[string]any{"kind": "config", "key": "managed_settings", "refused": "claude-code is not placed on g14"}}})
if r := next(); r["id"] != float64(52) || r["error"] != nil {
t.Fatalf("a refusal was answered %v", r)
}
if s := <-seen; s != `config managed_settings claude-code is not placed on g14` {
t.Fatalf("handled %q", s)
}
// A change for a watch this bundle never asked for is refused.
send(map[string]any{"jsonrpc": "2.0", "id": 53, "method": "mesh/provision", "params": map[string]any{"watch": "bundle-999",
"change": map[string]any{"kind": "config", "key": "x"}}})
if r := next(); r["id"] != float64(53) || !strings.Contains(r["error"].(map[string]any)["message"].(string), "names no watch") {
t.Fatalf("a change for no watch was answered %v", r)
}
// Each reader asks the runtime, and hands back its answer or its refusal.
for i, c := range []struct {
what, key, kind, askedKey string
result any
refusal string
want string
}{
{what: "config", key: "role", kind: "config", askedKey: "role", result: "the build machine", want: `"the build machine"`},
{what: "config", key: "nothing", kind: "config", askedKey: "nothing",
refusal: "claude-code declares no config value nothing", want: "refused: claude-code declares no config value nothing"},
{what: "bound", kind: "bound", askedKey: "mcp-endpoint:url", result: "http://127.0.0.1:9000", want: `"http://127.0.0.1:9000"`},
{what: "secret", kind: "secret", askedKey: "gitea-read",
refusal: "the vault hands a secret only to a caller the bus proves", want: "refused: the vault hands a secret only to a caller the bus proves"},
} {
id := 10 + i
send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "tools/call", "params": map[string]any{"name": "read",
"arguments": map[string]any{"what": c.what, "key": c.key}}})
get := asked(id)
if p := get["params"].(map[string]any); get["method"] != "mesh/provision.get" || p["kind"] != c.kind || p["key"] != c.askedKey || len(p) != 2 {
t.Fatalf("%s: asked %v", c.what, get)
}
if c.refusal != "" {
send(map[string]any{"jsonrpc": "2.0", "id": get["id"], "error": map[string]any{"code": -32000, "message": c.refusal}})
} else {
send(map[string]any{"jsonrpc": "2.0", "id": get["id"], "result": c.result})
}
if got := <-answers; got != c.want {
t.Errorf("%s %s: %q, want %q", c.what, c.key, got, c.want)
}
}
_ = inW.Close()
}
+203
View File
@@ -0,0 +1,203 @@
// Package seal is the mesh's sealed box (novox/hq ADR 0183, ADR 0206): a value sealed to one recipient's
// public key, which only that recipient's private key opens. X25519 for the agreement, HKDF-SHA256 for the
// key, AES-256-GCM for the box — the box the licence manager made and opened before it lived here, byte
// for byte, and the one the SDK's TypeScript `seal` makes and opens.
//
// A box is `{ v: 1, eph, iv, tag, ct }`, every field standard base64; `eph` is the one-time public key as
// SPKI DER. The key is derived with the salt eph-DER ‖ the recipient's raw public key and the info
// "novox-mesh sealed box v1", so a box cannot be re-addressed to another key.
//
// Keys travel as PEM: the public key as SPKI ("PUBLIC KEY"), the private key as PKCS #8 ("PRIVATE KEY").
// Standard library only.
package seal
import (
"crypto/aes"
"crypto/cipher"
"crypto/ecdh"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"encoding/pem"
"errors"
"fmt"
)
// Box is a value sealed to one recipient, as it travels in JSON.
type Box struct {
V int `json:"v"`
Eph string `json:"eph"`
IV string `json:"iv"`
Tag string `json:"tag"`
Ct string `json:"ct"`
}
// KeyPair is a recipient's keypair as the two PEM strings it is kept and sent as.
type KeyPair struct {
PublicKey string `json:"publicKey"`
PrivateKey string `json:"privateKey"`
}
// Version is the only box version Open opens.
const Version = 1
const info = "novox-mesh sealed box v1"
// GenerateKeyPair makes a fresh X25519 keypair, PEM-encoded.
func GenerateKeyPair() (KeyPair, error) {
priv, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
return KeyPair{}, err
}
pubDER, err := x509.MarshalPKIXPublicKey(priv.PublicKey())
if err != nil {
return KeyPair{}, err
}
privDER, err := x509.MarshalPKCS8PrivateKey(priv)
if err != nil {
return KeyPair{}, err
}
return KeyPair{
PublicKey: string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: pubDER})),
PrivateKey: string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: privDER})),
}, nil
}
// Seal seals plaintext to the recipient's PEM public key, with a fresh one-time key and iv.
func Seal(plaintext, recipientPEM string) (Box, error) {
eph, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
return Box{}, err
}
iv := make([]byte, 12)
if _, err := rand.Read(iv); err != nil {
return Box{}, err
}
return sealWith(plaintext, recipientPEM, eph, iv)
}
// sealWith is Seal with its two sources of randomness given, for the byte-for-byte tests.
func sealWith(plaintext, recipientPEM string, eph *ecdh.PrivateKey, iv []byte) (Box, error) {
recipient, err := publicFromPEM(recipientPEM)
if err != nil {
return Box{}, err
}
secret, err := eph.ECDH(recipient)
if err != nil {
return Box{}, err
}
ephDER, err := x509.MarshalPKIXPublicKey(eph.PublicKey())
if err != nil {
return Box{}, err
}
gcm, err := newGCM(boxKey(secret, ephDER, recipient.Bytes()))
if err != nil {
return Box{}, err
}
out := gcm.Seal(nil, iv, []byte(plaintext), nil)
ct, tag := out[:len(out)-gcm.Overhead()], out[len(out)-gcm.Overhead():]
b64 := base64.StdEncoding.EncodeToString
return Box{V: Version, Eph: b64(ephDER), IV: b64(iv), Tag: b64(tag), Ct: b64(ct)}, nil
}
// Open opens a box with the recipient's PEM private key. It fails for a box sealed to another key, one
// changed in any field, or one of another version.
func Open(box Box, privatePEM string) (string, error) {
if box.V != Version {
return "", fmt.Errorf("a sealed box of version %d, not %d", box.V, Version)
}
priv, err := privateFromPEM(privatePEM)
if err != nil {
return "", err
}
d := base64.StdEncoding.DecodeString
ephDER, err := d(box.Eph)
if err != nil {
return "", fmt.Errorf("the box's eph: %w", err)
}
ephKey, err := x509.ParsePKIXPublicKey(ephDER)
if err != nil {
return "", fmt.Errorf("the box's eph: %w", err)
}
eph, ok := ephKey.(*ecdh.PublicKey)
if !ok || eph.Curve() != ecdh.X25519() {
return "", errors.New("the box's eph is not an X25519 key")
}
secret, err := priv.ECDH(eph)
if err != nil {
return "", err
}
iv, err1 := d(box.IV)
tag, err2 := d(box.Tag)
ct, err3 := d(box.Ct)
if err := errors.Join(err1, err2, err3); err != nil {
return "", fmt.Errorf("the box's fields: %w", err)
}
gcm, err := newGCM(boxKey(secret, ephDER, priv.PublicKey().Bytes()))
if err != nil {
return "", err
}
if len(iv) != gcm.NonceSize() || len(tag) != gcm.Overhead() {
return "", errors.New("the box does not open with this key")
}
plain, err := gcm.Open(nil, iv, append(append([]byte{}, ct...), tag...), nil)
if err != nil {
return "", errors.New("the box does not open with this key")
}
return string(plain), nil
}
func publicFromPEM(p string) (*ecdh.PublicKey, error) {
block, _ := pem.Decode([]byte(p))
if block == nil {
return nil, errors.New("not a PEM public key")
}
k, err := x509.ParsePKIXPublicKey(block.Bytes)
if err != nil {
return nil, err
}
pub, ok := k.(*ecdh.PublicKey)
if !ok || pub.Curve() != ecdh.X25519() {
return nil, errors.New("not an X25519 public key")
}
return pub, nil
}
func privateFromPEM(p string) (*ecdh.PrivateKey, error) {
block, _ := pem.Decode([]byte(p))
if block == nil {
return nil, errors.New("not a PEM private key")
}
k, err := x509.ParsePKCS8PrivateKey(block.Bytes)
if err != nil {
return nil, err
}
priv, ok := k.(*ecdh.PrivateKey)
if !ok || priv.Curve() != ecdh.X25519() {
return nil, errors.New("not an X25519 private key")
}
return priv, nil
}
// boxKey is HKDF-SHA256 (RFC 5869) for one 32-byte block: written out because crypto/hkdf needs Go 1.24
// and this module promises 1.22; the vectors prove it equals the licence manager's hkdf.Key.
func boxKey(secret, ephDER, recipientRaw []byte) []byte {
salt := append(append([]byte{}, ephDER...), recipientRaw...)
extract := hmac.New(sha256.New, salt)
extract.Write(secret)
prk := extract.Sum(nil)
expand := hmac.New(sha256.New, prk)
expand.Write([]byte(info))
expand.Write([]byte{1})
return expand.Sum(nil)
}
func newGCM(key []byte) (cipher.AEAD, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
return cipher.NewGCM(block)
}
+156
View File
@@ -0,0 +1,156 @@
package seal
import (
"crypto/ecdh"
"encoding/base64"
"encoding/json"
"os"
"testing"
)
type fixture struct {
Recipient KeyPair `json:"recipient"`
Sealed []struct {
Plaintext string `json:"plaintext"`
Box Box `json:"box"`
} `json:"sealed"`
Vectors []struct {
Plaintext string `json:"plaintext"`
EphPrivate string `json:"ephPrivate"`
IV string `json:"iv"`
Box Box `json:"box"`
} `json:"vectors"`
}
func readFixture(t *testing.T) fixture {
t.Helper()
raw, err := os.ReadFile("testdata/sealed-by-licence-manager.json")
if err != nil {
t.Fatal(err)
}
var f fixture
if err := json.Unmarshal(raw, &f); err != nil {
t.Fatal(err)
}
if len(f.Sealed) == 0 || len(f.Vectors) == 0 {
t.Fatal("the fixture holds no boxes")
}
return f
}
// Every box the licence manager's own Seal made opens here to its plaintext.
func TestABoxTheLicenceManagerSealedOpens(t *testing.T) {
f := readFixture(t)
for _, s := range f.Sealed {
got, err := Open(s.Box, f.Recipient.PrivateKey)
if err != nil || got != s.Plaintext {
t.Fatalf("opened %q, %v; want %q", got, err, s.Plaintext)
}
}
}
// Given the same one-time key and iv, this package seals byte for byte what the licence manager sealed:
// the reverse direction, so whatever opens the licence manager's boxes opens these.
func TestSealIsByteForByteTheLicenceManagersBox(t *testing.T) {
f := readFixture(t)
for _, v := range f.Vectors {
d := base64.StdEncoding.DecodeString
ephRaw, err1 := d(v.EphPrivate)
iv, err2 := d(v.IV)
if err1 != nil || err2 != nil {
t.Fatal(err1, err2)
}
eph, err := ecdh.X25519().NewPrivateKey(ephRaw)
if err != nil {
t.Fatal(err)
}
got, err := sealWith(v.Plaintext, f.Recipient.PublicKey, eph, iv)
if err != nil {
t.Fatal(err)
}
if got != v.Box {
t.Fatalf("sealed %+v\nthe licence manager sealed %+v", got, v.Box)
}
if plain, err := Open(got, f.Recipient.PrivateKey); err != nil || plain != v.Plaintext {
t.Fatalf("reopened %q, %v", plain, err)
}
}
}
// A box the agent module's TypeScript sealed (the fixture the licence manager keeps) opens here too.
func TestABoxSealedInTypeScriptOpens(t *testing.T) {
raw, err := os.ReadFile("testdata/sealed-by-typescript.json")
if err != nil {
t.Fatal(err)
}
var f struct {
PrivateKey string `json:"privateKey"`
Box Box `json:"box"`
Plaintext string `json:"plaintext"`
}
if err := json.Unmarshal(raw, &f); err != nil {
t.Fatal(err)
}
if got, err := Open(f.Box, f.PrivateKey); err != nil || got != f.Plaintext {
t.Fatalf("opened %q, %v", got, err)
}
}
// What Seal makes opens with its recipient's key and no other, and a fresh box differs every time.
func TestABoxOpensOnlyForItsRecipient(t *testing.T) {
a, err := GenerateKeyPair()
if err != nil {
t.Fatal(err)
}
b, _ := GenerateKeyPair()
box, err := Seal("a token", a.PublicKey)
if err != nil {
t.Fatal(err)
}
if got, err := Open(box, a.PrivateKey); err != nil || got != "a token" {
t.Fatalf("opened %q, %v", got, err)
}
if _, err := Open(box, b.PrivateKey); err == nil {
t.Fatal("a box opened for another key")
}
again, _ := Seal("a token", a.PublicKey)
if again.Eph == box.Eph || again.IV == box.IV {
t.Fatal("two boxes share a one-time key or iv")
}
}
// A box changed in any field, or of another version, does not open.
func TestATamperedBoxDoesNotOpen(t *testing.T) {
f := readFixture(t)
good := f.Sealed[1].Box
flip := func(s string) string {
b, _ := base64.StdEncoding.DecodeString(s)
b[len(b)-1] ^= 1
return base64.StdEncoding.EncodeToString(b)
}
for name, box := range map[string]Box{
"ct": {V: 1, Eph: good.Eph, IV: good.IV, Tag: good.Tag, Ct: flip(good.Ct)},
"tag": {V: 1, Eph: good.Eph, IV: good.IV, Tag: flip(good.Tag), Ct: good.Ct},
"iv": {V: 1, Eph: good.Eph, IV: flip(good.IV), Tag: good.Tag, Ct: good.Ct},
"eph": {V: 1, Eph: flip(good.Eph), IV: good.IV, Tag: good.Tag, Ct: good.Ct},
"version": {V: 2, Eph: good.Eph, IV: good.IV, Tag: good.Tag, Ct: good.Ct},
} {
if _, err := Open(box, f.Recipient.PrivateKey); err == nil {
t.Fatalf("a box with a changed %s opened", name)
}
}
}
// A key that is not X25519 PEM is refused, not misread.
func TestAKeyOfAnotherKindIsRefused(t *testing.T) {
if _, err := Seal("x", "not a key"); err == nil {
t.Fatal("sealed to text")
}
f := readFixture(t)
if _, err := Open(f.Sealed[0].Box, f.Recipient.PublicKey); err == nil {
t.Fatal("opened with a public key")
}
if _, err := Seal("x", f.Recipient.PrivateKey); err == nil {
t.Fatal("sealed to a private key")
}
}
+121
View File
@@ -0,0 +1,121 @@
{
"about": "Made by the claude-licence-manager's seal.go (mesh-catalog feat/57-config-and-provisioning at 8aa37e8), unchanged but for its two sources of randomness, which the vectors fix. The recipient is a test key derived from a fixed label; it guards nothing.",
"recipient": {
"publicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VuAyEAkZXnghyfA1BR5rWtvBLmjdZMCbC99xxF3WKHe1E+3XM=\n-----END PUBLIC KEY-----\n",
"privateKey": "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VuBCIEIHjV954xXPa72FyGUw5PsWCbsHrliVp/7xoAgG9fGs7q\n-----END PRIVATE KEY-----\n"
},
"sealed": [
{
"plaintext": "",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAwN2gt32hv+9VyFh0oQ8oetPBHsZ9dt80DzwPwcB0bHg=",
"iv": "0xHCegxP6o48EGLq",
"tag": "VYQZm/lBZB0YMPUqYVPKaQ==",
"ct": ""
}
},
{
"plaintext": "a grant sealed by the licence manager",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEA54v96LEb8SwT2uJZCa90/2jcLvEbK/BZpbMS6TxAAW0=",
"iv": "xm6Jg4oOnRpRpDGK",
"tag": "cmbOzIV63Nv0lPfpTgP2kg==",
"ct": "InWe8PZ8hkd7F9QaHujjhdiBECKzLAK22d2aIEXwBhtxIowc/Q=="
}
},
{
"plaintext": "sk-ant-api03-test-0123456789abcdefghij\n",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAIXdum2FYLzvLDj/Of2253hH8Lu0szU4tLxJzfXTahDs=",
"iv": "KMPNa60C/Qjhtp0Z",
"tag": "PPa82ZdFyWz9kbLSSGkl3g==",
"ct": "Phu3ZsxIWGlBWbYOGzPopncEbc0VhTAloxzeHs4KDBMFZU9Pet2V"
}
},
{
"plaintext": "ünïcødé ✓ and a NUL \u0000 inside",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAo1WaRyrAJFM6If/cpFgdpZjsC6XebuJtVSugbI+m8RU=",
"iv": "H9+WvM6ZYbCYJj6D",
"tag": "K5p986nSpeItYK1m4bcVPQ==",
"ct": "eyZs3uVeZguNbqSmrMc1rWgxhW3nP0GABH6Ek06/8UuBqw=="
}
},
{
"plaintext": "a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; ",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAzjZdTMu08I0Z+FWXSr9f++EADfqg3qr2FVt+3Q/hxm0=",
"iv": "lTG2VwPA5SwHbZE/",
"tag": "Il6ICxoZgZr9dOgeIZOPrg==",
"ct": "ZWQ8m1nxrqq0oIGRoeNlUu9/JpTMrxlncTCL5cDqsJrx9NjS0nZX292JJ5p0+BYSVLUN+yGUuBOnH860ywvKK7XhPOc0Yo2nq7X2O3OYn9Za4y+HK6dVDM8zhsS8P3Di/MJCwS+AnremVCaj2k3xHL3oc331RBZZ5w2JfgSX3ouwl2+VbIPyJVSDsxnLLoHsiIR7nB0GNCe0fynhVkoIEwC8IXLnL6m3rjjN3YCQDmVP2UoftniLAypd5ioc9GCtpvUL6ktjs82JkvEtFbHtc+YzIjP6ZnjTj/bvbRHkIqN4MDiu+fvGOr0jtGWO24ecGC+6KtrKGOaEYHDqOLaK3c+UcrDsFYC+cuKCdCNhs5UNhYYylVXE648KFa488eFk"
}
}
],
"vectors": [
{
"plaintext": "",
"ephPrivate": "W8iT0SrRDnazANHYsXHC72sM1wzx+/Y8vowREeGGz/A=",
"iv": "DD7WV6Y8hLGvaG0b",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAdCCqlFhxbKBpv1JClXceH43AqEvcaM6SGfyXx05UJUs=",
"iv": "DD7WV6Y8hLGvaG0b",
"tag": "ctb+ZQjx8FyynvT88rfwkQ==",
"ct": ""
}
},
{
"plaintext": "a grant sealed by the licence manager",
"ephPrivate": "lzIbkYmIbkKTYDL4dZiAFUFv2h+GWBiqlQfnsVD2An8=",
"iv": "xz2FIBXNM3scOAJb",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAS3L49p4KrGMo/NmHZq8osoPbk2ZCQWLoNIKnpB4T5hg=",
"iv": "xz2FIBXNM3scOAJb",
"tag": "7dC8bYqzsJ/17EC7y6lhCA==",
"ct": "FEMfh4HMvMwqx4QE4YMMR3d3g5rNuxa/+u6u+i/aRLed3q8kBw=="
}
},
{
"plaintext": "sk-ant-api03-test-0123456789abcdefghij\n",
"ephPrivate": "tAjRQTQHk6nNJLblzdslDbWmyYm8cE6/G4ipXHR0zlE=",
"iv": "BPF1JDQ5mDNhKKr3",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEA1GianODxSXUHYMdUmZLcQztp5ry2/xtwUSeHZWrGFAo=",
"iv": "BPF1JDQ5mDNhKKr3",
"tag": "uiIIFD7LgPmLV3Y6uleSzg==",
"ct": "+maa0GDDOkcyzD0chFQ4a/0NvBrvl23NVqcGv1BEKgmMUenf9HHd"
}
},
{
"plaintext": "ünïcødé ✓ and a NUL \u0000 inside",
"ephPrivate": "QBYUxuk4jwiWKOPEWeE7v6WxY8BDmZZCDwlY1WJ1PSA=",
"iv": "u76JO8baqokfnjcK",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAePakOoCmmXLtnN3EykbpqfdbfXpkg04F2djxnJWH220=",
"iv": "u76JO8baqokfnjcK",
"tag": "sfBPRnwhfP/p7vlsUx+e9w==",
"ct": "B9yc0FbdKYDtDeX1jmKa+DBP8fPN1O9SbeYBDn9b3+Bq2Q=="
}
},
{
"plaintext": "a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; ",
"ephPrivate": "2VYOyzqLrb+zWUhVpHA9aV3vMAyRWlNPEq2ASf60F9M=",
"iv": "0Gv6ejq07LwbM8cY",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEARSOBczrJ6X1MkxT7Mrd1ryDLmXAMBgYIgpfwBjHz1GA=",
"iv": "0Gv6ejq07LwbM8cY",
"tag": "Vm+E0myXgdMwA6knJDg/+w==",
"ct": "tJwZpUXeQYOu2PG1g9AwIV8RhKKDorUHWQSygEus3xHOpuTwh3+D4j2htIerMB/3AKToQYABeZMyEAOzhL2x+Cg06WlUKrggZjvdX5W7X813KQBgzg0YnnCZTKTYlviDWe1E9TZ3aAauhlcgpBaQXUh/akkDiqeq2Ua5ruY6IGH5HgqWWPYJo3b4zkbArhvdP2HJX9Pn1mQa0BtGMinyutyD0am5vZuemdKGYS1Ao00M81sMXYc2CBy9K48Ofw8GapcmYNHsE0s6NSgNZumnVaR8TQRy/oqEdL3Q6z31hAZMzTdS0ckXg1NCgcgXVuzwNov/D4320E3DxFS3v8Z5p74ZkqAmo/1JcCNJwwQyRCWk/4iJFjP1OKEavOGo+bcp"
}
}
]
}
+12
View File
@@ -0,0 +1,12 @@
{
"privateKey": "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VuBCIEIAi2NK/bN+p7cqYUwv/kz72TgLdmJUfOHCDZTrMpQzpS\n-----END PRIVATE KEY-----\n",
"publicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VuAyEARYvD/w+9ah0KWS9T9pd6Ea6CBymUE48vEVk983QPKyY=\n-----END PUBLIC KEY-----\n",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAIYlSGrJvF8qSjR1aTFWbEQM/NFbZXrarglN0aRLZZ0E=",
"iv": "ABqT/hMukn6+xpjh",
"tag": "POzmsWTPHSn9xLMMJJ9Akg==",
"ct": "m2u0kuQ0PwrsGelM99Z5aBw6FCd6lFISUbwiK5TzzDw4ZrGtsHEvxytoIeFC"
},
"plaintext": "a grant sealed by the TypeScript agent module"
}
+25
View File
@@ -31,6 +31,16 @@ type StateChange struct {
Current bool `json:"current"`
}
// StateRevision is one value a bucket still keeps for a key: what was written, at which revision, by which
// operation ("put" or "delete", whose value is null) and when (novox/hq ADR 0307).
type StateRevision struct {
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
Op string `json:"op"`
At time.Time `json:"at"`
}
// KeptState is one state this module keeps, or reads.
type KeptState struct{ name string }
@@ -58,6 +68,21 @@ func (s KeptState) Get(key string) (*StateEntry, error) {
return &e, json.Unmarshal(raw, &e)
}
// History is every value the bucket still keeps for key, oldest first — as many as the state declared it
// keeps, deletes included — and empty for a key it holds nothing for (novox/hq ADR 0307). A read, like Get:
// the owner's instances and its readers may ask it, and it writes nothing.
func (s KeptState) History(key string) ([]StateRevision, error) {
raw, err := s.ask("history", map[string]any{"key": key})
if err != nil {
return nil, err
}
revisions := []StateRevision{}
if len(raw) == 0 || string(raw) == "null" {
return revisions, nil
}
return revisions, json.Unmarshal(raw, &revisions)
}
// Put writes a key — the owner's instances only — and returns the revision written.
func (s KeptState) Put(key string, value any) (uint64, error) {
raw, err := s.ask("put", map[string]any{"key": key, "value": value})
+142
View File
@@ -0,0 +1,142 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"strings"
"sync"
"testing"
"time"
)
// historying serves a bundle whose one tool, "history", runs run, answering each mesh/state ask with
// answer beside the loop, as the runtime does; it returns what the bundle asked.
func historying(t *testing.T, answer func(message) map[string]any, run func()) []message {
t.Helper()
t.Setenv(ServedAs, "issues")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() {
_ = ServeOn("", []Tool{{Name: "history", Run: func(map[string]any) (any, error) { run(); return "done", nil }}}, inR, outW)
}()
var wmu, amu sync.Mutex
var asked []message
send := func(m map[string]any) {
m["jsonrpc"] = "2.0"
b, _ := json.Marshal(m)
wmu.Lock()
defer wmu.Unlock()
_, _ = inW.Write(append(b, '\n'))
}
done := make(chan struct{})
go func() {
lines := bufio.NewScanner(outR)
for lines.Scan() {
var m message
if json.Unmarshal(lines.Bytes(), &m) != nil {
continue
}
if strings.HasPrefix(m.Method, "mesh/state.") {
amu.Lock()
asked = append(asked, m)
amu.Unlock()
a := answer(m)
a["id"] = m.ID
send(a)
continue
}
if m.Method == "" {
close(done)
return
}
}
}()
send(map[string]any{"id": 1, "method": "tools/call", "params": map[string]any{"name": "history"}})
t.Cleanup(func() { _ = inW.Close() })
select {
case <-done:
case <-time.After(10 * time.Second):
t.Fatal("the bundle did not answer")
}
amu.Lock()
defer amu.Unlock()
return asked
}
// History asks mesh/state.history with the state and the key alone, and hands back every kept value oldest
// first, a delete with its null value; nothing kept is an empty history; a refusal is the runtime's error
// (novox/hq ADR 0307).
func TestHistoryIsEveryKeptValueOldestFirst(t *testing.T) {
kept := []map[string]any{
{"key": "401", "value": map[string]any{"status": "open"}, "revision": 4, "op": "put", "at": "2026-10-10T19:46:00Z"},
{"key": "401", "value": map[string]any{"status": "fixing"}, "revision": 9, "op": "put", "at": "2026-10-11T08:01:02Z"},
{"key": "401", "value": nil, "revision": 12, "op": "delete", "at": "2026-10-11T09:30:00+02:00"},
}
var got, none, null []StateRevision
var refused error
asked := historying(t, func(m message) map[string]any {
var p map[string]any
_ = json.Unmarshal(m.Params, &p)
switch {
case p["state"] != "issues":
return map[string]any{"error": map[string]any{"code": -32000, "message": "issues was issued no state " + p["state"].(string)}}
case p["key"] == "401":
return map[string]any{"result": kept}
case p["key"] == "402":
return map[string]any{"result": nil}
}
return map[string]any{"result": []any{}}
}, func() {
var err error
if got, err = State("issues").History("401"); err != nil {
t.Errorf("history of 401: %v", err)
}
if none, err = State("issues").History("999"); err != nil {
t.Errorf("history of 999: %v", err)
}
if null, err = State("issues").History("402"); err != nil {
t.Errorf("history of 402: %v", err)
}
var h []StateRevision
h, refused = State("elsewhere.issues").History("401")
if h != nil {
t.Errorf("a refused history handed back %v", h)
}
})
if len(asked) != 4 {
t.Fatalf("asked %d times, not 4", len(asked))
}
var p map[string]any
_ = json.Unmarshal(asked[0].Params, &p)
if asked[0].Method != "mesh/state.history" || p["state"] != "issues" || p["key"] != "401" || len(p) != 2 {
t.Fatalf("asked %s %v", asked[0].Method, p)
}
if len(got) != 3 {
t.Fatalf("history of 401 is %d values, not 3: %+v", len(got), got)
}
want := []struct {
rev uint64
op string
value string
at time.Time
}{
{4, "put", `{"status":"open"}`, time.Date(2026, 10, 10, 19, 46, 0, 0, time.UTC)},
{9, "put", `{"status":"fixing"}`, time.Date(2026, 10, 11, 8, 1, 2, 0, time.UTC)},
{12, "delete", `null`, time.Date(2026, 10, 11, 7, 30, 0, 0, time.UTC)},
}
for i, w := range want {
r := got[i]
if r.Key != "401" || r.Revision != w.rev || r.Op != w.op || string(r.Value) != w.value || !r.At.Equal(w.at) {
t.Fatalf("value %d is %+v, not %+v", i, r, w)
}
}
if none == nil || len(none) != 0 || null == nil || len(null) != 0 {
t.Fatalf("nothing kept read as %v and %v, not an empty history", none, null)
}
if refused == nil || !strings.Contains(refused.Error(), "issued no state elsewhere.issues") {
t.Fatalf("a state not issued answered %v", refused)
}
}
+56 -10
View File
@@ -39,7 +39,8 @@ type Tool struct {
Description string
// Input is the JSON schema of the arguments; nil is a tool that takes nothing.
Input map[string]any
Run func(args map[string]any) (any, error)
// Run answers one call. Who called is CallerOf(args): the bus user the runtime proved, or Unproven.
Run func(args map[string]any) (any, error)
}
type message struct {
@@ -56,7 +57,11 @@ type message struct {
// channel is the one conversation with the runtime: what this bundle writes, serialised, and what
// it asked and waits for.
type channel struct {
mu sync.Mutex
mu sync.Mutex
// wmu orders the writes to the runtime and nothing else: a write waits on the pipe for as long as
// the runtime does not read, and mu must never wait with it, or the loop that reads the runtime's
// answers stops (issue 1286).
wmu sync.Mutex
enc *json.Encoder
next int
waiting map[string]chan message
@@ -65,6 +70,8 @@ type channel struct {
subscribed int
// watches are this bundle's state watches by the id each was asked with (ADR 0201).
watches map[string]func(StateChange) error
// provisioned are this bundle's watches of its values, by the id each was asked with (provision.go).
provisioned map[string]func(ProvisionChange) error
// takers and provers are this bundle's seat work and proof handlers, by worker and subject (ADR 0259).
takers map[string]func(Work) error
provers map[string]func(subject string, body json.RawMessage) (any, error)
@@ -87,8 +94,8 @@ type Envelope struct {
func (c *channel) say(m map[string]any) {
m["jsonrpc"] = "2.0"
c.mu.Lock()
defer c.mu.Unlock()
c.wmu.Lock()
defer c.wmu.Unlock()
_ = c.enc.Encode(m)
}
@@ -168,7 +175,8 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
byName[t.Name] = t
}
c := &channel{enc: json.NewEncoder(out), waiting: map[string]chan message{}, watches: map[string]func(StateChange) error{},
takers: map[string]func(Work) error{}, provers: map[string]func(string, json.RawMessage) (any, error){}}
provisioned: map[string]func(ProvisionChange) error{},
takers: map[string]func(Work) error{}, provers: map[string]func(string, json.RawMessage) (any, error){}}
currentMu.Lock()
current = c
currentMu.Unlock()
@@ -223,6 +231,8 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
var p struct {
Name string `json:"name"`
Arguments map[string]any `json:"arguments"`
// Meta is the runtime's, never the caller's: who called, taken from the subject (CallerOf).
Meta map[string]any `json:"_meta"`
}
_ = json.Unmarshal(m.Params, &p)
t, ok := byName[p.Name]
@@ -232,18 +242,18 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
}
// Beside the loop, never in it: a tool that emits waits for the runtime's answer, which
// arrives on the line this loop reads next — and calls do not queue behind a slow one.
go func(id any, t Tool, args map[string]any) {
go func(id any, t Tool, args map[string]any, caller string) {
if args == nil {
args = map[string]any{}
}
result, err := t.Run(args)
result, err := runAs(t, caller, args)
if err != nil {
c.say(map[string]any{"id": id, "result": map[string]any{"content": []map[string]any{{"type": "text", "text": err.Error()}}, "isError": true}})
return
}
text, _ := json.Marshal(result)
c.say(map[string]any{"id": id, "result": map[string]any{"content": []map[string]any{{"type": "text", "text": string(text)}}}})
}(id, t, p.Arguments)
}(id, t, p.Arguments, callerIn(p.Meta))
case "mesh/event":
// One event the runtime read from this module's consumer (ADR 0198), answered once every
// handler it is for has taken it; an error is answered as one, so the runtime does not
@@ -294,6 +304,28 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
}
c.say(map[string]any{"id": id, "result": map[string]any{}})
}(id, *p.Change)
case "mesh/provision":
// One change to a value of this module's that this bundle watches (provision.go), named by the
// watch's id, answered once its handler has taken it. Beside the loop, like a tool call.
var p struct {
Watch string `json:"watch"`
Change *ProvisionChange `json:"change"`
}
_ = json.Unmarshal(m.Params, &p)
c.mu.Lock()
handle := c.provisioned[p.Watch]
c.mu.Unlock()
if handle == nil || p.Change == nil {
c.say(map[string]any{"id": id, "error": map[string]any{"code": -32602, "message": "mesh/provision names no watch of this bundle"}})
continue
}
go func(id any, change ProvisionChange) {
if err := handle(change); err != nil {
c.say(map[string]any{"id": id, "error": map[string]any{"code": -32000, "message": err.Error()}})
return
}
c.say(map[string]any{"id": id, "result": map[string]any{}})
}(id, *p.Change)
case "mesh/work":
// One piece of a seat's work this bundle takes (novox/hq ADR 0259): taken by an answer that is
// not an error, offered again after one. Beside the loop, like a tool call.
@@ -444,10 +476,15 @@ func matchFrom(p []string, pi int, k []string, ki int) bool {
return matchFrom(p, pi+1, k, ki+1)
}
// EmitOptions is the provenance an emitter may attach (ADR 0042).
// EmitOptions is the provenance an emitter may attach (ADR 0042), and the message id it may choose.
type EmitOptions struct {
CausationID string
Schema string
// MessageID is the id this emit is sent under, as x-event-id (and, by the tool runner, as the bus's
// own Nats-Msg-Id): chosen by the emitter so a repeated publish inside the stream's duplicate window
// is kept to one, never generated once an id is given (novox/hq ADR 0436 clause 4). Empty is a random
// one, as on main.
MessageID string
}
// Emit asks the runtime to publish an event as this bundle's module, and returns once the bus has
@@ -467,8 +504,17 @@ func Emit(eventType string, body any, opts ...EmitOptions) error {
if node == "" {
node = "unknown"
}
id := ""
for _, o := range opts {
if o.MessageID != "" {
id = o.MessageID
}
}
if id == "" {
id = eventID()
}
headers := map[string]string{
"x-event-id": eventID(), "x-source": source, "x-node": node,
"x-event-id": id, "x-source": source, "x-node": node,
"x-time": time.Now().UTC().Format(time.RFC3339Nano), "content-type": "application/json",
}
for _, o := range opts {
+51
View File
@@ -224,3 +224,54 @@ func TestARefusedSubscribeLeavesNoHandlerBehind(t *testing.T) {
t.Fatalf("one event was handled %d times", handled)
}
}
// A message the runtime has not read yet does not stop the channel taking its answers: with one tool
// answer blocked on the pipe, a refused subscribe is still delivered to Subscribe, which returns the
// refusal — no lock of the channel is held across a write to the pipe (novox/hq issue 1286/AC1, 1286/AC2).
func TestARefusedSubscribeIsDeliveredWhileAMessageWaitsOnThePipe(t *testing.T) {
t.Setenv(ServedAs, "plex")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
defer func() { _ = inW.Close() }()
// Not the channel an earlier run left behind: Subscribe goes to the one this run serves.
currentMu.Lock()
current = nil
currentMu.Unlock()
go func() {
_ = ServeOn("", []Tool{{Name: "quick", Run: func(map[string]any) (any, error) { return "ok", nil }}}, inR, outW)
}()
subscribed := make(chan error, 1)
go func() {
for {
if _, err := live(); err == nil {
break
}
time.Sleep(time.Millisecond)
}
subscribed <- Subscribe("*.download.completed", func(Envelope) error { return nil })
}()
go func() {
lines := bufio.NewScanner(outR)
if !lines.Scan() {
return
}
var req map[string]any
_ = json.Unmarshal(lines.Bytes(), &req)
send := func(m map[string]any) { b, _ := json.Marshal(m); _, _ = inW.Write(append(b, '\n')) }
// A tool answer the runtime has not read: its write waits on the pipe, and no one reads further.
send(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": map[string]any{"name": "quick"}})
time.Sleep(100 * time.Millisecond)
send(map[string]any{"jsonrpc": "2.0", "id": req["id"], "error": map[string]any{"code": -32000, "message": "not yet"}})
}()
deadline := time.After(2 * time.Second)
select {
case err := <-subscribed:
if err == nil || err.Error() != "not yet" {
t.Fatalf("the refused subscribe said %v", err)
}
case <-deadline:
t.Fatal("Subscribe did not return in 2 s: the channel is stuck on a write to a pipe nobody reads")
}
// Let the waiting answer go, so nothing is left blocked.
go func() { _, _ = io.Copy(io.Discard, outR) }()
}
+1 -1
View File
@@ -20,7 +20,7 @@ node node_modules/typescript/bin/tsc --noEmit
npm test
if command -v go >/dev/null 2>&1; then
(cd go && test -z "$(gofmt -l .)" && go vet ./... && go test -count=1 ./...)
sh go/merge-check.sh
else
echo "NOT TESTED HERE: go/ (the Go binding) — the Go toolchain runs it from go/merge-check.sh"
fi
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@novox/mesh-sdk",
"version": "0.1.13",
"version": "0.1.14",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@novox/mesh-sdk",
"version": "0.1.13",
"version": "0.1.14",
"devDependencies": {
"@types/node": "^22.0.0",
"typescript": "^5.6.0"
+5 -3
View File
@@ -1,18 +1,20 @@
{
"name": "@novox/mesh-sdk",
"version": "0.1.13",
"version": "0.1.14",
"description": "The stable spine a Novox Mesh module's own code builds against.",
"type": "module",
"exports": {
".": "./dist/index.js",
"./contracts": "./dist/contracts/index.js",
"./provisioner": "./dist/provisioner/index.js",
"./provider-loop": "./dist/provider-loop/index.js",
"./provisioner": "./dist/provider-loop/index.js",
"./tools": "./dist/tools/index.js",
"./messaging": "./dist/messaging/index.js",
"./events": "./dist/events/index.js",
"./primitives": "./dist/primitives/index.js",
"./stdio": "./dist/stdio/index.js",
"./state": "./dist/state/index.js"
"./state": "./dist/state/index.js",
"./seal": "./dist/seal/index.js"
},
"files": [
"dist"
+5 -2
View File
@@ -1,13 +1,16 @@
// @novox/mesh-sdk — the stable spine a module's own code builds against (novox/hq ADR 0039).
//
// Import the area you need directly (`@novox/mesh-sdk/tools`, `/provisioner`, …) or the whole
// Import the area you need directly (`@novox/mesh-sdk/tools`, `/provider-loop`, …) or the whole
// surface from here. Everything specific to one module — its API client, its tool implementations,
// its create-a-resource adapter — lives in the module, never here.
export * from "./contracts/index.js";
export * as tools from "./tools/index.js";
export * as provisioner from "./provisioner/index.js";
export * as providerLoop from "./provider-loop/index.js";
/** @deprecated The provider loop's name before design 57 (novox/hq issue 678): `providerLoop`. */
export * as provisioner from "./provider-loop/index.js";
export * as messaging from "./messaging/index.js";
export * as events from "./events/index.js";
export * as primitives from "./primitives/index.js";
export * as state from "./state/index.js";
export * as seal from "./seal/index.js";
@@ -1,3 +1,13 @@
// The provider loop: the reconcile harness every provider on the controller's grant files shares.
// Called `provisioner` until design 57 (novox/hq issue 678): that word is mesh-provisioner's, which
// places every value a module needs (ADR 0349), so the loop is named for what it is. The old path
// `@novox/mesh-sdk/provisioner` and the names `runProvisioner`, `provisionerTools`, `Provisioner` and
// `ProvisionerOptions` stay as aliases of these until the providers that import them move to the Go
// helper that answers consumers over the bus (mesh-sdk go/provider, design 57 B2 and B3) and this loop
// retires with the grant files. The events (`provisioner.failing`, `.recovered`, `.retirement`) and the
// tool names (`provisioner_retirement`, …) are what the controller and the operator read on the bus,
// and keep their names.
//
// The reconcile harness every provider shares. The loop below is identical for postgres, redis,
// minio and umami: read the contributions the mesh delivered, bring each consumer's resource into
// being through the provider's adapter under the login and password the mesh minted, and retire
@@ -10,10 +20,20 @@
// does not hand one back — the consumer already receives its copy through the mesh's own channel.
// This is why the old $MESH_SEAL_KEY, the symmetric seal, and the *.grant.json / *.credential files
// are gone: they described a second, contradictory credential path the mesh does not have.
//
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
// whose create, check or secret has failed without one success in between for `failingAfterMs` is
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
// again every `sayAgainEveryMs` while it lasts; the first success after that is
// `provisioner.recovered`, as is a consumer the mesh stopped asking for, and the first success for
// each consumer since this process started. The controller keeps the newest per provider and consumer
// and `status` names it. Until novox/hq issue 272 only the Go copy of this loop (postgres, keycloak)
// said it; every provider on this one failed its consumers in its journal alone.
import { readFile } from "node:fs/promises";
import type { ToolDefinition } from "../contracts/index.js";
import { emit } from "../events/index.js";
import { registerModuleTools } from "../tools/index.js";
/** One consumer's resource to bring into being — everything the mesh derived and delivered. */
@@ -86,6 +106,9 @@ export interface Adapter {
/** Optional: every consumer the backend holds ACTIVE that the mesh made, read at start, so one the
* mesh stopped asking for while this process was down is still retired (by the same rules). */
held?(): Promise<string[]>;
/** Optional: the class of one of the adapter's own errors (novox/hq ADR 0224), when its code knows
* better than the error's words; "" or undefined falls back to `classOf` of the text. */
classify?(err: unknown): string | undefined;
}
/** One retired consumer, as `provisioner_retirement` answers it. */
@@ -127,9 +150,9 @@ export interface Act {
readonly via?: string;
}
/** The running provisioner: a stop function, as before, that also answers and takes the person's
/** The running provider loop: a stop function, as before, that also answers and takes the person's
* acts on retirement (novox/hq ADR 0230). */
export type Provisioner = (() => void) & {
export type ProviderLoop = (() => void) & {
retirement(): Promise<Retirement>;
approve(consumers: readonly string[], act: Act): Promise<{ retired: string[] }>;
reject(consumers: readonly string[], act: Act): Promise<{ kept: string[] }>;
@@ -139,7 +162,22 @@ export type Provisioner = (() => void) & {
/** The one event a provider says about retirement (novox/hq ADR 0230). */
export const RETIREMENT_EVENT = "provisioner.retirement";
export interface ProvisionerOptions {
/** The events a provider's standing toward one consumer is announced as (novox/hq ADR 0224). The
* controller derives the permission to emit them for every module that receives contributions. */
export const FAILING_EVENT = "provisioner.failing";
export const RECOVERED_EVENT = "provisioner.recovered";
/** The classes a failing consumer is announced with: what a person reading `status` needs to know
* before reading the journal (novox/hq ADR 0224). An adapter may say better (`classify`). */
export const CLASS_CREDENTIALS = "credentials-rejected";
export const CLASS_UNREACHABLE = "unreachable";
export const CLASS_SECRET = "secret-unreadable";
export const CLASS_REFUSED = "refused";
/** How long a consumer fails without one success before it is announced failing (ADR 0224). */
export const FAILING_AFTER_MS = 5 * 60_000;
export interface ProviderLoopOptions {
/** The contributions file the mesh writes for this resource (the module's `receives` path).
* Defaults to $MESH_RECEIVES. */
receives?: string;
@@ -165,10 +203,16 @@ export interface ProvisionerOptions {
/** Nor a set larger than this share of the consumers held, where more than one is held. Defaults
* to 0.5. */
retireFraction?: number;
/** How often a waiting retirement is said again, in ms. Defaults to 15 minutes. */
/** How often a waiting retirement, and a consumer still failing, is said again, in ms. Defaults to
* 15 minutes. */
sayAgainEveryMs?: number;
/** Publishes one of the provider's events (`provisioner.retirement`); usually the SDK's `emit`.
* Without it the provider's retirement is said in its journal only. */
/** How long, in ms, a consumer fails without one success before it is announced
* `provisioner.failing` (novox/hq ADR 0224). Defaults to FAILING_AFTER_MS, five minutes. */
failingAfterMs?: number;
/** Publishes one of the provider's events (`provisioner.failing`, `provisioner.recovered`,
* `provisioner.retirement`). Defaults to the SDK's `emit`, so every provider says them on the bus
* without asking (novox/hq issue 272); a failed publish is said in the journal and never fails
* the act it reports. */
announce?: (event: string, body: Record<string, unknown>) => Promise<void> | void;
/** The machine this provider runs on, said in each event. Defaults to $MESH_NODE. */
node?: string;
@@ -216,7 +260,7 @@ interface Contribution {
* `retireFraction` of those held where more than one is held, is not retired without a person: the
* provider waits for approve or reject. RETIRED → DELETED only through `provisioner_delete`.
*/
export function runProvisioner(resource: string, adapter: Adapter, opts: ProvisionerOptions = {}): Provisioner {
export function runProviderLoop(resource: string, adapter: Adapter, opts: ProviderLoopOptions = {}): ProviderLoop {
const receives = opts.receives ?? envOrThrow("MESH_RECEIVES");
const everyMs = opts.everyMs ?? 5000;
const verifyEveryMs = opts.verifyEveryMs ?? 60_000;
@@ -227,6 +271,8 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
const retireAtOnce = opts.retireAtOnce ?? 3;
const retireFraction = opts.retireFraction ?? 0.5;
const sayAgainEveryMs = opts.sayAgainEveryMs ?? 15 * 60_000;
const failingAfterMs = opts.failingAfterMs ?? FAILING_AFTER_MS;
const announce = opts.announce ?? ((event: string, body: Record<string, unknown>) => emit(event, body));
const node = opts.node ?? process.env.MESH_NODE ?? "";
const say = (line: string): void => console.error(`[provisioner:${resource}] ${line}`);
// Retirement (novox/hq ADR 0230). The unasked set last seen and in how many consecutive readable
@@ -245,6 +291,13 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
// itself. Each repeat doubles the wait before asking again, so such a bug costs one re-apply an
// hour rather than one a minute, and it is said loudly rather than quietly repeated.
const lost = new Map<string, { times: number; nextAt: number }>();
// Each consumer's unbroken run of failures — since when, how often, the last error, and when it was
// last announced (0: not yet) — and every consumer that has succeeded since this process started
// (novox/hq ADR 0224).
const trouble = new Map<string, {
node?: string; since: number; attempts: number; cls: string; text: string; saidAt: number;
}>();
const cleared = new Set<string>();
// login (`as`) -> what was last applied: the hash that says whether it changed, and the
// derivation to hand back on withdrawal, which by then is no longer in the file.
@@ -311,6 +364,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
`file and who this process runs as (novox/hq issue 225)`,
);
}
await failed(g.as, g.node, CLASS_SECRET, `secret not readable (${g.secret}): ${err}`);
continue;
}
waiting.delete(g.as);
@@ -328,11 +382,15 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
try {
if (await withTimeout(adapter.holds!(p), holdsTimeoutMs)) {
lost.delete(g.as);
await succeeded(g.as);
continue;
}
} catch (err) {
// Unable to ask is not evidence of loss. Kept as applied, asked again next time.
console.error(`[provisioner:${resource}] ${g.as}: could not check the backend, will ask again: ${scrub(err, password)}`);
// Unable to ask is not evidence of loss. Kept as applied, asked again next time — but a
// check that keeps failing is a consumer failed all the same (ADR 0224, issue 179).
const text = scrub(err, password);
console.error(`[provisioner:${resource}] ${g.as}: could not check the backend, will ask again: ${text}`);
await failed(g.as, g.node, classed(err, text), text);
// A backend that did not answer in time will not answer for the next consumer either, and
// each would cost a timeout, so the rest of this pass is not asked. Any other failure may
// be this consumer's alone, and the others are still asked.
@@ -356,6 +414,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
await adapter.reenable({ as: g.as, derived: was.derived ?? p.derived });
}
await adapter.create(p);
await succeeded(g.as);
applied.set(g.as, { hash: h, derived: p.derived, node: g.node });
if (retired.has(g.as)) {
// Asked for again: the ordinary create re-enabled it and cleared its mark (ADR 0230).
@@ -376,7 +435,9 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
}
}
} catch (err) {
console.error(`[provisioner:${resource}] ${g.as}: create failed, will retry: ${scrub(err, password)}`);
const text = scrub(err, password);
console.error(`[provisioner:${resource}] ${g.as}: create failed, will retry: ${text}`);
await failed(g.as, g.node, classed(err, text), text);
// A failed create is not a disagreement: asked again at the next check, the count unchanged.
if (reapplying !== undefined) lost.set(g.as, { times: reapplying - 1, nextAt: 0 });
}
@@ -386,6 +447,12 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
// `stablePasses` passes, and within the bound, or with a person's word (novox/hq ADR 0230).
// Only consumers this harness applied, or the backend says the mesh made, are touched.
const unasked = [...applied.keys()].filter((as) => !wantByAs.has(as)).sort();
// A consumer the mesh stopped asking for that this provider holds nothing for is no longer failed
// by anyone: said, so the standing the controller keeps for it is cleared rather than left naming a
// consumer that is gone. One still held is said recovered when it is retired (ADR 0224, 0230).
for (const as of [...trouble.keys()]) {
if (!wantByAs.has(as) && !applied.has(as)) await recovered(as, "no longer asked for");
}
const key = unasked.join(",");
if (pending && pending.consumers.join(",") !== key) {
say(`the retirement waiting for a person (${pending.consumers.join(", ")}) is no longer what the mesh says; settled, nothing retired`);
@@ -464,6 +531,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
}
applied.delete(as);
lost.delete(as);
await recovered(as, "retired");
retired.set(as, { node: was.node, at: at.getTime(), why, derived: was.derived, access: adapter.retire ? "disabled" : "kept" });
done.push(as);
} catch (err) {
@@ -485,16 +553,86 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
// Said on the bus, never failing the act it reports (the journal still has it).
async function tell(change: string, body: Record<string, unknown>): Promise<void> {
if (!opts.announce) return;
await said(RETIREMENT_EVENT, {
provider: resource, "provider-node": node, change, bound: bound(applied.size), at: iso(Date.now()), ...body,
}, change);
}
async function said(event: string, body: Record<string, unknown>, what = ""): Promise<void> {
try {
await opts.announce(RETIREMENT_EVENT, {
provider: resource, "provider-node": node, change, bound: bound(applied.size), at: iso(Date.now()), ...body,
});
await announce(event, body);
} catch (err) {
say(`could not announce ${RETIREMENT_EVENT} ${change}: ${err}`);
say(`could not announce ${event}${what ? ` ${what}` : ""}: ${err}`);
}
}
// An error's class: the adapter's word when it has one, else read from the text (ADR 0224).
function classed(err: unknown, text: string): string {
try {
const own = adapter.classify?.(err);
if (own) return own;
} catch {
// An adapter that cannot class its own error is read by the words.
}
// A check that did not answer in time is a backend out of reach, whatever its words.
if (err instanceof HoldsTimeout) return CLASS_UNREACHABLE;
return classOf(text);
}
// One more failure in a consumer's unbroken run, announced once the run has lasted failingAfterMs,
// then again every sayAgainEveryMs while it lasts (novox/hq ADR 0224).
async function failed(as: string, consumerNode: string | undefined, cls: string, text: string): Promise<void> {
const now = Date.now();
let s = trouble.get(as);
if (!s) {
s = { since: now, attempts: 0, cls, text, saidAt: 0 };
trouble.set(as, s);
}
s.node = consumerNode;
s.cls = cls;
s.text = text;
s.attempts++;
if (now - s.since < failingAfterMs) return;
if (s.saidAt !== 0 && now - s.saidAt < sayAgainEveryMs) return;
if (s.saidAt === 0) {
say(
`${as}: FAILING for ${Math.round((now - s.since) / 1000)}s (${s.attempts} attempts, ${cls}): ${text}. ` +
`Announced as ${FAILING_EVENT}; \`status\` names it until it succeeds (novox/hq ADR 0224)`,
);
}
s.saidAt = now;
await said(FAILING_EVENT, {
provider: resource, "provider-node": node, consumer: as, node: consumerNode ?? "",
class: cls, error: clip(text), since: iso(s.since), attempts: s.attempts,
});
}
// A success ends a consumer's run of failures; one that was announced is announced recovered. And
// the first success for a consumer since this process started is announced too, failing or not: a
// provider that announced a failure and was restarted has forgotten it, and without this the
// controller would name the consumer failing for ever after it recovered unheard.
async function succeeded(as: string): Promise<void> {
if (!trouble.has(as) && !cleared.has(as)) {
cleared.add(as);
await said(RECOVERED_EVENT, { provider: resource, "provider-node": node, consumer: as, why: "first-success" });
return;
}
cleared.add(as);
await recovered(as, "");
}
async function recovered(as: string, why: string): Promise<void> {
const s = trouble.get(as);
if (!s) return;
trouble.delete(as);
if (s.saidAt === 0) return; // never announced, so there is nothing to take back
if (!why) say(`${as}: recovered after ${Math.round((Date.now() - s.since) / 1000)}s and ${s.attempts} failed attempt(s)`);
await said(RECOVERED_EVENT, {
provider: resource, "provider-node": node, consumer: as, node: s.node ?? "",
since: iso(s.since), attempts: s.attempts, ...(why ? { why } : {}),
});
}
// What the backend holds that the mesh made: active ones become held, so one the mesh stopped asking
// for while this process was down is still retired, by the same rules; retired ones are known.
async function seed(): Promise<void> {
@@ -629,8 +767,8 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
const stop = () => {
stopped = true;
};
const handle: Provisioner = Object.assign(stop, { retirement, approve, reject, deleteRetired });
if (opts.module) registerModuleTools(opts.module, () => provisionerTools(handle));
const handle: ProviderLoop = Object.assign(stop, { retirement, approve, reject, deleteRetired });
if (opts.module) registerModuleTools(opts.module, () => providerLoopTools(handle));
return handle;
}
@@ -639,7 +777,7 @@ const text = (v: unknown): string => (typeof v === "string" ? v : "");
/** The four tools every provider serves for retirement (novox/hq ADR 0230), the controller's `retire`
* and `cleanup` verbs asking them per machine. */
export function provisionerTools(p: Provisioner): ToolDefinition[] {
export function providerLoopTools(p: ProviderLoop): ToolDefinition[] {
const act = { type: "string" };
return [
{
@@ -713,6 +851,27 @@ async function readContributions(
return (doc.given as Contribution[]).filter((g) => g.as && g.secret);
}
/** An error's class read from its text — the words the backends the mesh runs use (novox/hq ADR
* 0224): the same words, in the same order, as the Go copy of this loop reads. */
export function classOf(text: string): string {
const t = text.toLowerCase();
const credentials = ["invalid_grant", "invalid user credentials", "password authentication failed",
"authentication failed", "unauthorized", " 401"];
if (credentials.some((w) => t.includes(w))) return CLASS_CREDENTIALS;
const unreachable = ["connection refused", "no such host", "i/o timeout", "deadline exceeded",
"connection reset", "network is unreachable", "no route to host", "eof",
// Node's own words for the same faults.
"econnrefused", "enotfound", "etimedout", "econnreset", "ehostunreach", "enetunreach", "eai_again"];
if (unreachable.some((w) => t.includes(w))) return CLASS_UNREACHABLE;
return CLASS_REFUSED;
}
/** An announced error kept to what belongs in a status line. */
function clip(text: string): string {
const most = 300;
return text.length <= most ? text : `${text.slice(0, most)}…`;
}
/** A check that did not answer in time: the backend, not the consumer, is the likely cause. */
class HoldsTimeout extends Error {}
@@ -751,3 +910,12 @@ function envOrThrow(name: string): string {
if (!v) throw new Error(`${name} is not set — the provisioner cannot run without it`);
return v;
}
/** @deprecated The provider loop's name before design 57: {@link runProviderLoop}. */
export const runProvisioner = runProviderLoop;
/** @deprecated {@link providerLoopTools}. */
export const provisionerTools = providerLoopTools;
/** @deprecated {@link ProviderLoop}. */
export type Provisioner = ProviderLoop;
/** @deprecated {@link ProviderLoopOptions}. */
export type ProvisionerOptions = ProviderLoopOptions;
+90
View File
@@ -0,0 +1,90 @@
// The sealed box itself (novox/hq ADR 0183, ADR 0206), with its two sources of randomness as arguments so
// the tests can prove it byte for byte against the Go package's vectors. Modules import `./index.js`.
import {
createCipheriv,
createDecipheriv,
createPrivateKey,
createPublicKey,
diffieHellman,
hkdfSync,
type KeyObject,
} from "node:crypto";
/** A value sealed to one recipient, as it travels in JSON; every field but `v` is standard base64. */
export interface SealedBox {
readonly v: number;
/** The one-time public key, SPKI DER. */
readonly eph: string;
readonly iv: string;
readonly tag: string;
readonly ct: string;
}
/** The only box version `open` opens. */
export const VERSION = 1;
const INFO = "novox-mesh sealed box v1";
const IV_BYTES = 12;
const TAG_BYTES = 16;
function x25519(key: KeyObject, what: string): KeyObject {
if (key.asymmetricKeyType !== "x25519") throw new Error(`not an X25519 ${what} key`);
return key;
}
export function publicFromPem(pem: string): KeyObject {
return x25519(createPublicKey({ key: pem, format: "pem" }), "public");
}
export function privateFromPem(pem: string): KeyObject {
return x25519(createPrivateKey({ key: pem, format: "pem" }), "private");
}
/** The recipient's raw 32-byte public key: the last 32 bytes of its SPKI DER. */
function raw(pub: KeyObject): Buffer {
const der = pub.export({ format: "der", type: "spki" });
return der.subarray(der.length - 32);
}
function boxKey(secret: Buffer, ephDer: Buffer, recipientRaw: Buffer): Buffer {
return Buffer.from(hkdfSync("sha256", secret, Buffer.concat([ephDer, recipientRaw]), INFO, 32));
}
/** `seal` with the one-time private key and the iv given. */
export function sealWith(plaintext: string, recipientPem: string, eph: KeyObject, iv: Buffer): SealedBox {
const recipient = publicFromPem(recipientPem);
x25519(eph, "one-time");
if (iv.length !== IV_BYTES) throw new Error(`an iv of ${iv.length} bytes, not ${IV_BYTES}`);
const secret = diffieHellman({ privateKey: eph, publicKey: recipient });
const ephDer = createPublicKey(eph).export({ format: "der", type: "spki" });
const cipher = createCipheriv("aes-256-gcm", boxKey(secret, ephDer, raw(recipient)), iv);
const ct = Buffer.concat([cipher.update(plaintext, "utf8"), cipher.final()]);
return {
v: VERSION,
eph: ephDer.toString("base64"),
iv: iv.toString("base64"),
tag: cipher.getAuthTag().toString("base64"),
ct: ct.toString("base64"),
};
}
/** Open a box with the recipient's PEM private key; throws for a box to another key, changed, or of another version. */
export function open(box: SealedBox, privatePem: string): string {
if (box.v !== VERSION) throw new Error(`a sealed box of version ${box.v}, not ${VERSION}`);
const priv = privateFromPem(privatePem);
const ephDer = Buffer.from(box.eph, "base64");
const eph = x25519(createPublicKey({ key: ephDer, format: "der", type: "spki" }), "one-time");
const iv = Buffer.from(box.iv, "base64");
const tag = Buffer.from(box.tag, "base64");
// node accepts a shorter tag than 16 bytes; the box never has one, so a short one is refused here.
if (iv.length !== IV_BYTES || tag.length !== TAG_BYTES) throw new Error("the box does not open with this key");
const secret = diffieHellman({ privateKey: priv, publicKey: eph });
const decipher = createDecipheriv("aes-256-gcm", boxKey(secret, ephDer, raw(createPublicKey(priv))), iv);
decipher.setAuthTag(tag);
try {
return Buffer.concat([decipher.update(Buffer.from(box.ct, "base64")), decipher.final()]).toString("utf8");
} catch {
throw new Error("the box does not open with this key");
}
}
+29
View File
@@ -0,0 +1,29 @@
// Sealing to one recipient (novox/hq ADR 0183, ADR 0206): X25519 for the agreement, HKDF-SHA256 for the
// key, AES-256-GCM for the box. The TypeScript twin of the Go package `go/seal`, byte for byte: the same
// fixtures prove both. A box is `{ v: 1, eph, iv, tag, ct }`; keys travel as PEM (SPKI public, PKCS #8
// private).
import { generateKeyPairSync, randomBytes } from "node:crypto";
import { sealWith, open, VERSION, type SealedBox } from "./box.js";
export { open, VERSION, type SealedBox };
/** A recipient's keypair as the two PEM strings it is kept and sent as. */
export interface KeyPair {
readonly publicKey: string;
readonly privateKey: string;
}
/** A fresh X25519 keypair, PEM-encoded. */
export function generateKeyPair(): KeyPair {
const { publicKey, privateKey } = generateKeyPairSync("x25519");
return {
publicKey: publicKey.export({ format: "pem", type: "spki" }) as string,
privateKey: privateKey.export({ format: "pem", type: "pkcs8" }) as string,
};
}
/** Seal plaintext to the recipient's PEM public key, with a fresh one-time key and iv. */
export function seal(plaintext: string, recipientPem: string): SealedBox {
return sealWith(plaintext, recipientPem, generateKeyPairSync("x25519").privateKey, randomBytes(12));
}
+8 -7
View File
@@ -8,7 +8,7 @@ import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { runProvisioner, provisionerTools, RETIREMENT_EVENT, type Provisioner } from "../dist/provisioner/index.js";
import { runProviderLoop, providerLoopTools, RETIREMENT_EVENT, type ProviderLoop } from "../dist/provider-loop/index.js";
import { resetTools, collectTools } from "../dist/tools/index.js";
const EVERY = 10;
@@ -32,7 +32,7 @@ interface World {
reenabled: string[];
events: Record<string, unknown>[];
said: string[];
p: Provisioner;
p: ProviderLoop;
done(): void;
}
@@ -62,9 +62,10 @@ async function world(
...(opts.retire === false ? {} : { async retire(p: { as: string; why: string }) { w.retired.push({ as: p.as, why: p.why }); } }),
...(opts.held ? { async held() { return opts.held!; } } : {}),
};
w.p = runProvisioner("postgres-database", adapter, {
w.p = runProviderLoop("postgres-database", adapter, {
receives, everyMs: EVERY, node: "anchor", stableForMs: opts.stableForMs ?? 0,
announce: (event, body) => { assert.equal(event, RETIREMENT_EVENT); w.events.push(body); },
// Only retirement is these tests'; the standing (provisioner.failing, .recovered) is standing.test.ts's.
announce: (event, body) => { if (event === RETIREMENT_EVENT) w.events.push(body); },
});
// A pass is counted by the file being read: wait wall time for n passes and some slack.
w.done = () => { w.p(); console.error = was; };
@@ -167,7 +168,7 @@ test("five passes are not enough: the same set must also hold for stableForMs",
});
test("the default hold is ten minutes", async () => {
const { STABLE_FOR_MS } = await import("../dist/provisioner/index.js");
const { STABLE_FOR_MS } = await import("../dist/provider-loop/index.js");
assert.equal(STABLE_FOR_MS, 10 * 60_000);
});
@@ -326,7 +327,7 @@ test("the four tools are registered for the module and answer", async () => {
const dir = await mkdtemp(join(tmpdir(), "retire-tools-"));
const receives = join(dir, "r.json");
await writeFile(receives, JSON.stringify({ requirement: "r", given: [] }));
const p = runProvisioner("r", { async create() {}, async remove() {} }, { receives, everyMs: EVERY, module: "postgres" });
const p = runProviderLoop("r", { async create() {}, async remove() {} }, { receives, everyMs: EVERY, module: "postgres" });
try {
const mine = collectTools({}).find((c) => c.module === "postgres");
assert.deepEqual(mine?.tools.map((t) => t.name), [
@@ -335,7 +336,7 @@ test("the four tools are registered for the module and answer", async () => {
const answer = (await mine!.tools[0].run({})) as { resource: string; waiting: unknown };
assert.equal(answer.resource, "r");
assert.equal(answer.waiting, null);
assert.equal(provisionerTools(p).length, 4);
assert.equal(providerLoopTools(p).length, 4);
} finally {
p();
resetTools();
+26 -13
View File
@@ -6,7 +6,7 @@ import { join } from "node:path";
import { compareVersions } from "../dist/primitives/index.js";
import { registerModuleTools, collectTools, collectToolsEach, resetTools, serveTools, listTools, invokeTool } from "../dist/tools/index.js";
import { runProvisioner } from "../dist/provisioner/index.js";
import { runProviderLoop } from "../dist/provider-loop/index.js";
import { emit, on, type Event } from "../dist/events/index.js";
import { useBroker } from "../dist/messaging/index.js";
@@ -47,7 +47,7 @@ test("provisioner creates each consumer with the mesh's login and password; remo
doc([{ from: "webapp", node: "anchor", as: "webapp-anchor", secret: join(dir, "webapp.secret"), values: { name: "webapp" } }]),
);
const stop = runProvisioner(
const stop = runProviderLoop(
"analytics",
{
async create(p) {
@@ -101,7 +101,7 @@ test("provisioner hands the adapter what the mesh derived, and still has it at w
});
await writeFile(receives, doc([entry("mesh-anchor-ncloud")]));
const stop = runProvisioner(
const stop = runProviderLoop(
"s3-bucket",
{
async create(p) {
@@ -153,7 +153,7 @@ test("provisioner stops calling an unreadable secret a race, and says what to lo
console.error = (line: unknown) => { said.push(String(line)); };
let stop: (() => void) | undefined;
try {
stop = runProvisioner("db", { async create() {}, async remove() {} }, { receives, everyMs: 5 });
stop = runProviderLoop("db", { async create() {}, async remove() {} }, { receives, everyMs: 5 });
await waitFor(() => said.some((l) => l.includes("CANNOT READ")), 3000);
} finally {
stop?.();
@@ -185,7 +185,7 @@ test("provisioner removes nothing while its contributions file cannot be used (n
]);
await writeFile(receives, both);
const stop = runProvisioner(
const stop = runProviderLoop(
"postgres-database",
{
async create(p) {
@@ -239,7 +239,7 @@ test("provisioner applies again what the backend no longer holds, and trusts mem
const backend = new Set<string>();
let creates = 0;
let asked = 0;
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create(p) {
@@ -271,7 +271,7 @@ test("provisioner applies again what the backend no longer holds, and trusts mem
// An adapter without holds is trusted from memory, as before: a forgotten backend stays forgotten.
const forgetful = new Set<string>();
let plainCreates = 0;
const stopPlain = runProvisioner(
const stopPlain = runProviderLoop(
"cache",
{
async create(p) {
@@ -299,7 +299,7 @@ test("provisioner keeps a consumer applied when the backend cannot be asked", as
);
let creates = 0;
let asked = 0;
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create() {
@@ -330,7 +330,7 @@ test("provisioner backs off when create does not satisfy holds, and says so", as
const original = console.error;
console.error = (...a: unknown[]) => logged.push(a.join(" "));
try {
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create() {
@@ -371,7 +371,7 @@ test("provisioner treats a failed or hung holds as could-not-ask, and never logs
const original = console.error;
console.error = (...a: unknown[]) => logged.push(a.join(" "));
try {
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create() {
@@ -404,7 +404,7 @@ test("provisioner treats a failed or hung holds as could-not-ask, and never logs
const hungLogged: string[] = [];
console.error = (...a: unknown[]) => hungLogged.push(a.join(" "));
try {
const stopHung = runProvisioner(
const stopHung = runProviderLoop(
"cache",
{
async create() {
@@ -438,7 +438,7 @@ test("provisioner stops a verify pass at a timeout, and retries a failed re-appl
try {
// A timeout on "a" means "b" is not asked in that pass.
const askedFor: string[] = [];
const stopHung = runProvisioner(
const stopHung = runProviderLoop(
"cache",
{
async create() {},
@@ -460,7 +460,7 @@ test("provisioner stops a verify pass at a timeout, and retries a failed re-appl
let createCalls = 0;
const logged: string[] = [];
console.error = (...a: unknown[]) => logged.push(a.join(" "));
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create(p) {
@@ -649,3 +649,16 @@ test("each registration is given the environment asked for it, and sees no other
assert.deepEqual(seen.beta, { BETA_URL: "http://127.0.0.1:3000" });
resetTools();
});
test("the provider loop is named for what it is, and its old name stays an alias until its providers move (issue 678)", async () => {
const loop = await import("../dist/provider-loop/index.js");
const old = await import("../dist/index.js");
assert.equal(old.provisioner.runProvisioner, loop.runProviderLoop);
assert.equal(old.provisioner.provisionerTools, loop.providerLoopTools);
assert.equal(old.providerLoop.runProviderLoop, loop.runProviderLoop);
const pkg = JSON.parse(await (await import("node:fs/promises")).readFile(new URL("../package.json", import.meta.url), "utf8"));
assert.equal(pkg.exports["./provider-loop"], "./dist/provider-loop/index.js");
assert.equal(pkg.exports["./provisioner"], pkg.exports["./provider-loop"]);
// The events and tools are what the controller and the operator read on the bus: their names stay.
assert.equal(loop.FAILING_EVENT, "provisioner.failing");
});
+71
View File
@@ -0,0 +1,71 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { createPrivateKey } from "node:crypto";
import { seal, open, generateKeyPair, type SealedBox } from "../dist/seal/index.js";
import { sealWith } from "../dist/seal/box.js";
// The fixtures the Go package proves itself against: boxes the licence manager's own seal.go made, and
// the one the agent module's TypeScript made. One format, three implementations.
const fixtures = new URL("../go/seal/testdata/", import.meta.url);
const lm = JSON.parse(readFileSync(new URL("sealed-by-licence-manager.json", fixtures), "utf8")) as {
recipient: { publicKey: string; privateKey: string };
sealed: { plaintext: string; box: SealedBox }[];
vectors: { plaintext: string; ephPrivate: string; iv: string; box: SealedBox }[];
};
// PKCS #8 for an X25519 private key is this prefix and the 32 raw bytes.
const pkcs8 = (raw: Buffer) =>
createPrivateKey({ key: Buffer.concat([Buffer.from("302e020100300506032b656e04220420", "hex"), raw]), format: "der", type: "pkcs8" });
test("seal: every box the licence manager sealed opens to its plaintext", () => {
assert.ok(lm.sealed.length > 0);
for (const s of lm.sealed) assert.equal(open(s.box, lm.recipient.privateKey), s.plaintext);
});
test("seal: given the same one-time key and iv, the box is byte for byte the licence manager's", () => {
assert.ok(lm.vectors.length > 0);
for (const v of lm.vectors) {
const box = sealWith(v.plaintext, lm.recipient.publicKey, pkcs8(Buffer.from(v.ephPrivate, "base64")), Buffer.from(v.iv, "base64"));
assert.deepEqual(box, v.box);
assert.equal(open(box, lm.recipient.privateKey), v.plaintext);
}
});
test("seal: the agent module's TypeScript box opens", () => {
const f = JSON.parse(readFileSync(new URL("sealed-by-typescript.json", fixtures), "utf8"));
assert.equal(open(f.box, f.privateKey), f.plaintext);
});
test("seal: a box opens for its recipient only, and two boxes share no one-time key or iv", () => {
const a = generateKeyPair();
const b = generateKeyPair();
const box = seal("a token", a.publicKey);
assert.equal(open(box, a.privateKey), "a token");
assert.throws(() => open(box, b.privateKey));
const again = seal("a token", a.publicKey);
assert.notEqual(again.eph, box.eph);
assert.notEqual(again.iv, box.iv);
});
test("seal: a box changed in any field, or of another version, does not open", () => {
const good = lm.sealed[1].box;
const flip = (s: string) => {
const b = Buffer.from(s, "base64");
b[b.length - 1] ^= 1;
return b.toString("base64");
};
for (const box of [
{ ...good, ct: flip(good.ct) },
{ ...good, tag: flip(good.tag) },
{ ...good, iv: flip(good.iv) },
{ ...good, eph: flip(good.eph) },
{ ...good, v: 2 },
]) assert.throws(() => open(box as SealedBox, lm.recipient.privateKey));
});
test("seal: a key that is not X25519 PEM is refused", () => {
assert.throws(() => seal("x", "not a key"));
assert.throws(() => open(lm.sealed[0].box, lm.recipient.publicKey));
});
+272
View File
@@ -0,0 +1,272 @@
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224, issue 272): not on
// the first failure, which may be a restart; after failingAfterMs of failures with no success between;
// again every sayAgainEveryMs while it lasts; and recovered on the first success, or when the consumer
// goes. The same behaviour as the Go copy of the loop's standing_test.go, with the clock shortened.
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
runProviderLoop, classOf, FAILING_EVENT, RECOVERED_EVENT, RETIREMENT_EVENT,
CLASS_CREDENTIALS, CLASS_UNREACHABLE, CLASS_SECRET, CLASS_REFUSED, type Adapter,
} from "../dist/provider-loop/index.js";
import { useBroker } from "../dist/messaging/index.js";
const EVERY = 10;
const FAILING_AFTER = 150;
const SAY_AGAIN = 400;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
async function until(cond: () => boolean, ms = 4000): Promise<void> {
const start = Date.now();
while (!cond()) {
if (Date.now() - start > ms) throw new Error("timed out waiting");
await sleep(5);
}
}
interface Said { event: string; body: Record<string, unknown> }
interface World {
dir: string;
ask(list: { as: string; node?: string; values?: Record<string, unknown> }[]): Promise<void>;
failing: Error | null;
holdsErr: Error | null;
created: string[];
/** Every event but the first success since start, which has its own test. */
said: Said[];
all: Said[];
stop(): void;
}
async function world(opts: { classify?: boolean; announce?: false; stableForMs?: number } = {}): Promise<World> {
const dir = await mkdtemp(join(tmpdir(), "standing-"));
const receives = join(dir, "postgres-database.json");
const w = { dir, failing: null, holdsErr: null, created: [], said: [], all: [] } as unknown as World;
w.ask = async (list) => {
const given = [];
for (const g of list) {
await writeFile(join(dir, `${g.as}.secret`), `pw-${g.as}\n`);
given.push({ from: g.as, node: g.node ?? "anchor", as: g.as, secret: join(dir, `${g.as}.secret`), values: g.values });
}
await writeFile(receives, JSON.stringify({ contributions: 1, requirement: "postgres-database", given }));
};
const adapter: Adapter = {
async create(p) {
if (w.failing) throw w.failing;
w.created.push(p.as);
},
async remove() {},
async retire() {},
async holds() {
if (w.holdsErr) throw w.holdsErr;
return true;
},
...(opts.classify ? { classify: () => "its-own" } : {}),
};
const was = console.error;
console.error = () => {};
const stop = runProviderLoop("postgres-database", adapter, {
receives, everyMs: EVERY, verifyEveryMs: 0, node: "home-server",
failingAfterMs: FAILING_AFTER, sayAgainEveryMs: SAY_AGAIN, stableForMs: opts.stableForMs ?? 0,
...(opts.announce === false ? {} : {
announce: (event: string, body: Record<string, unknown>) => {
w.all.push({ event, body });
if (body.why !== "first-success") w.said.push({ event, body });
},
}),
});
w.stop = () => {
stop();
console.error = was;
void rm(dir, { recursive: true, force: true });
};
return w;
}
const events = (said: Said[]) => said.filter((s) => s.event !== RETIREMENT_EVENT).map((s) => s.event).join(",");
test("a consumer failed for minutes is announced, naming it, its machine and the class, again while it lasts, and recovered", async () => {
const w = await world();
try {
w.failing = new Error(`token request failed: 401 {"error":"invalid_grant"} for pw-a`);
await w.ask([{ as: "a", node: "anchor" }]);
await sleep(FAILING_AFTER / 2);
assert.equal(events(w.said), "", "announced before failingAfterMs");
await until(() => w.said.length === 1);
const b = w.said[0].body;
assert.equal(w.said[0].event, FAILING_EVENT);
assert.equal(b.consumer, "a");
assert.equal(b.node, "anchor");
assert.equal(b.class, CLASS_CREDENTIALS);
assert.equal(b.provider, "postgres-database");
assert.equal(b["provider-node"], "home-server");
assert.ok((b.attempts as number) > 3, `attempts ${b.attempts}`);
assert.match(String(b.since), /^\d{4}-\d{2}-\d{2}T/);
assert.doesNotMatch(String(b.error), /pw-a/, "the consumer's password travelled");
// Said again while it lasts, not every pass.
await sleep(SAY_AGAIN / 2);
assert.equal(events(w.said), FAILING_EVENT, "repeated too soon");
await until(() => w.said.length === 2);
assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`);
// The first success takes it back.
w.failing = null;
await until(() => w.said.length === 3);
assert.equal(w.said[2].event, RECOVERED_EVENT);
assert.equal(w.said[2].body.consumer, "a");
assert.equal(w.said[2].body.node, "anchor");
} finally {
w.stop();
}
});
test("one success between failures starts the run again", async () => {
const w = await world();
try {
w.failing = new Error("connection refused");
await w.ask([{ as: "a" }]);
await sleep(FAILING_AFTER * 0.6);
w.failing = null;
await until(() => w.created.length === 1);
w.failing = new Error("connection refused");
await w.ask([{ as: "a", values: { name: "changed" } }]);
await sleep(FAILING_AFTER * 0.6);
assert.equal(events(w.said), "", "two short runs are not one long one");
} finally {
w.stop();
}
});
// The check that failed for a day on 2026-10-05 (issue 179): consumers already made, every check
// refused. A check that cannot be asked is a failure too.
test("a check that keeps failing is a failure too", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }]);
await until(() => w.created.length === 1);
w.holdsErr = new Error("401 invalid_grant");
await until(() => w.said.length === 1);
assert.equal(w.said[0].event, FAILING_EVENT);
assert.equal(w.said[0].body.class, CLASS_CREDENTIALS);
w.holdsErr = null;
await until(() => w.said.length === 2);
assert.equal(events(w.said), `${FAILING_EVENT},${RECOVERED_EVENT}`);
} finally {
w.stop();
}
});
test("an unreadable secret is announced as such", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }]);
await rm(join(w.dir, "a.secret"));
await until(() => w.said.length === 1);
assert.equal(w.said[0].event, FAILING_EVENT);
assert.equal(w.said[0].body.class, CLASS_SECRET);
} finally {
w.stop();
}
});
test("a consumer no longer asked for is no longer failing", async () => {
const w = await world();
try {
w.failing = new Error("boom");
await w.ask([{ as: "a" }, { as: "b" }]);
await until(() => w.said.length === 2);
assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`);
await w.ask([{ as: "a" }]);
await until(() => w.said.some((s) => s.event === RECOVERED_EVENT));
const last = w.said.find((s) => s.event === RECOVERED_EVENT)!;
assert.equal(last.body.consumer, "b");
assert.equal(last.body.why, "no longer asked for");
} finally {
w.stop();
}
});
test("a failing consumer the mesh stopped asking for is said recovered when it is retired", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }, { as: "b" }, { as: "c" }]);
await until(() => w.created.length === 3);
w.holdsErr = new Error("boom");
await until(() => w.said.filter((s) => s.event === FAILING_EVENT).length >= 3);
// c is no longer asked for while it is still failing; the others then recover.
await w.ask([{ as: "a" }, { as: "b" }]);
w.holdsErr = null;
await until(() => w.said.some((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c"));
const c = w.said.find((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c")!;
assert.equal(c.body.why, "retired");
} finally {
w.stop();
}
});
test("an error is classed by its words", () => {
const table: Record<string, string> = {
'Keycloak token request failed: 401 {"error":"invalid_grant"}': CLASS_CREDENTIALS,
'FATAL: password authentication failed for user "postgres"': CLASS_CREDENTIALS,
"dial tcp 127.0.0.1:5432: connect: connection refused": CLASS_UNREACHABLE,
"context deadline exceeded": CLASS_UNREACHABLE,
"Error: connect ECONNREFUSED 127.0.0.1:9000": CLASS_UNREACHABLE,
"getaddrinfo ENOTFOUND minio": CLASS_UNREACHABLE,
'extension "nope" is not available': CLASS_REFUSED,
};
for (const [text, want] of Object.entries(table)) assert.equal(classOf(text), want, text);
});
test("an adapter that classes its own errors is believed", async () => {
const w = await world({ classify: true });
try {
w.failing = new Error("anything");
await w.ask([{ as: "a" }]);
await until(() => w.said.length === 1);
assert.equal(w.said[0].body.class, "its-own");
} finally {
w.stop();
}
});
// A provider restarted after announcing a failure has forgotten it; its first success for each
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
test("the first success since start is announced once", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }]);
await until(() => w.all.length >= 1);
await sleep(EVERY * 10);
assert.equal(w.all.length, 1, JSON.stringify(w.all));
assert.equal(w.all[0].event, RECOVERED_EVENT);
assert.equal(w.all[0].body.why, "first-success");
assert.equal(w.all[0].body.consumer, "a");
} finally {
w.stop();
}
});
// No provider on main passed `announce`, so none said anything (issue 272): the loop says it through
// the SDK's own emit unless it is given another way.
test("without announce the standing is said through the SDK's emit", async () => {
const published: { key: string; body: Record<string, unknown> }[] = [];
useBroker(() => ({
async publish(env: { key: string; body: Record<string, unknown> }) { published.push({ key: env.key, body: env.body }); },
async subscribe() { return () => {}; },
}) as never);
const w = await world({ announce: false });
try {
w.failing = new Error("connection refused");
await w.ask([{ as: "a" }]);
await until(() => published.some((p) => p.key === FAILING_EVENT));
const p = published.find((x) => x.key === FAILING_EVENT)!;
assert.equal(p.body.consumer, "a");
assert.equal(p.body.class, CLASS_UNREACHABLE);
} finally {
w.stop();
}
});