Compare commits

...
Author SHA1 Message Date
jschoubben f185aedbf5 The provider helper reads an ended or pending assignment as a removed key (issue 1098)
mesh/delivery-group group feat/chain-on-the-bus stopped: a member was stopped
mesh/delivery stopped: the pull request closed unmerged
mesh/merge-gate head f185aedbf5d6 merged into trunk f70a99dbf485 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head f185aedbf5d6 merged into trunk f70a99dbf485 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
2026-10-18 02:38:58 +02:00
mesh-tools f70a99dbf4 Merge pull request 'Feature providers-after-switch-on, step 1: the provider helper's fixes and the moved-provider harness (issues 945, 825, 849)' (#31) from feat/providers-after-switch-on into main 2026-10-16 19:13:11 +00:00
fix-sdk-pas a4e7f480b3 Apply the plan's credential form only to an offer that declares a credential (issue 825)
mesh/merge-gate head a4e7f480b3b0 merged into trunk 219bedbdd2b3 beside mesh-catalog#337@d2992f81 +1 more — pass, with a note: builds mesh-sdk (+41 depende…
mesh/repo-check head a4e7f480b3b0 merged into trunk 219bedbdd2b3 beside mesh-catalog#337@d2992f81 +1 more — pass: its merge-check.sh passed, each part in i…
mesh/delivery delivered
mesh/delivery-group group feat/providers-after-switch-on merged: every member is merged
2026-10-16 19:44:26 +02:00
fix-sdk-pas d7cac79585 Carry the grant-file and keycloak-shape moved-path tests into harness (issue 849) 2026-10-16 19:37:02 +02:00
fix-sdk-pas 377fd6d67d Move the moved path's shared driving into harness.Moved, so postgres and keycloak keep only what differs (issue 849) 2026-10-16 19:34:02 +02:00
fix-sdk-pas bb7c82cac6 Move the moved-provider harness and its retirement into go/provider/harness (issue 849) 2026-10-16 19:29:27 +02:00
fix-sdk-pas 0db956dd83 Skip a bad assignment, honour the plan's credential form, delete only what was read, redact every credential form (issue 825) 2026-10-16 19:28:25 +02:00
fix-sdk-pas 40f3658557 Name the vault's credential under vault-name so the state guard accepts the bound entry (issue 945) 2026-10-16 19:26:58 +02:00
mesh-tools 219bedbdd2 Merge pull request 'Feature 57 (config and provisioning): release mesh-sdk' (#30) from release/57-config-and-provisioning into main 2026-10-16 09:32:31 +00:00
mesh-tools 1bc5719b8b Feature 57 (config and provisioning): release mesh-sdk, the reviewed release branch on top of main
mesh/merge-gate head 1bc5719b8b85 merged into trunk a2d2af00dc5d beside mesh-host#116@bff1ac14 — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ac…
mesh/repo-check head 1bc5719b8b85 merged into trunk a2d2af00dc5d beside mesh-host#116@bff1ac14 — pass: its merge-check.sh passed, each part in its toolchai…
mesh/delivery delivered
mesh/delivery-group group release/57-config-and-provisioning merged: every member is merged
2026-10-16 08:24:11 +00:00
mesh-admin a2d2af00dc Merge pull request 'go/asks: a message carries its note apart, and an explanation is held to a glance (issues 383 and 639)' (#25) from fix/639-an-ask-reads-at-a-glance into main 2026-10-13 19:02:17 +00:00
jochen 5933eb757d go/asks: whole details never digest as another ask's Whole and Details (the review of #25)
mesh/merge-gate head 5933eb757d44 merged into trunk a0fc82fdcd7c — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 5933eb757d44 merged into trunk a0fc82fdcd7c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
WholeDetails was written after a tag, so an ask with only whole details "+ x: 1" encoded as one whose Whole
was "whole-details" and Details "+ x: 1". Whole and Details are now written whenever whole details are
given, then the whole details, untagged: the number of trailing values tells the three shapes apart, and an
ask without whole details digests as before. The note's comment says what ADR 0347 §5 holds of cuts.
2026-10-13 20:29:24 +02:00
jochen f4745447ba go/asks: an ask may carry its Details whole, for a channel that proves who answers (hq issue 383)
mesh/merge-gate head f4745447ba6d merged into trunk a0fc82fdcd7c — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head f4745447ba6d merged into trunk a0fc82fdcd7c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery superseded: a newer head of the same pull request
The review of mesh-controller #304: a settings proposal with more than three keys showed only the names
of the rest, and its Details could carry no value under the content rule, so Approve bound values the
operator never saw. WholeDetails carries every value whole for the note on a proving channel, held to
the secret rule alone, as Whole is; it is digested only when given.
2026-10-13 16:56:22 +02:00
jochen 9588871d66 go/asks: a message carries its note apart, and an explanation is held to a glance (hq issues 639 and 383)
mesh/merge-gate head 9588871d6629 merged into trunk a0fc82fdcd7c — pass, with a note: builds mesh-sdk (+40 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 9588871d6629 merged into trunk a0fc82fdcd7c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery superseded: a newer head of the same pull request
The operator will not read a Telegram ask of thirty plain lines. The secondary words (who asked, what each
answer does, the expiry, the fingerprint, how to read it whole) travel as Message.Note, so a channel can
show them small or collapsed; an ask's Details go there. Glance names the bound of an explanation: a
summary of at most 300 characters, then at most three points of at most 160.
2026-10-13 16:15:20 +02:00
mesh-admin a0fc82fdcd Merge pull request '0.1.14: the provider loop says a consumer it keeps failing on the bus, and its recovery (hq ADR 0224, issue 272)' (#24) from fix/272-the-provider-loop-says-its-standing into main 2026-10-13 06:37:30 +00:00
jochen d49d3a14d8 0.1.14: the provider loop says a consumer it keeps failing on the bus, and its recovery (hq ADR 0224, issue 272)
mesh/merge-gate head d49d3a14d82a merged into trunk c1ae30b0a2c8 — pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change…
mesh/repo-check head d49d3a14d82a merged into trunk c1ae30b0a2c8 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
No provider on runProvisioner passed announce, and the loop had no standing per
consumer, so the object store, the SQL and document stores, the mail server and
the others failed their consumers in their journals alone. The loop now keeps the
Go copy's standing (failing after five minutes, again every fifteen, recovered on
success, when unasked or retired, and on the first success since start) and says
it through the SDK's emit unless given another announce.
2026-10-13 08:26:33 +02:00
mesh-admin c1ae30b0a2 Merge pull request 'go/outward: the content rule in one place, for the operator channel and node-reads (hq ADR 0310 §4, issue 418)' (#23) from fix/418-node-reads into main 2026-10-12 13:20:22 +00:00
jochen 67bfa22132 go/outward: refuse a password in an address, a value given to an env-style secret name, and base64 wherever its slashes fall (review 53, issue 418)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
mesh/delivery-group group fix/418-node-reads merged: every member is merged
node-reads applies CheckSecret to env, unit and configuration files, where
user:pass@ in a URL, DB_PASS=, MESH_TOKEN= and name = value with spaces passed,
and a random base64 value passed when its slashes or dashes cut it into short
pieces. A run is now judged in pieces only where it is a path, base64 padding
marks a value, and letters alone are random when the case changes as a random
string's does, so a CamelCase name still passes.
2026-10-12 14:27:49 +02:00
jochen 4dc5f29d3e go/outward: the content rule in one place, so the operator channel and node-reads apply the same one (hq ADR 0310 §4, issue 418)
mesh/delivery-group group fix/418-node-reads rejected: a member's own check failed
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery superseded: a newer head of the same pull request
The operator channel's rule (messenger's content.go, with the secret class
split out as in issue 383's branch) moves here as package outward: Check for
what may leave on a channel, CheckSecret for a secret's shape alone, which is
what a node-reads answer is held to. The messenger switches to it once
mesh-catalog #178, which changes that file, has merged.
2026-10-12 13:52:57 +02:00
mesh-admin 9747f3e1b1 Merge pull request 'go: a module reads the values its bucket still keeps for a key (ADR 0307, issue 502)' (#22) from adr-0307-state-history into main 2026-10-12 08:21:13 +00:00
jochen d41d255f28 go: a module reads the values its bucket still keeps for a key, so the tracker can carry them into its log (hq ADR 0307)
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery-group group adr-0307-state-history merged: every member is merged
2026-10-12 10:17:56 +02:00
mesh-admin 5794ccd569 Merge pull request 'go: a module appends to and reads its log through the runtime (ADR 0297, issues 501, 502)' (#21) from adr-0297-log into main 2026-10-11 18:08:52 +00:00
jochen eaed934629 go: a module appends to and reads its log through the runtime, so its record outlives a bucket's history (hq ADR 0297)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
2026-10-11 19:16:09 +02:00
mesh-admin 4cf45b5ca0 Merge pull request 'go: a tool reads its caller from the runtime (hq issue 365)' (#20) from fix/365-a-tool-call-names-its-caller into main 2026-10-11 09:19:42 +00:00
jschoubben d9e24e95a2 go: a tool reads its caller from the runtime, so a record can say who acted (hq issue 365)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
mesh/delivery-group group fix/365-a-tool-call-names-its-caller merged: every member is merged
The runtime names the caller it took from the subject a call arrived on in the
call's _meta; stdio.CallerOf(args) hands it to the tool. Never read from the
arguments, and Unproven when the runtime named none, so a bundle on an older
runtime still says, explicitly, that its caller was not proven.
2026-10-11 05:11:08 +02:00
mesh-admin ee1e1a8303 Merge pull request 'cli: a client reads the answer its failed write left waiting, so a refusal is heard (issue 444)' (#19) from fix/444-a-client-reads-the-answer-its-write-could-not-finish into main 2026-10-11 00:56:01 +00:00
jschoubben 24f5ee81bd cli: after a failed write take only a refusal, since nothing can have run, and pin that the read keeps the ask's deadline (hq issue 444)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
2026-10-11 02:49:17 +02:00
jschoubben b4900f52b8 cli: read the engine's answer after a failed request write, so a refusal is not lost as a broken pipe (hq issue 444)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery superseded: a newer head of the same pull request
2026-10-11 02:43:54 +02:00
mesh-admin 19cfe0ef03 Merge pull request 'go/health: a health tool answers healthy, unhealthy or waits (issue 386, ADR 0283)' (#18) from feat/386-a-wait-for-the-operator into main 2026-10-10 19:13:46 +00:00
jochen 33917f91ac Refuse every form of path or address in a wait's words, a contradicting healthy answer, and a malformed wait at Waiting
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
mesh/delivery-group group feat/386-a-wait-for-the-operator merged: every member is merged
From the review of mesh-sdk#18: quoted and bracketed paths, home and relative
paths, IPv6, host and port, UNC and drive paths passed the check.
2026-10-10 21:10:01 +02:00
jochen 622b6b4f55 Add go/health: a health tool answers healthy, unhealthy or waits (hq ADR 0283, issue 386)
mesh/delivery-group group feat/386-a-wait-for-the-operator checking: 1 of 2 member(s) ready
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery superseded: a newer head of the same pull request
A module waiting for the operator's secret had only 'not healthy' to say,
so its first-node gate failed and the operator was told nothing to do.
The answer gains a state and named waits, and Read refuses any wait it
cannot check, so a malformed wait never excuses a fault.
2026-10-10 21:06:44 +02:00
mesh-admin 67dfeafd51 Merge pull request 'go/asks: an ask says what a channel that proves who answers shows whole, and what Details shows (issue 383)' (#17) from fix/383-an-ask-says-what-a-proving-channel-shows-whole into main 2026-10-10 13:50:56 +00:00
jochen 4f3efc98e0 go/asks: an ask says what a channel that proves who answers shows whole, and what Details shows (hq issue 383)
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
A settings proposal reached the phone with its values masked, so the operator could not
tell what they were approving. An ask now carries Whole, shown in place of Explanation on
a kind that proves its sender and carries the ask's answers, and Details, what the
Details answer shows. Both enter the digest only when given, so an ask without them
digests as before and a router and an asker of different builds still agree on it.
2026-10-10 15:23:41 +02:00
mesh-admin 7605a3fa56 Merge pull request 'go: a counter by compare-and-set, so a number is handed out once (hq research 042)' (#16) from feat/a-counter-by-compare-and-set into main 2026-10-10 10:37:11 +00:00
jochen 9c548753b8 go: say an Increment error other than a lost write may have spent its number, refuse null, and prove the writers collided
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery delivered
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/delivery-group group feat/a-counter-by-compare-and-set merged: every member is merged
The review of 2026-10-10: a timeout can come after the write landed, so a caller tolerates gaps; a null value
is no counter; and a contention test where no write was lost would prove nothing.
2026-10-10 12:33:41 +02:00
jochen c58ce2c7fe go: a counter by compare-and-set, so a number is handed out once (hq research 042)
mesh/delivery-group group feat/a-counter-by-compare-and-set checking: 0 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
The coming issues module hands out issue numbers from its own state; two instances must never hand out one
number twice. Create and Update already write by compare-and-set (ADR 0259), so State(...).Increment reads,
writes one more at the revision read, and on a lost write reads and tries again, bounded by the caller's
attempts. A lost write past the bound is ErrChanged, any other refusal and a value that is no counter are
returned at once: nothing is reported counted that was not written.

Part of the issue tracker decision, hq research 042.
2026-10-10 12:31:05 +02:00
mesh-admin ff1fcd4441 Merge pull request 'go/cli: carry mesh-cli's standard input with its line, for the controller's terminal alone (hq ADR 0259 §10, ADR 0272)' (#15) from feat/a-terminal-line-takes-standard-input into main 2026-10-09 15:31:47 +00:00
jschoubben f047d0a4a9 go/cli: carry mesh-cli's standard input with its line, bounded at 64 KiB, for the controller's terminal alone
mesh/merge-gate pass: builds mesh-sdk, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), go/merge-check.sh (go)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1791659640519725407 (novox/mesh-sdk at 19cfe0ef into main, done, …
mesh/delivery-group group feat/a-terminal-line-takes-standard-input closed: a member was closed before the group merged
A secret given at the terminal (secret accept … --from -) has no way through mesh-cli today: the controller
runs every line with no standard input. The request now carries it, refused over 64 KiB by mesh-cli and by
the reader alike, and MaxRequest grows by what base64 makes of that bound.
2026-10-09 16:33:44 +02:00
52 changed files with 8878 additions and 71 deletions
+26 -6
View File
@@ -20,7 +20,13 @@ Not the manifest schema (the control plane owns and parses that, in Go). The sha
contains — the contract both a provider and a consumer conform to), and the tool and event types.
These change rarely and deliberately; when one does, a rebuild of everything is *correct*.
### `provisioner` — the reconcile harness every provider shares
### `provider-loop` — the reconcile harness every provider on the grant files shares
Called `provisioner` until design 57 (novox/hq issue 678); that word is mesh-provisioner's now. The old path and
names (`/provisioner`, `runProvisioner`, `provisionerTools`) stay aliases until the providers importing them move to
the Go helper that answers consumers over the bus (`go/provider`, design 57 B2 and B3) and this loop retires with the
grant files. Its events and tools keep their names (`provisioner.failing`, `provisioner_retirement`, …): the
controller and the operator read them.
The loop is identical for postgres, redis, minio and umami: watch the grants directory, for each
requested grant call the provider's adapter, write the sealed credential, handle withdrawal, and
keep converging (`watch`). That loop lives here. A **module provides only the adapter** — "create
@@ -39,11 +45,21 @@ never called to retire**; it is the delete of an adapter without `delete`, reach
person's `provisioner_delete`. A stable set of more than 3, or more than half of those held where more
than one is held, is not retired without a person: the provider waits for
`provisioner_retire_approve` or `provisioner_retire_reject`, and says `provisioner.retirement`
(`waiting`) through the `announce` option. Asked for again, a retired consumer is created — re-enabled
(`waiting`). Asked for again, a retired consumer is created — re-enabled
— at once. Given `module`, the four tools (`provisioner_retirement`, `provisioner_retire_approve`,
`provisioner_retire_reject`, `provisioner_delete`) are registered for it; `provisionerTools(handle)`
builds them otherwise. Additions and changes are never delayed.
**A provider that keeps failing a consumer says so on the bus** (novox/hq ADR 0224). A consumer whose
create, check (`holds`) or secret has failed for five minutes without one success between
(`failingAfterMs`) is announced `provisioner.failing` — the consumer, its machine, the class of error
(`credentials-rejected`, `unreachable`, `secret-unreadable`, `refused`; an adapter's `classify` is
believed over the words), the error with the password removed, since when and how many attempts — and
again every fifteen minutes while it lasts. The first success after it is `provisioner.recovered`, as is
a consumer no longer asked for or retired, and the first success for each consumer since the process
started. Every event goes through the `announce` option, which defaults to the SDK's `emit`: a provider
says them without asking, and a failed publish is said in its journal, never failing the act.
### `tools` — the tool-serving harness
`registerModuleTools`, the `ToolDefinition` type, and the registry/worker that loads a module's
tools and exposes them through the mesh's command surface. *How* a tool is declared and served is
@@ -96,7 +112,8 @@ an event under the module's own name or kind, `stdio.SeatTake` takes a held seat
and `stdio.SeatAnswer` carry a proof (a code typed by a person, by request and reply, never kept),
`stdio.SeatRecord` reads a record kept for the module, and `State(...).Create` and `.Update` write by
compare-and-set. The runtime does each only as far as the module's membership lists. `go/asks` is the
contract on top: the `Ask` an asker publishes and the checks it is held to, the `Warrant` it acts on
contract on top: the `Ask` an asker publishes and the checks it is held to (its `Whole` is what a channel that proves
who answers shows in place of its `Explanation`, and its `Details` what the Details answer shows, hq issue 383), the `Warrant` it acts on
(`For`, `Says`), and the `Message`, `Choice`, `LinkAsked` and `Code` a channel takes and says, with the
subjects each travels on.
`go/asker` is an asking module's client (ADR 0259 §10): `Ask` keeps an ask in the module's own state and then
@@ -104,6 +121,9 @@ publishes it under the module's name; `Cancel`; `Where` reads its state from the
`Take` hand the module each warrant once — marked acted on by compare-and-set before the module acts, and only
for an ask it made, an option it offered, at that option's level, before it expired. A lost compare-and-set is
`stdio.ErrChanged`, told by the runtime's error code.
`State(...).Increment(key, attempts)` is a counter on top of the two (for numbers handed out once, such as an
issue's): it reads the key, writes one more at the revision read, and on a lost write reads and tries again, at
most `attempts` times; past that it is `stdio.ErrChanged` and nothing was counted.
### `primitives` — sealing, semver, resolved-env
Seal/unseal for secrets, semantic version comparison, and the accessor a module uses to read its
@@ -132,12 +152,12 @@ import { UmamiClient } from "../client.js"; // the client lives in the module, n
registerModuleTools("umami", (env) => getUmamiTools(new UmamiClient(env.UMAMI_URL)));
```
A provider's provisioner:
A provider's loop:
```ts
import { runProvisioner, Grant, Credential } from "@novox/mesh-sdk/provisioner";
import { runProviderLoop, Grant, Credential } from "@novox/mesh-sdk/provider-loop";
runProvisioner("analytics", {
runProviderLoop("analytics", {
async create(grant: Grant): Promise<Credential> { /* create a umami site, return the grant */ },
async remove(grant: Grant): Promise<void> { /* delete the site */ },
});
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+3 -3
View File
@@ -2,8 +2,8 @@
`stdio.Serve("", tools)` speaks MCP over stdio to the node's tool runtime, as the module the runtime
names (novox/hq ADR 0188, ADR 0193). A `Tool` is a name, a description, an argument schema and a
function; calls run concurrently. A tool named `<seat>.<verb>` is the seat's implementation.
`stdio.Emit(type, body)` publishes an event as the module, `stdio.Subscribe(pattern, handler)` receives the module's events (taken when the handler returns nil, offered again when it returns an error — the one rule is in the repository README, "Taking an event") and `stdio.Ask(key, body)` calls a tool, and `stdio.State(name)` gets, puts, deletes, lists and watches the module's declared state — all through the runtime (novox/hq ADR 0193, 0198, 0201). Standard library only.
function; calls run concurrently. `stdio.CallerOf(args)` inside a tool is who called it: the bus user the runtime took from the subject the call arrived on, named in the call's `_meta`, never read from the arguments — or `stdio.Unproven` when the runtime named none (novox/hq issue 365). A tool named `<seat>.<verb>` is the seat's implementation.
`stdio.Emit(type, body)` publishes an event as the module, `stdio.Subscribe(pattern, handler)` receives the module's events (taken when the handler returns nil, offered again when it returns an error — the one rule is in the repository README, "Taking an event") and `stdio.Ask(key, body)` calls a tool, and `stdio.State(name)` gets, puts, deletes, lists, watches and reads the kept history of the module's declared state, and `stdio.Log(name)` appends to, reads, and measures a log of operations the module declared — all through the runtime (novox/hq ADR 0193, 0198, 0201, 0297, 0307). `stdio.Config(key)` and `stdio.Bound(provision, key)` read, and `stdio.WatchConfig` and `stdio.WatchBound` watch, the module's own config values and the non-secret values of what it is bound to, for what its manifest declares: the runtime asks its machine's provisioner client as the module it launched, never a module the code names. `stdio.Secret(name)` is refused while the module's code runs in the tool runner; its secrets reach it through its files (novox/hq ADR 0349 §14, ADR 0370). `provider` is a providing module's half of a binding (novox/hq ADR 0349 §12, issue 678): it sees each consumer that binds the provider in the plans of the assignments mirror, writes one entry per consumer to `<module>_bound` (`<node>.<module>.<provision>[.<local>]`: served values composed with the consumer's identity, the login, and the name of the vault's credential, never a secret), hands the provider's adapter the vault's value (`provision/<provider>/<provision>` shared, `pair/<provider>/<provision>/<node>/<module>[/<local>]` per consumer), refuses a derived value to a consumer bound under a local name, naming both ends, and removes the entry of one that unbinds; the bus and the vault are interfaces the provider wires, the vault as `provider.ConsumersFile`, the file its machine's provisioner client fills from `${consumers:<provision>}` with each bound consumer's credential (novox/hq ADR 0373), since a provider in the tool runner never asks the vault itself. Standard library only.
## Installing
@@ -16,4 +16,4 @@ at the same version as the TypeScript package.
What it holds, under ADR 0039's test: the stdio loop and the tool type, and `cli`, the local protocol
`mesh-cli` speaks to the node-engine on its machine (novox/hq ADR 0272): the socket's path and one request and one
answer. Nothing else — no bus, no client, nothing of any module's. `go build ./example` and pipe a `tools/list` request into it.
answer; and `seal`, the mesh's sealed box (novox/hq ADR 0206: X25519, HKDF-SHA256, AES-256-GCM, `{v, eph, iv, tag, ct}`), byte for byte the box the licence manager made before it lived here and the TypeScript `@novox/mesh-sdk/seal` makes. Nothing else — no bus, no client, nothing of any module's. `go build ./example` and pipe a `tools/list` request into it.
+82 -3
View File
@@ -154,7 +154,9 @@ func canonical(b *strings.Builder, v string) {
//
// It is over the ask's named fields in a fixed order, each written canonically, and the expiry as UTC
// RFC 3339 to the nanosecond — never over a language's encoding of the struct, so a field added to Ask
// later changes no digest until it is added here, on purpose.
// later changes no digest until it is added here, on purpose. Whole and Details (novox/hq issue 383) follow
// the options only when the ask gives either: an ask without them digests as it did before they existed, so
// a router and an asker of different builds still agree on every such ask.
func (a Ask) Digest() string {
var b strings.Builder
b.WriteString("novox.ask.v1\n")
@@ -168,6 +170,18 @@ func (a Ask) Digest() string {
canonical(&b, v)
}
}
// Whole and Details are written whenever either or WholeDetails is given, and WholeDetails after them whenever
// it is given (novox/hq ADR 0347): the fields after the options are then always one of three fixed shapes —
// none, the two, or the three — and the count of values the sequence holds tells them apart, so no two asks
// encode the same (the review of mesh-sdk #25). An ask with none of them digests as before they existed, and
// one without WholeDetails as before it existed.
if a.Whole != "" || a.Details != "" || a.WholeDetails != "" {
canonical(&b, a.Whole)
canonical(&b, a.Details)
}
if a.WholeDetails != "" {
canonical(&b, a.WholeDetails)
}
sum := sha256.Sum256([]byte(b.String()))
return "sha256:" + hex.EncodeToString(sum[:])
}
@@ -190,6 +204,65 @@ type Ask struct {
// About is what the ask is about (a condition's key): a newer ask about it replaces the older.
About string `json:"about,omitempty"`
Urgent bool `json:"urgent,omitempty"`
// Whole is the explanation with its exact values whole, shown in place of Explanation on a channel kind
// that proves who answers (verified-sender) and carries the ask's answers (novox/hq issue 383): what the
// person approves — a mount point, a share, a private address — must be readable where they approve it.
// The asker withholds a value shaped like a secret in it, and the router refuses the ask when one is left;
// Explanation stays under the whole content rule everywhere else. Empty, Explanation is shown everywhere.
Whole string `json:"whole,omitempty"`
// Details are the ask's secondary words, line by line under the content rule: who proposed it, a
// fingerprint, how to read it whole. The router shows them in the note of the ask's own message
// (Message.Note), after what the ask's options do and when it expires — never in its body (novox/hq
// issue 639).
Details string `json:"details,omitempty"`
// WholeDetails are the Details with their exact values whole, shown in place of Details where Whole is shown
// in place of the explanation (a kind that proves who answers, given the ask's answers): every value an
// approval sets, whole, so the person reads exactly what they approve (novox/hq issue 383). Held, like
// Whole, to the secret rule alone; the router refuses the ask when a secret's shape is left in it. Empty,
// Details are shown everywhere.
WholeDetails string `json:"whole-details,omitempty"`
}
// What an ask's explanation is held to, so it reads at a glance on a phone (novox/hq issue 639): a summary
// of two or three short sentences on its first line — what changes, why, what it costs or risks — then at
// most three points, a line each, only where they add something. Whole is held to the same. Who asked, when,
// the fingerprint and how to read it whole are Details, never the explanation.
const (
SummaryLength = 300
MostPoints = 3
PointLength = 160
)
// Glance says why an explanation does not read at a glance, or nil: its first line, the summary, longer than
// SummaryLength; more than MostPoints lines after it; a point longer than PointLength. Blank lines are not
// counted. An asker holds its asks to it in its own tests; the router says, without refusing, an ask that
// does not hold to it.
func Glance(explanation string) error {
var lines []string
for _, l := range strings.Split(explanation, "\n") {
if strings.TrimSpace(l) != "" {
lines = append(lines, strings.TrimSpace(l))
}
}
var problems []string
switch {
case len(lines) == 0:
problems = append(problems, "it has no summary")
case len([]rune(lines[0])) > SummaryLength:
problems = append(problems, fmt.Sprintf("its summary is %d characters, more than %d", len([]rune(lines[0])), SummaryLength))
}
if len(lines) > 1+MostPoints {
problems = append(problems, fmt.Sprintf("it has %d points after its summary, more than %d", len(lines)-1, MostPoints))
}
for i, l := range lines {
if i > 0 && len([]rune(l)) > PointLength {
problems = append(problems, fmt.Sprintf("its point %d is %d characters, more than %d", i, len([]rune(l)), PointLength))
}
}
if len(problems) > 0 {
return errors.New("the explanation does not read at a glance: " + strings.Join(problems, "; "))
}
return nil
}
// The bounds of an ask (novox/hq ADR 0234 §8, ADR 0259 §4).
@@ -420,8 +493,14 @@ type Message struct {
Title string `json:"title"`
Body string `json:"body"`
Buttons []Button `json:"buttons,omitempty"`
Urgent bool `json:"urgent,omitempty"`
Silent bool `json:"silent,omitempty"`
// Note is the message's secondary words, after its body and its buttons: who asked and when, what each
// answer does, when it expires, and the ask's Details. A channel shows it smaller, collapsed or set apart
// where it can, and after the body where it cannot (novox/hq issue 639). A channel that bounds a message's
// length cuts the body first and the note only when the title and the note alone do not fit, saying each
// cut (novox/hq ADR 0347 §5).
Note string `json:"note,omitempty"`
Urgent bool `json:"urgent,omitempty"`
Silent bool `json:"silent,omitempty"`
// Reply is the Choice or LinkAsked this answers, by its ID: the channel shows it where that was made.
Reply string `json:"reply,omitempty"`
// To is the account linked as the operator on this kind, for a channel that verifies its sender: where
+110 -2
View File
@@ -230,9 +230,31 @@ func TestDigestsAreCanonicalOverNamedFields(t *testing.T) {
t.Errorf("changing %s left the digest the same", name)
}
}
// Whole and Details (issue 383) are in the digest once given, and change it; an ask without either digests
// exactly as before they existed — the encoding by hand, so no build can drift from it unseen.
plain := Ask{ID: "p", Headline: "h", Explanation: "e", Who: Operator, Expires: now, OnExpiry: "o", About: "a",
Options: []Option{{ID: "ok", Label: "OK", Does: "d", Level: Acknowledge}}}
byHand := sha256.Sum256([]byte("novox.ask.v1\n1:p\n1:h\n1:e\n8:operator\n20:2026-10-08T14:00:00Z\n1:o\n1:a\n5:false\n1:1\n" +
"2:ok\n2:OK\n1:d\n11:acknowledge\n0:\n"))
if plain.Digest() != "sha256:"+hex.EncodeToString(byHand[:]) {
t.Errorf("an ask without Whole and Details does not digest as it did before them: %s", plain.Digest())
}
withWhole, withDetails := plain, plain
withWhole.Whole = "e, whole"
withDetails.Details = "fingerprint 1234"
if withWhole.Digest() == plain.Digest() || withDetails.Digest() == plain.Digest() || withWhole.Digest() == withDetails.Digest() {
t.Error("Whole or Details left the digest the same")
}
both := withWhole
both.Details = "fingerprint 1234"
changed := both
changed.Whole += "!"
if changed.Digest() == both.Digest() {
t.Error("changing Whole beside Details left the digest the same")
}
// A field added to Ask or Option without being added to Digest fails here.
if n := reflect.TypeOf(Ask{}).NumField(); n != 9 {
t.Errorf("Ask has %d fields, and Digest covers 9: add the new one to Digest and to this test", n)
if n := reflect.TypeOf(Ask{}).NumField(); n != 12 {
t.Errorf("Ask has %d fields, and Digest covers 12: add the new one to Digest and to this test", n)
}
if n := reflect.TypeOf(Option{}).NumField(); n != 5 {
t.Errorf("Option has %d fields, and Digest covers 5: add the new one to Digest and to this test", n)
@@ -257,3 +279,89 @@ func TestAWarrantForAnAskWithNoExpiryIsRefused(t *testing.T) {
t.Error("a warrant for an ask with no expiry was taken")
}
}
// An explanation that reads at a glance passes; a wall of text says each way it is one (novox/hq issue 639).
func TestAnExplanationReadsAtAGlance(t *testing.T) {
if err := Glance("Your agents' memory follows you to every machine. A file changed on two keeps both.\n\n• Only the memory folders are carried\n• A conflict reaches you as a warning"); err != nil {
t.Fatalf("a summary and two points: %v", err)
}
var wall strings.Builder
wall.WriteString("ADR 0343 was proposed by fix-367 and approved by the knowledge base's own review.")
for i := 1; i <= 13; i++ {
wall.WriteString("\n" + strings.Repeat("A clause lead. ", 3))
}
err := Glance(wall.String())
if err == nil || !strings.Contains(err.Error(), "13 points") {
t.Fatalf("thirteen clause lines pass as a glance: %v", err)
}
if err := Glance(strings.Repeat("word ", 70)); err == nil || !strings.Contains(err.Error(), "summary is 349") {
t.Fatalf("a long summary passes: %v", err)
}
if err := Glance("short.\n" + strings.Repeat("p", PointLength+1)); err == nil || !strings.Contains(err.Error(), "point 1") {
t.Fatalf("a long point passes: %v", err)
}
if err := Glance(" \n"); err == nil {
t.Fatal("an empty explanation passes")
}
}
// A message's note travels beside its body, and is left out of the encoding when empty, so a channel older
// than it reads every message it read before.
func TestAMessageCarriesItsNoteApart(t *testing.T) {
raw, _ := json.Marshal(Message{Handle: "h", Title: "t", Body: "b", Note: "Asked by x."})
if !strings.Contains(string(raw), `"note":"Asked by x."`) {
t.Fatalf("the note is not encoded: %s", raw)
}
raw, _ = json.Marshal(Message{Handle: "h", Title: "t"})
if strings.Contains(string(raw), "note") {
t.Fatalf("an empty note is encoded: %s", raw)
}
}
// WholeDetails is digested when given, and an ask without it digests as before it existed: a router and an asker
// of different builds still agree on every such ask.
func TestWholeDetailsAreDigestedOnlyWhenGiven(t *testing.T) {
a := Ask{ID: "a", Headline: "h", Explanation: "e", Details: "d"}
before := a.Digest()
a.WholeDetails = "+ shares: media=/storage/media"
if a.Digest() == before {
t.Fatal("the whole details are not in the digest")
}
b := a
b.WholeDetails = "+ shares: media=/storage/other"
if a.Digest() == b.Digest() {
t.Fatal("two whole details digest the same")
}
a.WholeDetails = ""
if a.Digest() != before {
t.Fatal("an ask without whole details digests otherwise than before")
}
}
// No two asks encode the same (the review of mesh-sdk #25): whole details given beside nothing else are not read as
// a Whole and Details that happen to spell the same values, nor as any other split of the three fields.
func TestWholeDetailsCollideWithNoOtherAsk(t *testing.T) {
base := Ask{ID: "a", Headline: "h", Explanation: "e"}
shapes := []Ask{}
for _, f := range []struct{ whole, details, wholeDetails string }{
{"", "", "+ x: 1"},
{"whole-details", "+ x: 1", ""},
{"", "+ x: 1", ""},
{"+ x: 1", "", ""},
{"", "", "whole-details"},
{"", "+ x: 1", "+ x: 1"},
{"+ x: 1", "+ x: 1", ""},
{"", "", ""},
} {
a := base
a.Whole, a.Details, a.WholeDetails = f.whole, f.details, f.wholeDetails
shapes = append(shapes, a)
}
seen := map[string]int{}
for i, a := range shapes {
if j, ok := seen[a.Digest()]; ok {
t.Errorf("asks %d and %d digest the same: %+v / %+v", j, i, shapes[j], a)
}
seen[a.Digest()] = i
}
}
+59
View File
@@ -0,0 +1,59 @@
package stdio
import (
"reflect"
"regexp"
"sync"
"unsafe"
)
// Unproven is the caller of a call the runtime named no caller for: one that arrived on the subject that names
// none, or from a runtime that names none (novox/hq issue 365). Said as a word, never left empty, so a record
// shows it was not proven.
const Unproven = "unproven"
// MetaCaller is where in a `tools/call`'s `_meta` the runtime names the caller: the bus user that published the
// call, which the runtime takes from the subject the call arrived on — that subject's last token, which the
// bus lets each user publish with its own name alone (novox/hq issue 365, by ADR 0259 §3's precedent). The
// runtime writes `_meta`; a caller writes only `arguments`, so no caller can name another.
const MetaCaller = "mesh/caller"
// busUser is the shape of a bus user's name: `<node>.<module>`, `person.<name>`, `controller` — names whose
// parts are safe in a subject. Anything else the runtime named is not a caller and reads as Unproven.
var busUser = regexp.MustCompile(`^[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*$`)
// calls holds the caller of every call while its tool runs, by the arguments map that call was handed.
var calls sync.Map // unsafe.Pointer → string
func argsKey(args map[string]any) unsafe.Pointer {
return reflect.ValueOf(args).UnsafePointer()
}
// CallerOf is who called the tool that was handed args, as the runtime proved it: the bus user that
// published the call, or Unproven. Read from the runtime, never from the arguments' contents — a `caller`
// argument names nobody. Valid while the tool runs; arguments no call was handed have no caller.
func CallerOf(args map[string]any) string {
if args == nil {
return Unproven
}
if c, ok := calls.Load(argsKey(args)); ok {
return c.(string)
}
return Unproven
}
// callerIn is the caller a `tools/call`'s `_meta` names, or Unproven.
func callerIn(meta map[string]any) string {
if s, ok := meta[MetaCaller].(string); ok && busUser.MatchString(s) {
return s
}
return Unproven
}
// runAs runs a tool with the caller held for its arguments for as long as it runs.
func runAs(t Tool, caller string, args map[string]any) (any, error) {
key := argsKey(args)
calls.Store(key, caller)
defer calls.Delete(key)
return t.Run(args)
}
+105
View File
@@ -0,0 +1,105 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"testing"
"time"
)
// callWith serves one tool that answers the caller CallerOf reads from its arguments, sends it one tools/call
// with the params given, and returns the caller it answered.
func callWith(t *testing.T, params map[string]any) string {
t.Helper()
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() {
_ = ServeOn("tracker", []Tool{{Name: "who", Run: func(args map[string]any) (any, error) {
return map[string]any{"caller": CallerOf(args)}, nil
}}}, inR, outW)
}()
defer inW.Close()
b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": params})
_, _ = inW.Write(append(b, '\n'))
lines := bufio.NewScanner(outR)
done := make(chan string, 1)
go func() {
if lines.Scan() {
var m struct {
Result struct {
Content []struct {
Text string `json:"text"`
} `json:"content"`
} `json:"result"`
}
_ = json.Unmarshal(lines.Bytes(), &m)
var answer struct {
Caller string `json:"caller"`
}
if len(m.Result.Content) > 0 {
_ = json.Unmarshal([]byte(m.Result.Content[0].Text), &answer)
}
done <- answer.Caller
}
}()
select {
case got := <-done:
return got
case <-time.After(3 * time.Second):
t.Fatal("the bundle did not answer the call")
return ""
}
}
// **A tool reads its caller from the runtime, never from the call's arguments** (novox/hq issue 365): the
// runtime names the caller it took from the subject the call arrived on in the call's `_meta`, and CallerOf
// hands it to the tool by the arguments the tool was given.
func TestAToolReadsTheCallerTheRuntimeNamed(t *testing.T) {
got := callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"title": "x"},
"_meta": map[string]any{MetaCaller: "shanks.node-tools"}})
if got != "shanks.node-tools" {
t.Fatalf("the tool read caller %q, want the one the runtime named", got)
}
}
// A caller named in the arguments is not the caller: an argument is the caller's to write.
func TestACallerInTheArgumentsIsNeverTheCaller(t *testing.T) {
got := callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"caller": "novox.node-tools",
"_meta": map[string]any{MetaCaller: "novox.node-tools"}, MetaCaller: "novox.node-tools"}})
if got != Unproven {
t.Fatalf("a caller the arguments claimed was read as %q, want %q", got, Unproven)
}
got = callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"caller": "novox.node-tools"},
"_meta": map[string]any{MetaCaller: "shanks.node-tools"}})
if got != "shanks.node-tools" {
t.Fatalf("with a caller in the arguments the tool read %q, want the runtime's", got)
}
}
// A runtime that names no caller — one from before issue 365, or a call on the subject that names none — is
// said as Unproven, explicitly; a caller that is not a bus user's name is Unproven too.
func TestACallTheRuntimeNamedNoCallerForIsUnproven(t *testing.T) {
for _, params := range []map[string]any{
{"name": "who", "arguments": map[string]any{}},
{"name": "who"},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{}},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: ""}},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: "a b"}},
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: 7}},
} {
if got := callWith(t, params); got != Unproven {
t.Errorf("%v: the tool read caller %q, want %q", params, got, Unproven)
}
}
}
// Arguments the runtime never handed a tool have no caller.
func TestArgumentsOfNoCallHaveNoCaller(t *testing.T) {
if got := CallerOf(map[string]any{"x": 1}); got != Unproven {
t.Fatalf("CallerOf of a map no call was given answered %q", got)
}
if got := CallerOf(nil); got != Unproven {
t.Fatalf("CallerOf(nil) answered %q", got)
}
}
+44 -7
View File
@@ -24,9 +24,13 @@ import (
// address (ADR 0272 §3).
const Socket = "/run/nox-mesh-host/cli.sock"
// MaxRequest bounds what the node-engine reads of a request. A command line is a few hundred bytes; settings given
// inline are a few kilobytes.
const MaxRequest = 64 << 10
// MaxStdin bounds the standard input mesh-cli carries for a line: a secret, a key, a settings file — kilobytes. More
// is refused by mesh-cli before anything is asked, and by the node-engine and the controller again.
const MaxStdin = 64 << 10
// MaxRequest bounds what the node-engine reads of a request: a command line of a few hundred bytes, settings given
// inline of a few kilobytes, and standard input of at most MaxStdin, which JSON writes as base64 (4 bytes for every 3).
const MaxRequest = 64<<10 + MaxStdin*4/3 + 4
// MaxAnswer bounds what mesh-cli reads of an answer. The controller cuts what it runs to fit one bus message, and
// says so; this is the local bound above that.
@@ -36,9 +40,16 @@ const MaxAnswer = 16 << 20
// and a minute for the link on either side.
const AskWithin = 6 * time.Minute
// Request is what mesh-cli asks: the controller's command line, as words.
// Request is what mesh-cli asks: the controller's command line, as words, and what mesh-cli's own standard input
// held when it was not a terminal.
//
// **Stdin is for the controller's terminal alone** (novox/hq ADR 0259 §10, ADR 0272): a secret given with
// `secret accept … --from -`, so it never stands on a command line. The controller gives it to a line that runs as
// its terminal and refuses any other line that carries it; nobody keeps it — not the node-engine, not the
// controller's record of calls, not its journal.
type Request struct {
Line []string `json:"line"`
Line []string `json:"line"`
Stdin []byte `json:"stdin,omitempty"`
}
// Answer is what comes back: what the command printed, how it exited, and whether it ran as the controller's
@@ -87,6 +98,9 @@ func ReadRequest(r io.Reader) (Request, error) {
if len(req.Line) == 0 {
return Request{}, errors.New("the request names no command")
}
if len(req.Stdin) > MaxStdin {
return Request{}, fmt.Errorf("%w: standard input of more than %d bytes", ErrTooLong, MaxStdin)
}
return req, nil
}
@@ -102,9 +116,18 @@ func ReadAnswer(r io.Reader) (Answer, error) {
// Ask is mesh-cli's side: connect to the socket, ask, and wait for the answer within AskWithin or ctx.
func Ask(ctx context.Context, socket string, line []string) (Answer, error) {
return AskWith(ctx, socket, Request{Line: line})
}
// AskWith asks a whole request: a line, and the standard input it carries, at most MaxStdin.
func AskWith(ctx context.Context, socket string, req Request) (Answer, error) {
line := req.Line
if len(line) == 0 {
return Answer{}, errors.New("no command given")
}
if len(req.Stdin) > MaxStdin {
return Answer{}, fmt.Errorf("standard input is more than %d bytes, which mesh-cli does not carry; nothing was asked", MaxStdin)
}
ctx, cancel := context.WithTimeout(ctx, AskWithin)
defer cancel()
var d net.Dialer
@@ -116,8 +139,22 @@ func Ask(ctx context.Context, socket string, line []string) (Answer, error) {
if deadline, ok := ctx.Deadline(); ok {
_ = conn.SetDeadline(deadline)
}
if err := WriteRequest(conn, Request{Line: line}); err != nil {
return Answer{}, fmt.Errorf("asking the node-engine: %w", err)
if err := WriteRequest(conn, req); err != nil {
// The engine may have answered before it read the request — a refusal (busy, an unknown account) written
// and the connection closed — and the write then fails on the close. That answer is already waiting, and
// it says why where the write error says only that the pipe broke (novox/hq issue 444, found under issue
// 434). So read it, within the deadline the connection already has, and only when none came say the write.
// Only a refusal is taken: the request was never wholly written, so nothing can have run, and any other
// answer here is not one a correct engine gives — passing it on as success would hide the failure.
a, rerr := ReadAnswer(conn)
if rerr != nil {
return Answer{}, fmt.Errorf("asking the node-engine: %w; and no answer came (%v)", err, rerr)
}
if a.Refused == "" {
return Answer{}, fmt.Errorf("asking the node-engine: %w; and the answer that came is not a refusal, "+
"though the request was never wholly written", err)
}
return a, nil
}
a, err := ReadAnswer(conn)
if err != nil {
+36
View File
@@ -3,6 +3,7 @@ package cli
import (
"bytes"
"context"
"encoding/json"
"errors"
"net"
"path/filepath"
@@ -44,6 +45,17 @@ func TestARequestAndAnAnswerSurviveTheirRoundTrip(t *testing.T) {
// A request over the bound is refused, not read whole; one naming no command is refused.
func TestARequestOverTheBoundOrNamingNothingIsRefused(t *testing.T) {
long := Request{Line: []string{strings.Repeat("x", MaxRequest)}}
big := Request{Line: []string{"secret"}, Stdin: bytes.Repeat([]byte("s"), MaxStdin+1)}
var bb bytes.Buffer
if err := WriteRequest(&bb, big); err != nil {
t.Fatal(err)
}
if _, err := ReadRequest(&bb); !errors.Is(err, ErrTooLong) {
t.Fatalf("standard input over its bound was read: %v", err)
}
if _, err := AskWith(context.Background(), "/nonexistent.sock", big); err == nil || !strings.Contains(err.Error(), "nothing was asked") {
t.Fatalf("mesh-cli would ask with standard input over its bound: %v", err)
}
var b bytes.Buffer
if err := WriteRequest(&b, long); err != nil {
t.Fatal(err)
@@ -103,3 +115,27 @@ func TestAnUnknownOutcomeIsNotARefusal(t *testing.T) {
t.Fatalf("an unknown outcome with no call names one: %+v", b)
}
}
// Standard input up to its bound travels whole, beside the line: a secret given with --from -.
func TestStandardInputTravelsWithItsLine(t *testing.T) {
given := append(bytes.Repeat([]byte{0, 0xff, '\n', 'x'}, MaxStdin/4-1), "end"...)
var b bytes.Buffer
if err := WriteRequest(&b, Request{Line: []string{"secret", "accept", "anchor", "telegram", "telegram-token", "--from", "-"},
Stdin: given}); err != nil {
t.Fatal(err)
}
if b.Len() > MaxRequest {
t.Fatalf("a request with standard input at its bound is %d bytes, over MaxRequest %d", b.Len(), MaxRequest)
}
got, err := ReadRequest(&b)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got.Stdin, given) || got.Line[1] != "accept" {
t.Fatalf("standard input did not travel whole: %d bytes, want %d", len(got.Stdin), len(given))
}
none, _ := json.Marshal(Request{Line: []string{"status"}})
if strings.Contains(string(none), "stdin") {
t.Fatalf("a request without standard input says it has one: %s", none)
}
}
+123
View File
@@ -0,0 +1,123 @@
package cli
import (
"context"
"errors"
"net"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// anEngineThatClosesFirst listens like the node-engine before mesh-host#80: it writes its answer, if any, and
// closes without reading the request (novox/hq issue 434). It answers one connection.
func anEngineThatClosesFirst(t *testing.T, answer *Answer) string {
t.Helper()
socket := filepath.Join(t.TempDir(), "cli.sock")
l, err := net.Listen("unix", socket)
if err != nil {
t.Skipf("no unix socket here: %v", err)
}
t.Cleanup(func() { l.Close() })
go func() {
conn, err := l.Accept()
if err != nil {
return
}
if answer != nil {
_ = WriteAnswer(conn, *answer)
}
conn.Close()
}()
return socket
}
// aLineTheSocketCannotHold is a request larger than a unix socket buffers, so its write cannot finish before
// the engine reads it — and an engine that closes first makes it fail every time, not only when it loses a race.
func aLineTheSocketCannotHold() []string { return []string{"status", strings.Repeat("x", 4<<20)} }
// An engine that refuses and closes before reading the request is heard: its refusal reaches the caller, not
// the broken pipe its close left the client's write (novox/hq issue 444).
func TestARefusalWrittenBeforeTheRequestIsReadReachesTheCaller(t *testing.T) {
busy := Refusal("the node-engine is busy: too many lines at once")
socket := anEngineThatClosesFirst(t, &busy)
a, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
if err != nil {
t.Fatalf("the engine said %q and the caller heard: %v", busy.Refused, err)
}
if a.Refused != busy.Refused || a.Exit != 1 {
t.Fatalf("the engine said %q and the caller heard %+v", busy.Refused, a)
}
}
// When the write fails and no answer is waiting, the error says both: what failed, and that no answer came.
func TestAFailedWriteWithNoAnswerSaysBoth(t *testing.T) {
socket := anEngineThatClosesFirst(t, nil)
start := time.Now()
_, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
if err == nil {
t.Fatal("a request the engine never read, and never answered, came back without an error")
}
if !strings.Contains(err.Error(), "asking the node-engine") || !strings.Contains(err.Error(), "no answer came") {
t.Fatalf("the error does not say what failed and that no answer came: %v", err)
}
if waited := time.Since(start); waited > 30*time.Second {
t.Fatalf("waited %s on a connection the engine had closed", waited)
}
}
// After a failed write only a refusal is taken: nothing can have run, so an answer saying it ran and exited 0
// is an error, never a success (novox/hq issue 444).
func TestAnAnswerOtherThanARefusalAfterAFailedWriteIsAnError(t *testing.T) {
ran := Answer{Exit: 0, Stdout: []byte("ok")}
socket := anEngineThatClosesFirst(t, &ran)
a, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
if err == nil {
t.Fatalf("a request never wholly written came back as success: %+v", a)
}
if !strings.Contains(err.Error(), "not a refusal") {
t.Fatalf("the error does not say the answer was not a refusal: %v", err)
}
}
// Reading after a failed write never waits longer than the ask already allowed: a peer that never reads and
// never answers is given up on at the context's deadline, not later (novox/hq issue 444).
func TestAFailedWriteWaitsNoLongerThanTheDeadline(t *testing.T) {
socket := filepath.Join(t.TempDir(), "cli.sock")
l, err := net.Listen("unix", socket)
if err != nil {
t.Skipf("no unix socket here: %v", err)
}
t.Cleanup(func() { l.Close() })
held := make(chan net.Conn, 1)
go func() {
conn, err := l.Accept()
if err != nil {
return
}
held <- conn // neither read nor answered, nor closed, until the test ends
}()
t.Cleanup(func() {
select {
case c := <-held:
c.Close()
default:
}
})
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
start := time.Now()
_, err = Ask(ctx, socket, aLineTheSocketCannotHold())
waited := time.Since(start)
if err == nil {
t.Fatal("a peer that never read nor answered gave no error")
}
if !errors.Is(err, os.ErrDeadlineExceeded) {
t.Fatalf("the error is not the deadline: %v", err)
}
if waited > 3*time.Second {
t.Fatalf("waited %s with a deadline of 1s", waited)
}
}
+191
View File
@@ -0,0 +1,191 @@
// Package health is the answer a module's health tool gives, and how the node-engine reads it (novox/hq ADR 0240
// rule 2, ADR 0283).
//
// A health tool answers one of three:
//
// - healthy: everything the module runs works;
// - unhealthy: something is wrong, said in Why;
// - waits: nothing is wrong but a part that cannot work until the operator gives something — one of the module's
// own secrets, or one of its settings. Each Wait names exactly one of them, the part, and what the operator
// gives, in words with no path, address or value.
//
// The answer keeps `healthy` beside `state`, so a node-engine older than ADR 0283 reads a wait as not healthy, as
// it did before. An answer without `state` is read from `healthy`.
//
// The module writes its answer with Healthy, Unhealthy or Waiting, and the node-engine reads it with Read. Read
// never trusts a shape it cannot check: a wait it cannot read makes the answer unhealthy, saying why.
package health
import (
"encoding/json"
"fmt"
"regexp"
"strings"
)
// The three states an answer says.
const (
StateHealthy = "healthy"
StateUnhealthy = "unhealthy"
StateWaits = "waits"
)
// Answer is a health tool's answer.
type Answer struct {
// State is healthy, unhealthy or waits.
State string `json:"state"`
// Healthy is State == healthy, kept for a reader older than State.
Healthy bool `json:"healthy"`
// Why is the answer in one or more plain sentences.
Why string `json:"why"`
// Waits is what the module waits for, when State is waits; and, beside a fault, what it also waits for.
Waits []Wait `json:"waits,omitempty"`
}
// Wait is one thing a module waits for the operator to give: exactly one of Secret and Setting.
type Wait struct {
// Part is what waits, in words: "the source games".
Part string `json:"part"`
// Secret is one of the module's own secrets, by name (a member of a secret family by its full name).
Secret string `json:"secret,omitempty"`
// Setting is one of the module's settings, by name.
Setting string `json:"setting,omitempty"`
// What is what the operator gives, in words: "the password of the source games".
What string `json:"what"`
}
// Healthy is a healthy answer.
func Healthy(why string) Answer {
return Answer{State: StateHealthy, Healthy: true, Why: why}
}
// Unhealthy is an unhealthy answer. Waits may be given beside the fault: they are said, and excuse nothing.
func Unhealthy(why string, waits ...Wait) Answer {
return Answer{State: StateUnhealthy, Why: why, Waits: waits}
}
// Waiting is an answer that says nothing is wrong but what the operator has yet to give. With no wait it is
// unhealthy: a module that waits names what for.
func Waiting(why string, waits ...Wait) Answer {
if len(waits) == 0 {
return Unhealthy(orSaid(why, "it says it waits") + " (and names nothing it waits for)")
}
for _, w := range waits {
if err := w.Check(); err != nil {
return Unhealthy(orSaid(why, "it says it waits") + " (" + err.Error() + ")")
}
}
return Answer{State: StateWaits, Why: why, Waits: waits}
}
// name is a secret's or a setting's name: what a manifest may declare.
var name = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// looksLikeAddress finds what the operator's words never carry: a URL, an IPv4 or IPv6 address, a host and port, a
// path (absolute, from a home or relative, on its own or after a bracket or a quote), a UNC path or a drive path.
// It cannot find a value typed in plain words ("the password hunter2"): a module never puts one there, and the
// words are its own, not the operator's.
var looksLikeAddress = regexp.MustCompile(strings.Join([]string{
`://`,
`\b[0-9]{1,3}(\.[0-9]{1,3}){3}\b`,
`[0-9A-Fa-f]{0,4}::[0-9A-Fa-f]{0,4}`,
`\b[0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4}){3,}\b`,
`\b[A-Za-z0-9.-]+:[0-9]{1,5}\b`,
`(^|[\s(\[{"'\x60])[~.]{0,2}/`,
`\\\\`,
`\b[A-Za-z]:\\`,
}, "|"))
// Check says why a wait cannot be read, or nil.
func (w Wait) Check() error {
switch {
case w.Secret != "" && w.Setting != "":
return fmt.Errorf("a wait names both the secret %q and the setting %q; it names one", w.Secret, w.Setting)
case w.Secret == "" && w.Setting == "":
return fmt.Errorf("a wait names neither a secret nor a setting")
case w.Secret != "" && !name.MatchString(w.Secret):
return fmt.Errorf("a wait names the secret %q, which is not a name", w.Secret)
case w.Setting != "" && !name.MatchString(w.Setting):
return fmt.Errorf("a wait names the setting %q, which is not a name", w.Setting)
case strings.TrimSpace(w.What) == "":
return fmt.Errorf("a wait for %s does not say what the operator gives", w.Names())
case strings.TrimSpace(w.Part) == "":
return fmt.Errorf("a wait for %s does not say which part waits", w.Names())
}
for _, words := range []string{w.What, w.Part} {
if strings.ContainsAny(words, "\n\r") {
return fmt.Errorf("a wait for %s says more than one line", w.Names())
}
if looksLikeAddress.MatchString(words) {
return fmt.Errorf("a wait for %s says a path or an address, which the operator's words never carry", w.Names())
}
}
return nil
}
// Names is what the wait names, as "the secret x" or "the setting y".
func (w Wait) Names() string {
if w.Secret != "" {
return "the secret " + w.Secret
}
if w.Setting != "" {
return "the setting " + w.Setting
}
return "nothing"
}
// Read reads a health tool's result. An answer it cannot read at all is an error; an answer whose wait it cannot
// read is unhealthy, saying why, so a malformed wait never excuses anything.
func Read(result []byte) (Answer, error) {
var a Answer
if err := json.Unmarshal(result, &a); err != nil {
return Answer{}, fmt.Errorf("it answered something other than healthy or not: %w", err)
}
switch a.State {
case "":
if a.Healthy {
a.State = StateHealthy
} else {
a.State = StateUnhealthy
}
case StateHealthy:
if !a.Healthy {
return Unhealthy(fmt.Sprintf("it answered the state healthy and healthy false, which contradict each other (%s)",
orSaid(a.Why, "and said nothing more"))), nil
}
// A healthy module waits for nothing: a wait beside healthy is a contradiction, and is dropped.
a.Waits = nil
case StateUnhealthy:
a.Healthy = false
case StateWaits:
a.Healthy = false
if len(a.Waits) == 0 {
return Unhealthy(orSaid(a.Why, "it says it waits") + " (and names nothing it waits for)"), nil
}
for _, w := range a.Waits {
if err := w.Check(); err != nil {
return Unhealthy(orSaid(a.Why, "it says it waits") + " (" + err.Error() + ")"), nil
}
}
default:
return Unhealthy(fmt.Sprintf("it answered the state %q, which is none of healthy, unhealthy or waits", a.State)), nil
}
if a.State != StateWaits {
// Waits beside a fault are said and excuse nothing; one that cannot be read is dropped, not believed.
var kept []Wait
for _, w := range a.Waits {
if w.Check() == nil {
kept = append(kept, w)
}
}
a.Waits = kept
}
return a, nil
}
func orSaid(s, otherwise string) string {
if strings.TrimSpace(s) == "" {
return otherwise
}
return s
}
+150
View File
@@ -0,0 +1,150 @@
package health
import (
"encoding/json"
"strings"
"testing"
)
func roundTrip(t *testing.T, a Answer) Answer {
t.Helper()
b, err := json.Marshal(a)
if err != nil {
t.Fatal(err)
}
got, err := Read(b)
if err != nil {
t.Fatal(err)
}
return got
}
var password = Wait{Part: "the source games", Secret: "smb-password-games", What: "the password of the source games"}
func TestEachConstructorSaysItsStateAndKeepsHealthyForAnOlderReader(t *testing.T) {
for _, c := range []struct {
a Answer
state string
healthy bool
}{
{Healthy("all armed"), StateHealthy, true},
{Unhealthy("the share is unreachable"), StateUnhealthy, false},
{Waiting("the source games waits for its password", password), StateWaits, false},
} {
b, _ := json.Marshal(c.a)
var old struct {
Healthy bool `json:"healthy"`
Why string `json:"why"`
}
if err := json.Unmarshal(b, &old); err != nil || old.Healthy != c.healthy || old.Why == "" {
t.Fatalf("an older reader of %s reads %+v (%v)", b, old, err)
}
if got := roundTrip(t, c.a); got.State != c.state || got.Healthy != c.healthy {
t.Fatalf("%s read as %+v", b, got)
}
}
}
func TestAWaitIsReadWithWhatItNames(t *testing.T) {
got := roundTrip(t, Waiting("waits", password, Wait{Part: "the source games", Setting: "smb-users", What: "the username of the source games"}))
if got.State != StateWaits || len(got.Waits) != 2 || got.Waits[0].Secret != "smb-password-games" || got.Waits[1].Setting != "smb-users" {
t.Fatalf("read as %+v", got)
}
}
func TestAnAnswerWithoutStateIsReadFromHealthy(t *testing.T) {
for raw, want := range map[string]string{`{"healthy":true,"why":"ok"}`: StateHealthy, `{"healthy":false,"why":"down"}`: StateUnhealthy} {
got, err := Read([]byte(raw))
if err != nil || got.State != want {
t.Fatalf("%s read as %+v, %v", raw, got, err)
}
}
}
func TestAWaitThatCannotBeReadIsUnhealthyAndSaysWhy(t *testing.T) {
for name, c := range map[string]struct {
w []Wait
says string
}{
"no wait": {nil, "names nothing"},
"neither": {[]Wait{{Part: "p", What: "w"}}, "neither a secret nor a setting"},
"both": {[]Wait{{Part: "p", Secret: "a", Setting: "b", What: "w"}}, "both"},
"not a name": {[]Wait{{Part: "p", Secret: "Smb Password", What: "w"}}, "not a name"},
"no what": {[]Wait{{Part: "p", Secret: "a"}}, "does not say what"},
"no part": {[]Wait{{Secret: "a", What: "w"}}, "which part"},
"two lines": {[]Wait{{Part: "p", Secret: "a", What: "the password\nhunter2"}}, "more than one line"},
"an address": {[]Wait{{Part: "p", Secret: "a", What: "the password of 192.0.2.76"}}, "path or an address"},
"a url": {[]Wait{{Part: "p", Secret: "a", What: "the password of smb://host/share"}}, "path or an address"},
"a path": {[]Wait{{Part: "the source at /mnt/games", Secret: "a", What: "the password"}}, "path or an address"},
} {
b, _ := json.Marshal(Answer{State: StateWaits, Why: "it waits", Waits: c.w})
got, err := Read(b)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if got.State != StateUnhealthy || got.Healthy || !strings.Contains(got.Why, c.says) {
t.Fatalf("%s: read as %+v, want unhealthy saying %q", name, got, c.says)
}
}
}
func TestAnUnknownStateIsUnhealthy(t *testing.T) {
got, err := Read([]byte(`{"state":"fine","healthy":true}`))
if err != nil || got.State != StateUnhealthy || got.Healthy {
t.Fatalf("read as %+v, %v", got, err)
}
}
func TestWaitsBesideAFaultAreSaidAndExcuseNothing(t *testing.T) {
got := roundTrip(t, Unhealthy("the share is unreachable", password, Wait{Part: "p", What: "w"}))
if got.State != StateUnhealthy || len(got.Waits) != 1 {
t.Fatalf("read as %+v: the readable wait is kept, the other dropped, the state stays unhealthy", got)
}
if h := roundTrip(t, Answer{State: StateHealthy, Healthy: true, Why: "ok", Waits: []Wait{password}}); len(h.Waits) != 0 {
t.Fatalf("a healthy answer kept its waits: %+v", h)
}
}
func TestWaitingWithNothingIsUnhealthy(t *testing.T) {
if a := Waiting("it waits"); a.State != StateUnhealthy {
t.Fatalf("Waiting with no wait is %+v", a)
}
}
func TestNotAnAnswerIsAnError(t *testing.T) {
if _, err := Read([]byte(`"yes"`)); err == nil {
t.Fatal("a string read as an answer")
}
}
func TestEveryFormOfPathOrAddressIsRefused(t *testing.T) {
for _, words := range []string{
"the source (/mnt/games)", `the source "/mnt/games"`, "the folder ~/games", "the folder ./games",
"2001:db8::1", "fe80::1%eth0", "2001:0db8:0000:0000:0000:0000:0000:0001", "nas.lan:445", "192.0.2.7:445",
`\\nas\games`, `C:\Users\x`, "smb://nas/games", "198.51.100.4",
} {
w := Wait{Part: "the source games", Secret: "smb-password-games", What: "the password of " + words}
if err := w.Check(); err == nil {
t.Errorf("%q passed as words for the operator", words)
}
}
for _, words := range []string{"the password of the source games", "the username of the source games-2", "the API key, issued by the vendor"} {
w := Wait{Part: "the source games", Secret: "smb-password-games", What: words}
if err := w.Check(); err != nil {
t.Errorf("%q refused: %v", words, err)
}
}
}
func TestHealthyStateWithHealthyFalseIsUnhealthy(t *testing.T) {
got, err := Read([]byte(`{"state":"healthy","healthy":false,"why":"x"}`))
if err != nil || got.State != StateUnhealthy || !strings.Contains(got.Why, "contradict") {
t.Fatalf("read as %+v, %v", got, err)
}
}
func TestWaitingChecksItsWaits(t *testing.T) {
if a := Waiting("it waits", Wait{Part: "p", What: "w"}); a.State != StateUnhealthy || !strings.Contains(a.Why, "neither") {
t.Fatalf("Waiting with a malformed wait is %+v", a)
}
}
+123
View File
@@ -0,0 +1,123 @@
package stdio
import (
"encoding/json"
"fmt"
"time"
)
// Log — a module's record of operations on the bus, kept as long as the log is (novox/hq ADR 0297). Not the
// current value: that is state, and a log's state is its fold. Each entry is one operation on one key, its
// sequence in the log is its identity, and the order of the log is the order of the operations. A module
// declares each log it owns in its manifest (`logs`) and names it here as it named it there; a log is its
// owner's alone, and the runtime refuses, with the reason, one it did not issue.
// LogEntry is one operation on one key, as the log holds it.
type LogEntry struct {
Sequence uint64 `json:"sequence"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
At time.Time `json:"at"`
}
// LogPage is one page of a read: its entries, in order, the sequence to read on from (0 when nothing
// follows) and the log's last sequence when it was read.
type LogPage struct {
Entries []LogEntry `json:"entries"`
Next uint64 `json:"next"`
Last uint64 `json:"last"`
}
// LogInfo is how much a log holds: its entries and bytes, its cap in bytes, and its first and last sequence.
type LogInfo struct {
Entries uint64 `json:"entries"`
Bytes uint64 `json:"bytes"`
MaxBytes uint64 `json:"max_bytes"`
First uint64 `json:"first"`
Last uint64 `json:"last"`
}
// KeptLog is one log this module owns.
type KeptLog struct{ name string }
// Log is a log this module owns, by its local name.
func Log(name string) KeptLog { return KeptLog{name: name} }
const logTimeout = 30 * time.Second
func (l KeptLog) ask(verb string, params map[string]any) (json.RawMessage, error) {
c, err := live()
if err != nil {
return nil, err
}
params["log"] = l.name
return c.ask("mesh/log."+verb, params, logTimeout)
}
func (l KeptLog) append(params map[string]any) (uint64, error) {
raw, err := l.ask("append", params)
if err != nil {
return 0, err
}
var appended struct {
Sequence uint64 `json:"sequence"`
}
return appended.Sequence, json.Unmarshal(raw, &appended)
}
// Append writes an entry for key, whatever came before it, and returns its sequence. A key is one or more
// names of letters, digits, `-` and `_`, joined by `.`; a value is JSON of at most 256 KiB. A full log
// refuses the entry: it never drops an older one to make room.
func (l KeptLog) Append(key string, value any) (uint64, error) {
return l.append(map[string]any{"key": key, "value": value})
}
// AppendAt writes an entry for key only if the key's last entry is still at expect — the sequence the caller
// read, 0 for a key with no entry yet — and returns its sequence. When another entry for the key came in
// between, nothing is written and the error is ErrChanged, as for a lost State update: read again and
// decide again.
func (l KeptLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
return l.append(map[string]any{"key": key, "value": value, "expect": expect})
}
// Read is the entries from the sequence from on, oldest first, at most limit of them (0 is the runtime's
// page, 100; it gives at most 1000). keys narrows it: a key, with `*` for one name and `>` for the rest;
// empty is every key. Read on from the page's Next until it is 0.
func (l KeptLog) Read(from uint64, keys string, limit int) (*LogPage, error) {
if limit < 0 {
return nil, fmt.Errorf("a read of %s asks for at least no entries: limit %d", l.name, limit)
}
params := map[string]any{"from": from}
if keys != "" {
params["keys"] = keys
}
if limit > 0 {
params["limit"] = limit
}
raw, err := l.ask("read", params)
if err != nil {
return nil, err
}
var p LogPage
return &p, json.Unmarshal(raw, &p)
}
// Last is a key's last entry, or nil when the key has none.
func (l KeptLog) Last(key string) (*LogEntry, error) {
raw, err := l.ask("last", map[string]any{"key": key})
if err != nil || len(raw) == 0 || string(raw) == "null" {
return nil, err
}
var e LogEntry
return &e, json.Unmarshal(raw, &e)
}
// Info is how much the log holds and how much it may.
func (l KeptLog) Info() (*LogInfo, error) {
raw, err := l.ask("info", map[string]any{})
if err != nil {
return nil, err
}
var i LogInfo
return &i, json.Unmarshal(raw, &i)
}
+354
View File
@@ -0,0 +1,354 @@
package stdio
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"sync"
"testing"
"time"
)
// logRuntime is the runtime's side of mesh/log.append, read, last and info as the bus does them (novox/hq
// ADR 0297): one sequence per entry across the log, an append with expect refused unless the key's last
// entry is still at it (0 for a key with none), that refusal answered with CodeChanged as a lost state
// update is, and a read paged by its limit with the sequence to read on from. asked keeps every ask's
// method and params, as the bundle sent them.
type logRuntime struct {
mu sync.Mutex
entries []LogEntry
asked []message
}
func (r *logRuntime) last(key string) uint64 {
for i := len(r.entries) - 1; i >= 0; i-- {
if r.entries[i].Key == key {
return r.entries[i].Sequence
}
}
return 0
}
// matches is a key pattern as the runtime reads it: `*` one name, `>` the rest.
func matches(pattern, key string) bool {
if pattern == "" {
return true
}
p, k := strings.Split(pattern, "."), strings.Split(key, ".")
for i, t := range p {
if t == ">" {
return len(k) > i
}
if i >= len(k) || (t != "*" && t != k[i]) {
return false
}
}
return len(p) == len(k)
}
func (r *logRuntime) answer(m message) map[string]any {
var p struct {
Log string `json:"log"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Expect *uint64 `json:"expect"`
From uint64 `json:"from"`
Keys string `json:"keys"`
Limit int `json:"limit"`
}
_ = json.Unmarshal(m.Params, &p)
r.mu.Lock()
defer r.mu.Unlock()
r.asked = append(r.asked, m)
if p.Log != "changes" {
return map[string]any{"error": map[string]any{"code": -32000, "message": "issues was issued no log " + p.Log}}
}
switch strings.TrimPrefix(m.Method, "mesh/log.") {
case "append":
if p.Expect != nil && r.last(p.Key) != *p.Expect {
return map[string]any{"error": map[string]any{"code": CodeChanged, "message": p.Key + ": the key was written by another writer first"}}
}
seq := uint64(len(r.entries)) + 1
r.entries = append(r.entries, LogEntry{Sequence: seq, Key: p.Key, Value: p.Value,
At: time.Date(2026, 10, 11, 18, 0, int(seq), 0, time.UTC)})
return map[string]any{"result": map[string]any{"sequence": seq}}
case "read":
limit := p.Limit
if limit == 0 {
limit = 100
}
page := map[string]any{"entries": []LogEntry{}, "next": 0, "last": len(r.entries)}
var got []LogEntry
for _, e := range r.entries {
if e.Sequence < p.From || !matches(p.Keys, e.Key) {
continue
}
if len(got) == limit {
page["next"] = e.Sequence
break
}
got = append(got, e)
}
if got != nil {
page["entries"] = got
}
return map[string]any{"result": page}
case "last":
seq := r.last(p.Key)
if seq == 0 {
return map[string]any{"result": nil}
}
return map[string]any{"result": r.entries[seq-1]}
case "info":
var bytes int
for _, e := range r.entries {
bytes += len(e.Value)
}
info := map[string]any{"entries": len(r.entries), "bytes": bytes, "max_bytes": 1 << 30, "first": 0, "last": len(r.entries)}
if len(r.entries) > 0 {
info["first"] = 1
}
return map[string]any{"result": info}
}
return map[string]any{"error": map[string]any{"code": -32601, "message": "no " + m.Method}}
}
// logging serves a bundle whose one tool, "log", runs run against r, answering each mesh/log ask beside
// the loop as the runtime does. It returns what the tool answered.
func logging(t *testing.T, r *logRuntime, run func() (any, error)) map[string]any {
t.Helper()
t.Setenv(ServedAs, "issues")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() {
_ = ServeOn("", []Tool{{Name: "log", Run: func(map[string]any) (any, error) { return run() }}}, inR, outW)
}()
var wmu sync.Mutex
send := func(m map[string]any) {
m["jsonrpc"] = "2.0"
b, _ := json.Marshal(m)
wmu.Lock()
defer wmu.Unlock()
_, _ = inW.Write(append(b, '\n'))
}
done := make(chan map[string]any, 1)
go func() {
lines := bufio.NewScanner(outR)
lines.Buffer(make([]byte, 1<<20), 1<<24)
for lines.Scan() {
var m message
if json.Unmarshal(lines.Bytes(), &m) != nil {
continue
}
if strings.HasPrefix(m.Method, "mesh/log.") {
go func(m message) {
a := r.answer(m)
a["id"] = m.ID
send(a)
}(m)
continue
}
var answer map[string]any
_ = json.Unmarshal(lines.Bytes(), &answer)
if answer["id"] == float64(1) {
done <- answer
return
}
}
}()
send(map[string]any{"id": 1, "method": "tools/call", "params": map[string]any{"name": "log"}})
t.Cleanup(func() { _ = inW.Close() })
select {
case a := <-done:
return a
case <-time.After(10 * time.Second):
t.Fatal("the bundle did not answer")
return nil
}
}
// params is what the bundle sent with its n-th ask, and the ask's method.
func (r *logRuntime) params(t *testing.T, n int) (string, map[string]any) {
t.Helper()
r.mu.Lock()
defer r.mu.Unlock()
if n >= len(r.asked) {
t.Fatalf("the bundle asked %d times, not %d", len(r.asked), n+1)
}
var p map[string]any
_ = json.Unmarshal(r.asked[n].Params, &p)
return r.asked[n].Method, p
}
// A bundle appends to, reads, and measures its log through the runtime (novox/hq ADR 0297): each verb is
// asked as mesh/log.<verb> naming the log, its answer is handed back whole, a key with no entry is nil,
// and a read pages on from Next until it is 0.
func TestABundleKeepsALogThroughTheRuntime(t *testing.T) {
r := &logRuntime{}
var (
first, second, third uint64
pages []*LogPage
last, none *LogEntry
info *LogInfo
narrowed *LogPage
)
got := logging(t, r, func() (any, error) {
changes := Log("changes")
var err error
if first, err = changes.Append("401", map[string]any{"op": "opened", "by": "fix@g14"}); err != nil {
return nil, err
}
if second, err = changes.Append("402", map[string]any{"op": "opened"}); err != nil {
return nil, err
}
if third, err = changes.AppendAt("401", map[string]any{"op": "moved", "to": "fixing"}, first); err != nil {
return nil, err
}
for _, k := range []string{"403", "404", "405"} {
if _, err = changes.Append(k, map[string]any{"op": "opened"}); err != nil {
return nil, err
}
}
for from := uint64(1); ; {
p, err := changes.Read(from, "", 2)
if err != nil {
return nil, err
}
pages = append(pages, p)
if p.Next == 0 {
break
}
from = p.Next
}
if narrowed, err = changes.Read(0, "401", 0); err != nil {
return nil, err
}
if last, err = changes.Last("401"); err != nil {
return nil, err
}
if none, err = changes.Last("999"); err != nil {
return nil, err
}
info, err = changes.Info()
return "done", err
})
if out, isError := text(t, got); isError || out != `"done"` {
t.Fatalf("the tool answered %s", out)
}
if first != 1 || second != 2 || third != 3 {
t.Fatalf("appended at %d, %d, %d", first, second, third)
}
// Asked as the contract names them: the verb, the log, and only what was given.
method, p := r.params(t, 0)
if method != "mesh/log.append" || p["log"] != "changes" || p["key"] != "401" || p["value"].(map[string]any)["by"] != "fix@g14" {
t.Fatalf("an append asked %s %v", method, p)
}
if _, has := p["expect"]; has {
t.Fatalf("an Append without expect named one: %v", p)
}
if method, p = r.params(t, 2); method != "mesh/log.append" || p["expect"] != float64(1) {
t.Fatalf("an AppendAt asked %s %v", method, p)
}
if method, p = r.params(t, 6); method != "mesh/log.read" || p["from"] != float64(1) || p["limit"] != float64(2) || p["keys"] != nil {
t.Fatalf("a read asked %s %v", method, p)
}
// Paged two at a time through six entries: three pages, every entry once, in order.
if len(pages) != 3 {
t.Fatalf("read %d pages, not 3", len(pages))
}
var seqs []uint64
for _, pg := range pages {
if pg.Last != 6 {
t.Fatalf("a page said the last sequence is %d", pg.Last)
}
for _, e := range pg.Entries {
seqs = append(seqs, e.Sequence)
}
}
if fmt.Sprint(seqs) != "[1 2 3 4 5 6]" || pages[0].Next != 3 || pages[2].Next != 0 {
t.Fatalf("paged %v, next %d then %d", seqs, pages[0].Next, pages[2].Next)
}
if e := pages[1].Entries[0]; e.Key != "401" || string(e.Value) != `{"op":"moved","to":"fixing"}` ||
!e.At.Equal(time.Date(2026, 10, 11, 18, 0, 3, 0, time.UTC)) {
t.Fatalf("an entry read as %+v", e)
}
// Narrowed to a key, with the runtime's page: no limit sent.
method, p = r.params(t, 9)
if method != "mesh/log.read" || p["keys"] != "401" || p["limit"] != nil {
t.Fatalf("a narrowed read asked %s %v", method, p)
}
if len(narrowed.Entries) != 2 || narrowed.Entries[0].Sequence != 1 || narrowed.Entries[1].Sequence != 3 || narrowed.Next != 0 {
t.Fatalf("narrowed to 401: %+v", narrowed)
}
if last == nil || last.Sequence != 3 || last.Key != "401" {
t.Fatalf("the last of 401 is %+v", last)
}
if method, p = r.params(t, 10); method != "mesh/log.last" || p["key"] != "401" {
t.Fatalf("a last asked %s %v", method, p)
}
if none != nil {
t.Fatalf("a key with no entry has a last: %+v", none)
}
if info == nil || info.Entries != 6 || info.MaxBytes != 1<<30 || info.First != 1 || info.Last != 6 || info.Bytes == 0 {
t.Fatalf("the log's info is %+v", info)
}
if method, p = r.params(t, 12); method != "mesh/log.info" || p["log"] != "changes" || len(p) != 1 {
t.Fatalf("an info asked %s %v", method, p)
}
}
// An AppendAt on a stale expect is ErrChanged and writes nothing: another entry for the key came in
// between. Expect 0 is a key with no entry yet, and is ErrChanged once the key has one. A log not issued
// is refused with the runtime's reason, and is not ErrChanged.
func TestAnAppendAtAStaleSequenceIsErrChangedAndWritesNothing(t *testing.T) {
r := &logRuntime{}
var errs []error
got := logging(t, r, func() (any, error) {
changes := Log("changes")
read, err := changes.AppendAt("401", map[string]any{"op": "opened"}, 0)
if err != nil {
return nil, err
}
// Another writer is first.
if _, err := changes.Append("401", map[string]any{"op": "noted"}); err != nil {
return nil, err
}
_, stale := changes.AppendAt("401", map[string]any{"op": "moved"}, read)
_, again := changes.AppendAt("401", map[string]any{"op": "opened"}, 0)
_, other := Log("elsewhere").Append("401", map[string]any{})
errs = []error{stale, again, other}
return "done", nil
})
if out, isError := text(t, got); isError || out != `"done"` {
t.Fatalf("the tool answered %s", out)
}
if !errors.Is(errs[0], ErrChanged) || !errors.Is(errs[1], ErrChanged) {
t.Fatalf("a stale expect answered %v and %v, not ErrChanged", errs[0], errs[1])
}
if errs[2] == nil || errors.Is(errs[2], ErrChanged) || !strings.Contains(errs[2].Error(), "no log elsewhere") {
t.Fatalf("a log not issued answered %v", errs[2])
}
if len(r.entries) != 2 {
t.Fatalf("the log holds %d entries, not 2: a refused append wrote", len(r.entries))
}
}
// A read asking for fewer than no entries is refused before the runtime is asked.
func TestAReadOfANegativeLimitIsRefused(t *testing.T) {
r := &logRuntime{}
var err error
logging(t, r, func() (any, error) {
_, err = Log("changes").Read(0, "", -1)
return "done", nil
})
if err == nil || len(r.asked) != 0 {
t.Fatalf("a negative limit answered %v after %d asks", err, len(r.asked))
}
}
+276
View File
@@ -0,0 +1,276 @@
// Package outward is the content rule (novox/hq ADR 0310 §4, to-be 45 §5, research 028 Q8, ADR 0227): what
// may leave the mesh, and what an answer may carry at all. One rule, here, so every module that answers outward
// applies the same one rather than a copy: the operator channel's router and every node-reads answer.
//
// Two strengths:
//
// - Check is the operator channel's: roles and words only. A message carrying an address, a path or
// anything shaped like a secret is refused, because a channel such as Telegram is not end-to-end
// encrypted for bots and this rule is the only thing between a condition's words and someone else's
// server. Deliberately wider than it must be: a commit id or a long random name is refused too.
// - CheckSecret is the secret class alone: what leaves the mesh nowhere and is answered nowhere. A path
// or an address is what a file read or an operator's approval is about; a secret's shape never is.
//
// A refusal says its class and a few words, never the text itself.
package outward
import (
"math"
"regexp"
"strings"
"unicode"
)
// Refusal says why a text may not leave: the class of what it carried, never the text itself.
type Refusal struct {
Class string // address, path or secret
What string // a few words: "an IPv4 address", "a URL", …
}
func (r Refusal) String() string { return r.Class + " (" + r.What + ")" }
var (
reURL = regexp.MustCompile(`(?i)\b[a-z][a-z0-9+.-]*://`)
reEmail = regexp.MustCompile(`[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,}`)
reIPv4 = regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}\b`)
reIPv6 = regexp.MustCompile(`(?i)(^|[^0-9a-z:])(([0-9a-f]{1,4}:){4,7}[0-9a-f]{1,4}|([0-9a-f]{1,4}:)*[0-9a-f]{0,4}::([0-9a-f]{1,4}:)*[0-9a-f]{0,4})([^0-9a-z:]|$)`)
reMAC = regexp.MustCompile(`(?i)\b([0-9a-f]{2}[:-]){5}[0-9a-f]{2}\b`)
rePEM = regexp.MustCompile(`-----BEGIN [A-Z ]+-----`)
reJWT = regexp.MustCompile(`\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}`)
reBotToken = regexp.MustCompile(`\b\d{6,}:[A-Za-z0-9_-]{30,}`)
reKnown = regexp.MustCompile(`\b(gh[pousr]_[A-Za-z0-9]{20,}|glpat-[A-Za-z0-9_-]{16,}|sk-[A-Za-z0-9_-]{16,}|xox[abprs]-[A-Za-z0-9-]{10,}|AKIA[0-9A-Z]{16})`)
reAssigned = regexp.MustCompile(`(?i)\b(password|passwd|passphrase|secret|token|api[_-]?key|apikey|credential|private[_-]?key)\s*[=:]\s*\S`)
// reNamedSecret is a value given to a name that ends in a secret's word after `_` or `-`, as an environment
// or configuration name is written: MESH_TOKEN=, DB_PASS=, aws_secret_access_key = …, SUDO_PASSWORD value.
// The word ends the name: what follows it is `=`, `:` or a space and then a value.
reNamedSecret = regexp.MustCompile(`(?i)(^|[^a-z0-9_-])[a-z0-9]+[_-]([a-z0-9]+[_-])*(password|passwd|passphrase|pass|pwd|secret|token|api[_-]?key|apikey|access[_-]?key|secret[_-]?key|private[_-]?key|credentials?|psk)(\s*[=:]\s*|\s+)[^\s=:]`)
// reURLCredential is a password inside an address: scheme://user:password@host.
reURLCredential = regexp.MustCompile(`(?i)\b[a-z][a-z0-9+.-]*://[^/\s:@]+:[^/\s@]+@`)
// reSecretRun is a run CheckSecret judges: reRun with `-`, so a base64url value is judged whole.
reSecretRun = regexp.MustCompile(`[A-Za-z0-9+/=_-]{20,}`)
reUUID = regexp.MustCompile(`^(?i)[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
reHex = regexp.MustCompile(`(?i)\b[0-9a-f]{32,}\b`)
reRun = regexp.MustCompile(`[A-Za-z0-9+/=_]{20,}`)
reWinPath = regexp.MustCompile(`(?i)\b[a-z]:\\`)
)
// topLevel are names that end a host name — the generic and country ones a mesh's names use, and the
// private ones (.internal, .lan, .home, .local) a mesh is likelier to.
var topLevel = map[string]bool{}
func init() {
for _, t := range strings.Fields(`com net org edu gov mil int io dev app cloud ai co me info biz xyz
site online tech page link
be nl de fr uk lu eu ch at it es pt se no dk fi pl cz us ca au nz jp cn ru in br ie
internal lan home local localdomain corp intranet private arpa test example invalid localhost`) {
topLevel[t] = true
}
}
// Check says whether a text may leave the mesh, and when not, why.
func Check(text string) (Refusal, bool) {
switch {
case reURL.MatchString(text):
return Refusal{"address", "a URL"}, false
case reEmail.MatchString(text):
return Refusal{"address", "a mail address"}, false
case reIPv4.MatchString(text):
return Refusal{"address", "an IPv4 address"}, false
case reMAC.MatchString(text):
return Refusal{"address", "a hardware address"}, false
case reIPv6.MatchString(text):
return Refusal{"address", "an IPv6 address"}, false
}
if refusal, ok := secretShape(text); !ok {
return refusal, false
}
if reWinPath.MatchString(text) {
return Refusal{"path", "a drive path"}, false
}
if refusal, ok := randomRun(text); !ok {
return refusal, false
}
for _, word := range strings.FieldsFunc(text, func(r rune) bool {
return unicode.IsSpace(r) || strings.ContainsRune("\"'`()[]{}<>,;|", r)
}) {
w := strings.TrimRight(word, ".:!?")
if isPath(w) {
return Refusal{"path", "a file path"}, false
}
if isHostName(w) {
return Refusal{"address", "a host name"}, false
}
}
return Refusal{}, true
}
// CheckSecret says whether a text carries a secret's shape — the one class that leaves the mesh nowhere. It is
// what an ask's whole words (asks.Ask.Whole, novox/hq issue 383) are held to: on a kind that proves who answers,
// a path or an address is what the operator approves and may be shown, a secret never. A path is read as one:
// a run with a slash in it is judged piece by piece between the slashes, so "/mnt/Photos_2024/Jochen" is a
// path and not a random string (the review of 2026-10-10); the named shapes — a key block, a signed token, a
// bot token, a known token shape, a value given to a secret's name, a long hexadecimal string — hold whatever
// the run holds.
func CheckSecret(text string) (Refusal, bool) {
if refusal, ok := secretShape(text); !ok {
return refusal, false
}
return randomRunOutsidePaths(text)
}
// randomRunOutsidePaths judges every long run of a value's characters, a path's pieces between its slashes
// alone (the review of mesh-sdk #23): a run is read as a path only where it is one — it starts with a slash,
// follows a dot or a tilde (a host's or a home's path), or a name's `=` gives it one — so a base64 value, whose
// slashes fall anywhere, is judged whole. A UUID is a name, never a secret.
func randomRunOutsidePaths(text string) (Refusal, bool) {
for _, loc := range reSecretRun.FindAllStringIndex(text, -1) {
var before byte
if loc[0] > 0 {
before = text[loc[0]-1]
}
run := text[loc[0]:loc[1]]
// Standard base64 ends in its padding: such a run is a value, even when it starts with a slash.
padded := strings.HasSuffix(run, "=") && !strings.Contains(strings.TrimRight(run, "="), "=")
for i, seg := range strings.Split(run, "=") {
path := !padded && (strings.HasPrefix(seg, "/") || (i == 0 && (before == '.' || before == '~')))
pieces := []string{seg}
if path {
pieces = strings.Split(seg, "/")
}
for _, piece := range pieces {
if len(piece) >= 20 && !reUUID.MatchString(piece) && looksRandomValue(piece) {
return Refusal{"secret", "a long random-looking string"}, false
}
}
}
}
return Refusal{}, true
}
// secretShape is the secret shapes a pattern names.
func secretShape(text string) (Refusal, bool) {
switch {
case rePEM.MatchString(text):
return Refusal{"secret", "a key block"}, false
case reJWT.MatchString(text):
return Refusal{"secret", "a signed token"}, false
case reBotToken.MatchString(text):
return Refusal{"secret", "a bot token"}, false
case reKnown.MatchString(text):
return Refusal{"secret", "a known token shape"}, false
case reURLCredential.MatchString(text):
return Refusal{"secret", "a password in an address"}, false
case reAssigned.MatchString(text), reNamedSecret.MatchString(text):
return Refusal{"secret", "a value given to a secret's name"}, false
case reHex.MatchString(text):
return Refusal{"secret", "a long hexadecimal string"}, false
}
return Refusal{}, true
}
// randomRun is a long unbroken run of characters spread as a random string's are.
func randomRun(text string) (Refusal, bool) {
for _, run := range reRun.FindAllString(text, -1) {
if looksRandom(run) {
return Refusal{"secret", "a long random-looking string"}, false
}
}
return Refusal{}, true
}
// isPath: absolute, home-relative or dot-relative, or two separators deep. A mesh address names one
// machine and one tool (`ace/postgres.query`) and has one; a ratio ("3/4") has digits only.
func isPath(w string) bool {
if w == "" {
return false
}
if strings.HasPrefix(w, "/") && len(w) > 1 {
return true
}
for _, p := range []string{"~/", "./", "../", "$HOME", "${"} {
if strings.HasPrefix(w, p) {
return true
}
}
return strings.Count(w, "/") >= 2 || strings.Contains(w, "\\")
}
// isHostName: two names or more, the last a top-level one. A condition key's last name is its kind
// (`machine.ace.silent`), which none of these is.
func isHostName(w string) bool {
w = strings.ToLower(w)
if w == "localhost" {
return true
}
parts := strings.Split(w, ".")
if len(parts) < 2 {
return false
}
for _, p := range parts {
if p == "" {
return false
}
}
return topLevel[parts[len(parts)-1]]
}
// looksRandom: letters and digits mixed, and the characters spread as a random string's are. A
// sentence's words are separated, so only an unbroken run reaches here.
func looksRandom(s string) bool {
var letters, digits int
counts := map[rune]int{}
for _, r := range s {
counts[r]++
switch {
case unicode.IsLetter(r):
letters++
case unicode.IsDigit(r):
digits++
}
}
if letters == 0 || digits == 0 {
// One class only: a word, or a number. A long number of digits alone is a count or a time.
return letters > 0 && hasUpperAndLower(s) && entropy(counts, len(s)) >= 4.0
}
return entropy(counts, len(s)) >= 3.3
}
// looksRandomValue is looksRandom for a value in a file: letters and digits mixed and spread as a random string's
// are; letters alone when the case changes as a random string's does — about as many capitals as small letters,
// where a CamelCase name (sshd's PasswordAuthentication) has one a word.
func looksRandomValue(s string) bool {
var upper, lower, digits int
counts := map[rune]int{}
for _, r := range s {
counts[r]++
switch {
case unicode.IsUpper(r):
upper++
case unicode.IsLower(r):
lower++
case unicode.IsDigit(r):
digits++
}
}
if upper+lower == 0 {
return false
}
if digits > 0 {
return entropy(counts, len(s)) >= 3.3
}
share := float64(upper) / float64(upper+lower)
return share >= 0.3 && share <= 0.7 && entropy(counts, len(s)) >= 3.5
}
func hasUpperAndLower(s string) bool {
return strings.IndexFunc(s, unicode.IsUpper) >= 0 && strings.IndexFunc(s, unicode.IsLower) >= 0
}
func entropy(counts map[rune]int, n int) float64 {
e := 0.0
for _, c := range counts {
p := float64(c) / float64(n)
e -= p * math.Log2(p)
}
return e
}
+164
View File
@@ -0,0 +1,164 @@
package outward
// The operator channel's cases, carried here with the rule (novox/hq ADR 0310 §4): the rule moved, its meaning did not.
import (
"encoding/base64"
"fmt"
"math/rand"
"testing"
)
func TestRolesAndWordsMayLeave(t *testing.T) {
for _, s := range []string{
"URGENT: the home server has not been heard for 15 min",
"key: provider.keycloak.ace.gitea.failing",
"more: conditions show machine.ace.silent",
"since: 2026-10-06 12:30 UTC",
"call call-1759752000123456789-12 has run past its bound of 10 min",
"ace/postgres.query answers no more",
"3/4 machines answered; plan 41 waits on build 7f3a9c1e",
"the bus advisory slow-consumer for ace_records",
"core.controller.novox.rolled-back",
"seat node-notifier has no live holder",
"HELD BACK: 5 message(s) over the cap of 20 an hour",
"STILL OPEN after 1.5 h: the controller's event loop takes no message",
"node-engine and node tools builds differ from the plan's",
} {
if r, ok := Check(s); !ok {
t.Errorf("refused %q: %s", s, r)
}
}
}
func TestAnAddressAPathOrASecretMayNot(t *testing.T) {
for s, class := range map[string]string{
"cannot reach 192.168.1.20": "address",
"listening on 10.0.0.7:5432": "address",
"route fd00:1234:5678::1 is gone": "address",
"fe80::1 answered": "address",
"2001:db8:0:0:0:0:2:1 answered": "address",
"see https://git.example.org/x": "address",
"git.novox.internal does not answer": "address",
"the mail for admin@example.org bounced": "address",
"zurag.be is down": "address",
"localhost refused": "address",
"the card aa:bb:cc:dd:ee:ff went away": "address",
"/var/lib/mesh-controller is full": "path",
"~/.config/hal/env changed": "path",
"read ./grants.json": "path",
"services/postgres/data/pg_hba.conf": "path",
"C:\\Users\\x": "path",
"token 123456789:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw": "secret",
"ghp_abcdefghijklmnopqrstuvwxyz0123456789": "secret",
"password=hunter2": "secret",
"api_key: x": "secret",
"-----BEGIN OPENSSH PRIVATE KEY-----": "secret",
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0": "secret",
"commit 3f2a9c1e5b7d4f6a8c0e2b4d6f8a0c2e4b6d8f0a": "secret",
"value Zx8Qp2Lm9Rt4Vb7Nc1Kd5Hs3Jf6": "secret",
} {
r, ok := Check(s)
if ok {
t.Errorf("let %q leave", s)
continue
}
if r.Class != class {
t.Errorf("%q refused as %s, want %s", s, r, class)
}
}
}
// The secret class alone (novox/hq ADR 0310 §4, issue 383): a path and an address pass, a path's pieces are
// judged between its slashes, and every named secret shape and a random run outside a path are refused.
func TestCheckSecretRefusesASecretsShapeAndNothingElse(t *testing.T) {
for text, may := range map[string]bool{
"+ shares: photos=/mnt/Photos_2024/Jochen": true,
"+ sources: games=smb://nas.lan/Recalbox_Games2024": true,
"+ library: /srv/media/Series_Archive/Season01": true,
"root:x:0:0::/root:/usr/bin/bash": true,
"nameserver 192.168.1.1": true,
"drwxr-x--- jochen users /home/jochen": true,
"+ key: xK9mQ2vL8pR4tW7yB3nF6hJ1dG5sA0zC": false,
"-----BEGIN OPENSSH PRIVATE KEY-----": false,
"password=hunter2": false,
"Authorization: token ghp_abcdefghijklmnopqrstuvwxyz0123456789": false,
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0": false,
"sha 3f2a9c1e5b7d4f6a8c0e2b4d6f8a0c2e4b6d8f0a": false,
} {
r, ok := CheckSecret(text)
if ok != may {
t.Errorf("CheckSecret(%q) = %v (%s), want %v", text, ok, r, may)
}
if !ok && r.Class != "secret" {
t.Errorf("CheckSecret(%q) refused as %s: only the secret class is refused there", text, r)
}
}
}
// The review of mesh-sdk #23: the shapes node-reads meets in env files, unit files, configurations and
// history files, each refused by its shape whatever the value looks like.
func TestCheckSecretRefusesTheShapesOfConfigurationFiles(t *testing.T) {
for _, text := range []string{
"url = nats://user:pa55word@host:4222",
"DATABASE_URL=postgres://app:hunter2@db/app",
"MESH_TOKEN=s3cr3t-v4lue",
"DB_PASS=hunter2",
"export GITEA_SECRET=abc",
"Environment=APP_API_KEY=xyz",
"SUDO_PASSWORD hunter2",
"aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"password = hunter2",
"wifi_psk: correcthorse",
"value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
} {
if r, ok := CheckSecret(text); ok || r.Class != "secret" {
t.Errorf("CheckSecret(%q) let it through (%v)", text, r)
}
}
// What a configuration file says that is not a secret.
for _, text := range []string{
"ExecStart=/usr/lib/systemd/systemd-resolved",
"UUID=3f2a9c1e-5b7d-4f6a-8c0e-2b4d6f8a0c2e /home btrfs defaults 0 0",
"password required pam_unix.so",
"bypass = true",
"PasswordAuthentication no",
"url = https://git.example.org/novox/mesh-sdk.git",
"HOME=/var/lib/mesh-reads",
"TOKEN_FILE=/var/lib/app/token",
} {
if r, ok := CheckSecret(text); !ok {
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
}
}
}
// Random values as base64 and base64url, wherever their slashes fall: every one is refused.
func TestCheckSecretRefusesRandomValuesAsBase64(t *testing.T) {
rng := rand.New(rand.NewSource(418))
passed := map[string]int{}
for i := 0; i < 1000; i++ {
for _, n := range []int{16, 32} {
b := make([]byte, n)
rng.Read(b)
for name, enc := range map[string]*base64.Encoding{"std": base64.StdEncoding, "url": base64.RawURLEncoding} {
v := enc.EncodeToString(b)
if _, ok := CheckSecret("key " + v); ok {
passed[fmt.Sprintf("%s/%d", name, n)]++
}
}
}
}
if len(passed) > 0 {
t.Errorf("random values passed as clean, of 1000 each: %v", passed)
}
}
// A CamelCase name is a word, not a value: the case changes once a word.
func TestCheckSecretLetsACamelCaseNameThrough(t *testing.T) {
for _, text := range []string{"PasswordAuthenticationIsOff", "ChallengeResponseAuthentication no", "KbdInteractiveAuthentication"} {
if r, ok := CheckSecret(text); !ok {
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
}
}
}
+72
View File
@@ -0,0 +1,72 @@
package provider
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
)
// The provider's consumers' credentials, from the file its machine's provisioner client fills (novox/hq ADR 0373,
// issue 679).
//
// **The provider never asks the vault.** A provider in the tool runner has no account of its own, and the vault
// hands a secret only to a node's provisioner client or to a module under an account of its own (ADR 0349 clause
// 11, ADR 0370 clause 2). So the provider's template names `${consumers:<provision>}`, and its machine's client
// fills that file with one entry per consumer bound to it for that provision, each credential asked of the vault in
// the client's own name; the file is the provider's alone (mode 0600, owned). ConsumersFile is that file as the
// helper's Vault: it reads the file on each ask, so a consumer that binds or a credential the vault remakes reaches
// the provider at its next pass.
// ConsumerCredential is one entry of the file: the consumer, the vault's name of its credential, and its value.
type ConsumerCredential struct {
// Consumer and Module are the consumer's machine and module, Local its local name for the provision, if any.
Consumer string `json:"consumer"`
Module string `json:"module"`
Local string `json:"local,omitempty"`
// Credential is the vault's name of the consumer's credential, PairName's.
Credential string `json:"credential"`
// Value is the credential itself.
Value string `json:"value"`
}
// ConsumersFile answers a credential from the files the provider machine's client fills: one per provision whose
// template names `${consumers:<provision>}`, each a JSON list of ConsumerCredential.
type ConsumersFile struct {
Paths []string
}
// Value is the credential named name, from the first file carrying it. A file not there yet is passed over; one not
// in the client's form is an error, never read as no consumers.
func (f ConsumersFile) Value(_ context.Context, name string) ([]byte, error) {
var missing []string
for _, at := range f.Paths {
raw, err := os.ReadFile(at)
if errors.Is(err, fs.ErrNotExist) {
missing = append(missing, at)
continue
}
if err != nil {
return nil, fmt.Errorf("the consumers' file %s: %w", at, err)
}
var entries []ConsumerCredential
if err := json.Unmarshal(raw, &entries); err != nil {
return nil, fmt.Errorf("the consumers' file %s is not the provisioner client's list of consumers", at)
}
for _, e := range entries {
if e.Credential == name {
if e.Value == "" {
return nil, fmt.Errorf("the consumers' file %s carries %s with no value", at, name)
}
return []byte(e.Value), nil
}
}
}
if len(missing) == len(f.Paths) {
return nil, fmt.Errorf("%s: no consumers' file is filled yet (%v)", name, f.Paths)
}
return nil, fmt.Errorf("%s is in no consumers' file: the vault holds none for it yet, or its machine's client "+
"has not filled it", name)
}
+69
View File
@@ -0,0 +1,69 @@
package provider
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
)
// The provider reads its consumers' credentials only from the file its machine's provisioner client fills (novox/hq
// ADR 0373 clause 3, issue 679): one entry per consumer, naming the consumer and the vault's name for its credential,
// and carrying the value. The helper is wired to it as its Vault and asks nothing of the vault itself.
func TestTheConsumersFileAnswersEachConsumersCredential(t *testing.T) {
dir := t.TempDir()
at := filepath.Join(dir, "consumers.json")
file := `[{"consumer":"ace","module":"letta","credential":"pair/novox/postgres-database/ace/letta","value":"s3cret"},
{"consumer":"g14","module":"gitea","local":"cache","credential":"pair/novox/postgres-database/g14/gitea/cache","value":"other"}]`
if err := os.WriteFile(at, []byte(file), 0o600); err != nil {
t.Fatal(err)
}
v := ConsumersFile{Paths: []string{filepath.Join(dir, "absent.json"), at}}
got, err := v.Value(context.Background(), PairName("novox", "postgres-database", "ace", "letta", ""))
if err != nil || string(got) != "s3cret" {
t.Fatalf("ace's letta: %q, %v; want s3cret", got, err)
}
got, err = v.Value(context.Background(), PairName("novox", "postgres-database", "g14", "gitea", "cache"))
if err != nil || string(got) != "other" {
t.Fatalf("g14's gitea under cache: %q, %v; want other", got, err)
}
// A consumer the file does not carry is waited for, and never told of a login its provider has not made.
_, err = v.Value(context.Background(), PairName("novox", "postgres-database", "shanks", "photos", ""))
if err == nil || !strings.Contains(err.Error(), "pair/novox/postgres-database/shanks/photos") {
t.Fatalf("a consumer the file does not carry: %v", err)
}
if strings.Contains(err.Error(), "s3cret") || strings.Contains(err.Error(), "other") {
t.Fatalf("an error names a credential: %v", err)
}
// Wired as the helper's Vault, the file is what makes each consumer's login.
bus := newStandIn()
bus.Put(context.Background(), AssignmentsBucket, "ace.letta", []byte(`{"node":"ace","module":"letta",
"provisioning":{"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres"}]}}`))
var made []string
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v,
Offers: map[string]Offer{"postgres-database": {Serves: map[string]any{"port": 5432}}},
Make: func(_ context.Context, c Consumer) error {
made = append(made, c.Entry.User+"="+string(c.Credential))
return nil
}, Logf: func(string, ...any) {}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
if len(made) != 1 || made[0] != "mesh_ace_letta=s3cret" {
t.Fatalf("made %v, want ace's letta under the file's credential", made)
}
}
// A file that is not the client's form is said, never read as no consumers.
func TestAConsumersFileNotInItsFormIsRefused(t *testing.T) {
at := filepath.Join(t.TempDir(), "consumers.json")
if err := os.WriteFile(at, []byte(`{"not": "a list"}`), 0o600); err != nil {
t.Fatal(err)
}
_, err := ConsumersFile{Paths: []string{at}}.Value(context.Background(), "pair/novox/p/ace/m")
if err == nil || !strings.Contains(err.Error(), at) {
t.Fatalf("a file not in the client's form: %v", err)
}
}
+62
View File
@@ -0,0 +1,62 @@
package provider
import (
"context"
"encoding/json"
"slices"
"testing"
)
// Issue 1098: an assignment with `ended` set, or with a `state`, is not a standing assignment: the helper reads it
// as a removed key, so the entry of its consumer is removed and the adapter hears it is gone. The fields the
// controller adds change nothing.
func TestAnEndedAssignmentIsReadAsARemovedKey(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.postgres-database"}) {
t.Fatalf("entries %v", got)
}
for _, mark := range []map[string]any{{"ended": "2026-10-18T01:00:00Z"}, {"state": "pending"}} {
raw := bus.get(AssignmentsBucket, "ace.letta")
var a map[string]any
if err := json.Unmarshal(raw, &a); err != nil {
t.Fatal(err)
}
for k, v := range mark {
a[k] = v
}
raw, _ = json.Marshal(a)
_ = bus.Put(context.Background(), AssignmentsBucket, "ace.letta", raw)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); len(got) != 0 || len(out) != 1 || out[0].Status != Removed {
t.Fatalf("%v: entries %v, outcomes %+v", mark, got, out)
}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
}
}
func TestStandingIsFalseOnlyForAnEndedOrPendingValue(t *testing.T) {
for raw, want := range map[string]bool{
`{"node":"a","module":"b"}`: true,
`{"node":"a","module":"b","ended":"","state":"","x":{"y":1}}`: true,
`{"node":"a","module":"b","ended":null}`: true,
`{"node":"a","module":"b","build":"b1","commit":"c","rollout":"r","since":"t","artifact":{"name":"n"}}`: true,
`{"node":"a","module":"b","ended":"2026-10-18T01:00:00Z"}`: false,
`{"node":"a","module":"b","state":"pending"}`: false,
} {
if got := Standing([]byte(raw)); got != want {
t.Errorf("%s: %v, want %v", raw, got, want)
}
}
}
+5
View File
@@ -0,0 +1,5 @@
package harness
import "context"
var ctx = context.Background()
+62
View File
@@ -0,0 +1,62 @@
package harness
// The moved path started before its runtime answers asks again until it does (issue 943).
import (
"context"
"errors"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// earlyBus refuses its first watches as the runtime does before Serve runs, then answers as the bucket bus does.
type earlyBus struct {
*bucketBus
refuse int
}
func (b *earlyBus) Watch(ctx context.Context, bucket string, handle func(string, []byte)) error {
if b.refuse > 0 {
b.refuse--
return errors.New("before Serve: a bundle reaches the bus through the runtime that launched it")
}
return b.bucketBus.Watch(ctx, bucket, handle)
}
func TestAMovedPathStartedTooEarlyAsksAgainUntilTheRuntimeAnswers(t *testing.T) {
w := newMovedWorld(t, map[[2]string]string{})
w.run.FirstRetry = time.Millisecond
w.run.loaded = false
bus := &earlyBus{bucketBus: w.bus, refuse: 3}
p := provider.New(provider.Options{Node: "novox", Module: "postgres", Bus: w.bus,
Vault: provider.ConsumersFile{Paths: []string{w.file}}, Make: w.run.Make, Gone: w.run.Gone,
Every: time.Hour, Logf: func(string, ...any) {}})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- w.run.Run(ctx, p, bus) }()
deadline := time.After(time.Second)
for {
w.run.mu.Lock()
loaded := w.run.loaded
w.run.mu.Unlock()
if loaded {
break
}
select {
case err := <-done:
cancel()
t.Fatalf("Run stopped before the runtime answered: %v", err)
case <-deadline:
cancel()
t.Fatal("the plans were never watched")
case <-time.After(time.Millisecond):
}
}
cancel()
<-done
if bus.refuse != 0 {
t.Fatalf("%d refusals left", bus.refuse)
}
}
+123
View File
@@ -0,0 +1,123 @@
package harness
// postgres handed to the provisioner still serves the consumers the controller's grant file gives (issue 937): a
// consumer not moved has no plan naming postgres, only its grant, until R1 (issue 682).
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"testing"
"time"
)
// granting gives the moved world a grant file, as the controller writes it for the consumers not moved.
func (w *movedWorld) granting(given ...map[string]any) {
if w.h.Receives == "" {
w.h.Receives = filepath.Join(w.t.TempDir(), "mesh.json")
}
dir := filepath.Dir(w.h.Receives)
for _, g := range given {
secret := filepath.Join(dir, g["as"].(string)+".secret")
if err := os.WriteFile(secret, []byte("pw-"+g["as"].(string)+"\n"), 0o600); err != nil {
w.t.Fatal(err)
}
g["secret"] = secret
}
if given == nil {
given = []map[string]any{}
}
raw, _ := json.Marshal(map[string]any{"requirement": ProvisionName, "given": given})
if err := os.WriteFile(w.h.Receives, raw, 0o600); err != nil {
w.t.Fatal(err)
}
}
// settled is enough passes and time for a retirement to be stable.
func (w *movedWorld) settled() {
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
w.pass()
}
// The logins the grant file gives are asked for: made as the grant path makes them, never offered for retirement,
// beside a moved consumer the helper makes from its plan.
func TestAConsumerOnlyTheGrantFileGivesIsAskedForAndMade(t *testing.T) {
bound, logins := bindings(1)
w := newMovedWorld(t, bound)
w.a.inv.Active = []string{"mesh_ace_baserow", "mesh_ace_letta", logins[0]}
w.bus.bind("ace", "app1", nil)
w.granting(
map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}},
map[string]any{"as": "mesh_ace_letta", "node": "ace",
"values": map[string]any{"name": "letta", "extensions": []any{"vector"}}},
)
w.settled()
if len(w.a.removed) != 0 || w.h.r.waiting != nil {
t.Fatalf("retired %v, waiting %+v: the grant file's logins were counted unasked", w.a.removed, w.h.r.waiting)
}
made := map[string]Provision{}
for _, c := range w.a.created {
made[c.As] = c
}
letta := made["mesh_ace_letta"]
if letta.Password != "pw-mesh_ace_letta" || letta.Consumer != "ace" ||
!reflect.DeepEqual(letta.Values, map[string]any{"name": "letta", "extensions": []any{"vector"}}) {
t.Fatalf("made %+v for letta, as its grant gives it", letta)
}
if _, ok := made["mesh_ace_baserow"]; !ok {
t.Fatalf("baserow not made: %+v", w.a.created)
}
if _, ok := made[logins[0]]; !ok {
t.Fatalf("the moved consumer not made: %+v", w.a.created)
}
}
// A login both a plan and the grant file name is the plan's: made once, with the plan's credential.
func TestALoginAPlanNamesIsNotMadeFromTheGrantFile(t *testing.T) {
bound, logins := bindings(1)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.granting(map[string]any{"as": logins[0], "node": "ace", "values": map[string]any{"name": "app1"}})
w.pass()
w.pass()
for _, c := range w.a.created {
if c.As == logins[0] && c.Password != "pw-app1" {
t.Fatalf("made %s with the grant's password %q, not the plan's", c.As, c.Password)
}
}
}
// An unreadable grant file is not nobody asking (issue 241): nothing is retired while it cannot be read.
func TestAnUnreadableGrantFileRetiresNothing(t *testing.T) {
bound, logins := bindings(1)
w := newMovedWorld(t, bound)
w.a.inv.Active = []string{"mesh_ace_baserow", logins[0]}
w.bus.bind("ace", "app1", nil)
w.h.Receives = filepath.Join(t.TempDir(), "absent.json")
w.settled()
if len(w.a.removed) != 0 || w.h.r.waiting != nil {
t.Fatalf("retired %v, waiting %+v with the grant file unreadable", w.a.removed, w.h.r.waiting)
}
}
// A login the grant file stops giving is retired by the usual rules: stable, then retired, never dropped.
func TestALoginTheGrantFileStopsGivingIsRetired(t *testing.T) {
bound, _ := bindings(1)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.granting(
map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}},
map[string]any{"as": "mesh_ace_letta", "node": "ace", "values": map[string]any{"name": "letta"}},
)
w.pass()
w.granting(map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}})
w.settled()
if !reflect.DeepEqual(w.a.removed, []string{"mesh_ace_letta"}) || len(w.a.deleted) != 0 {
t.Fatalf("retired %v, deleted %v", w.a.removed, w.a.deleted)
}
}
+572
View File
@@ -0,0 +1,572 @@
// Package harness is the reconcile loop and the retirement of a provider that keeps a resource per consumer; the
// module supplies an Adapter.
package harness
// The reconcile loop every provider shares, as the TypeScript SDK's runProvisioner runs it
// (@novox/mesh-sdk/provisioner, 0.1.12), line for line in behaviour. It moved here from the postgres and keycloak
// providers, which each carried an identical copy (novox/hq ADR 0039: the loop is the SDK's, the adapter is the
// module's; issue 849).
//
// Read the contributions the mesh delivered; bring each consumer's resource into being through the
// adapter, under the login and password the mesh minted; retire what the mesh no longer asks for.
// **A provider creates the credential the mesh minted, and seals nothing (novox/hq ADR 0048).**
//
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
// whose create, check or secret has failed without one success in between for FailingAfter is
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
// again every SayAgainEvery while it lasts; the first success after that is `provisioner.recovered`.
// The controller keeps the newest per provider and consumer and `status` names it. On 2026-10-05 the
// identity provider failed every consumer 31,000 times in a day and said so only in its journal
// (novox/hq issue 179).
//
// **A consumer the mesh stops asking for is retired, not withdrawn, and deleted only by a person
// (novox/hq ADR 0230, the operator's model of 2026-10-06).** Retiring disables its access — reversibly —
// and marks its login and data "to delete" with when and why; nothing is deleted. Asked for again, the
// ordinary create re-enables it as it was. It is retired only once the same set has gone unasked in
// StablePasses consecutive passes and for StableFor, and a set larger than the bound waits for a
// person. retirement.go.
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
"strings"
"sync"
"time"
)
// Provision is one consumer's resource to bring into being — everything the mesh derived and delivered.
type Provision struct {
// As is the login the mesh derived and gave the consumer to present.
As string
// Password is the one the mesh minted, read from the file the host unsealed.
Password string
// Values are what the consumer contributed (e.g. {"name": "letta", "extensions": ["vector"]}).
Values map[string]any
// Derived is what this provider's own definition derives for the consumer (novox/hq ADR 0201).
Derived map[string]any
// At is where the consumer is; Consumer is its node.
At string
Consumer string
}
// Adapter is the per-service half.
type Adapter interface {
// Create brings the consumer into being under the mesh's login and password — and, for one that
// was retired, re-enables it as it was and clears its mark to delete.
Create(ctx context.Context, p Provision) error
// Retire disables the consumer's access, reversibly, and marks its login and data to delete with
// when and why. It deletes nothing (novox/hq ADR 0230). derived is what the mesh last derived,
// remembered here; nil for a consumer this process did not make.
Retire(ctx context.Context, as string, derived map[string]any, why string, at time.Time) error
// Holds says whether the backend still holds the consumer exactly as p says. Read-only.
Holds(ctx context.Context, p Provision) (bool, error)
// Inventory is what the backend holds that the mesh made: consumers active and retired, read from
// the backend itself so a restart forgets nothing. Read-only.
Inventory(ctx context.Context) (Inventory, error)
// Delete removes one retired consumer — its login and its data — for good. Only ever asked by a
// person, through `cleanup delete`; the harness has checked it is retired and not asked for.
Delete(ctx context.Context, r Retired) (freedBytes int64, err error)
}
// Harness is the loop's settings and memory.
type Harness struct {
Resource string
Receives string
Adapter Adapter
Every time.Duration // 5s
VerifyEvery time.Duration // 60s
HoldsTimeout time.Duration // 30s
Log func(format string, args ...any)
Now func() time.Time
// Announce publishes one of the provider's standing events; nil announces nothing. Node is the
// machine this provider runs on, said in each.
Announce func(event string, body map[string]any)
Node string
// FailingAfter is how long a consumer fails without a success before it is announced (5m);
// SayAgainEvery is how often it is announced again while it lasts (15m), so a controller that
// missed the first hears the next, and a standing nobody repeats can be told from one that holds.
FailingAfter time.Duration
SayAgainEvery time.Duration
// StablePasses (5) is how many consecutive passes must see the same set no longer asked for, and
// StableFor (10m) how long it must have held since the first of them, before it is retired. RetireAtOnce (3) and RetireFraction (0.5) bound what is retired without a person:
// more consumers than RetireAtOnce, or — where more than one is held — a larger share of those held
// than RetireFraction, waits for `retire approve` (novox/hq ADR 0230).
StablePasses int
StableFor time.Duration
RetireAtOnce int
RetireFraction float64
// mu is held by a pass and by every tool a person asks, so the two never interleave.
mu sync.Mutex
verifiedAt time.Time
r retirement
applied map[string]appliedEntry
lost map[string]brake
waiting map[string]int
failing map[string]failure
trouble map[string]*standing
cleared map[string]bool
lastWarning string
}
// standing is one consumer's unbroken run of failures: since when, how often, and the last error.
type standing struct {
node string
since time.Time
attempts int
class string
text string
saidAt time.Time
}
// The events a provider's standing is announced as (novox/hq ADR 0224). The controller derives the
// permission to emit them for every module that receives contributions; no manifest lists them.
const (
EventFailing = "provisioner.failing"
EventRecovered = "provisioner.recovered"
)
// The classes of error a standing is announced with: what a person reading `status` needs to know
// before reading the journal. An adapter may say better (Classifier).
const (
ClassCredentials = "credentials-rejected"
ClassUnreachable = "unreachable"
ClassSecret = "secret-unreadable"
ClassRefused = "refused"
)
// Classifier is an adapter that can say what class an error of its own is.
type Classifier interface {
Class(err error) string
}
type appliedEntry struct {
hash string
derived map[string]any
node string
}
type brake struct {
times int
nextAt time.Time
}
type failure struct {
text string
times int
}
// The longest a consumer whose create keeps failing to satisfy holds waits between checks.
const maxBackoff = time.Hour
// How many passes a secret may be unreadable before it stops being called a race (issue 225), and
// once said loudly, how often it is repeated. The same cadence quiets a create that keeps failing
// the same way.
const (
patiently = 12
loudlyEvery = 240
)
type contribution struct {
As string `json:"as"`
Secret string `json:"secret"`
Node string `json:"node"`
At string `json:"at"`
Values map[string]any `json:"values"`
Derived map[string]any `json:"derived"`
}
func (h *Harness) init() {
if h.Every == 0 {
h.Every = 5 * time.Second
}
if h.VerifyEvery == 0 {
h.VerifyEvery = time.Minute
}
if h.HoldsTimeout == 0 {
h.HoldsTimeout = 30 * time.Second
}
if h.Now == nil {
h.Now = time.Now
}
if h.FailingAfter == 0 {
h.FailingAfter = 5 * time.Minute
}
if h.SayAgainEvery == 0 {
h.SayAgainEvery = 15 * time.Minute
}
if h.StablePasses == 0 {
h.StablePasses = 5
}
if h.StableFor == 0 {
h.StableFor = StableFor
}
if h.RetireAtOnce == 0 {
h.RetireAtOnce = 3
}
if h.RetireFraction == 0 {
h.RetireFraction = 0.5
}
if h.Log == nil {
h.Log = func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) }
}
if h.applied == nil {
h.applied = map[string]appliedEntry{}
h.lost = map[string]brake{}
h.waiting = map[string]int{}
h.failing = map[string]failure{}
h.trouble = map[string]*standing{}
h.cleared = map[string]bool{}
h.r.retired = map[string]retiredEntry{}
}
}
// Run reconciles until ctx ends. One consumer's failure never stops the others'.
func (h *Harness) Run(ctx context.Context) {
h.init()
for {
h.Reconcile(ctx)
select {
case <-ctx.Done():
return
case <-time.After(h.Every):
}
}
}
func (h *Harness) say(format string, args ...any) {
h.Log("[provisioner:"+h.Resource+"] "+format, args...)
}
func (h *Harness) warn(why string) {
if why == h.lastWarning {
return
}
if why != "" {
h.say("%s; nothing applied or removed until it can be read", why)
} else {
h.say("contributions file readable again")
}
h.lastWarning = why
}
// readContributions answers the consumers asked for, or nil when the file says nothing usable.
// **Nothing read is not nobody asking** (novox/hq issue 241): only a file that was read can retire anything.
func (h *Harness) readContributions() []contribution {
raw, err := os.ReadFile(h.Receives)
if err != nil {
h.warn(fmt.Sprintf("contributions file unreadable (%s): %v", h.Receives, err))
return nil
}
var doc struct {
Requirement string `json:"requirement"`
Given json.RawMessage `json:"given"`
}
if err := json.Unmarshal(raw, &doc); err != nil {
h.warn(fmt.Sprintf("contributions file is not JSON (%s): %v", h.Receives, err))
return nil
}
if doc.Requirement != "" && doc.Requirement != h.Resource {
h.warn(fmt.Sprintf("%s is for %s, not %s", h.Receives, doc.Requirement, h.Resource))
return nil
}
var given []contribution
if len(doc.Given) == 0 || string(doc.Given) == "null" || json.Unmarshal(doc.Given, &given) != nil {
h.warn(fmt.Sprintf("%s has no given list", h.Receives))
return nil
}
h.warn("")
out := []contribution{}
for _, g := range given {
// No `as` is not a credential grant: nothing to create for it.
if g.As != "" && g.Secret != "" {
out = append(out, g)
}
}
return out
}
func hashOf(as, password string, values, derived map[string]any) string {
// Derived is in the hash: a provider that renames what it derives gave a different resource.
b, _ := json.Marshal([]any{as, password, orEmpty(values), orEmpty(derived)})
return string(b)
}
func orEmpty(m map[string]any) map[string]any {
if m == nil {
return map[string]any{}
}
return m
}
// Reconcile is one pass.
func (h *Harness) Reconcile(ctx context.Context) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
given := h.readContributions()
if given == nil {
// Not a result: the passes that must agree before anything is retired start again.
h.r.key, h.r.count = "", 0
return
}
want := map[string]bool{}
for _, g := range given {
want[g.As] = true
}
h.seed(ctx, want)
h.apply(ctx, given)
// Retire what the mesh has stably stopped asking for — within the bound, or with a person.
h.retireUnasked(ctx, want)
h.r.lastWant = want
for as := range h.failing {
if !want[as] {
delete(h.failing, as)
}
}
// A consumer the mesh stopped asking for that this provider holds nothing for is no longer failed by
// anyone: said, so a standing the controller keeps for it is cleared rather than left naming a
// consumer that is gone. One still held is said recovered when it is retired.
for as := range h.trouble {
if _, held := h.applied[as]; !want[as] && !held {
h.recovered(as, "no longer asked for")
}
}
}
// apply makes each consumer the grant file gives, and checks every VerifyEvery that the backend still holds it. The
// caller holds h.mu. One consumer's failure never stops the others'.
func (h *Harness) apply(ctx context.Context, given []contribution) {
verifying := h.Now().Sub(h.verifiedAt) >= h.VerifyEvery
if verifying {
h.verifiedAt = h.Now()
}
for _, g := range given {
raw, err := os.ReadFile(g.Secret)
if err != nil {
// A secret the host has not written yet is a race on the first pass; past a minute it is
// a person's to look at, and said so (novox/hq issue 225).
n := h.waiting[g.As] + 1
h.waiting[g.As] = n
if n <= patiently {
h.say("%s: secret not readable yet (%s): %v", g.As, g.Secret, err)
} else if n == patiently+1 || n%loudlyEvery == 0 {
h.say("%s: CANNOT READ the secret after %d attempts (%s): %v. This is not a race any more — "+
"nothing has been provisioned for this consumer and nothing will be until somebody looks. "+
"Check who owns the file and who this process runs as (novox/hq issue 225)", g.As, n, g.Secret, err)
}
h.failed(g.As, g.Node, ClassSecret, fmt.Sprintf("secret not readable (%s): %v", g.Secret, err))
continue
}
delete(h.waiting, g.As)
password := strings.TrimSuffix(string(raw), "\n")
p := Provision{As: g.As, Password: password, Values: orEmpty(g.Values), Derived: orEmpty(g.Derived), At: g.At, Consumer: g.Node}
hash := hashOf(g.As, password, g.Values, g.Derived)
reapplying := 0
if was, ok := h.applied[g.As]; ok && was.hash == hash {
if !verifying {
continue
}
b, braked := h.lost[g.As]
if braked && h.Now().Before(b.nextAt) {
continue
}
hctx, cancel := context.WithTimeout(ctx, h.HoldsTimeout)
held, err := h.Adapter.Holds(hctx, p)
timedOut := errors.Is(hctx.Err(), context.DeadlineExceeded)
cancel()
if err != nil {
// Unable to ask is not evidence of loss. A backend that timed out will time out for
// the next consumer too, so the rest of this pass is not asked.
text := scrub(err, password)
h.say("%s: could not check the backend, will ask again: %s", g.As, text)
h.failed(g.As, g.Node, h.classOf(err, text), text)
if timedOut {
verifying = false
}
continue
}
if held {
delete(h.lost, g.As)
h.succeeded(g.As)
continue
}
reapplying = b.times + 1
if reapplying == 1 {
h.say("%s: the backend no longer holds it; applying again", g.As)
} else {
h.say("%s: still not held after being applied again (%d times in a row) — create does not "+
"produce what holds checks; applying again", g.As, reapplying)
}
}
if err := h.Adapter.Create(ctx, p); err != nil {
text := scrub(err, password)
f := h.failing[g.As]
if f.text != text {
f = failure{text: text}
}
f.times++
h.failing[g.As] = f
// Said each time it changes, and while it stays the same, as rarely as a lost secret.
if f.times == 1 || f.times%loudlyEvery == 0 {
h.say("%s: create failed, will retry: %s", g.As, text)
}
h.failed(g.As, g.Node, h.classOf(err, text), text)
if reapplying > 0 {
h.lost[g.As] = brake{times: reapplying - 1}
}
continue
}
if f, was := h.failing[g.As]; was {
h.say("%s: created, after %d failed attempt(s)", g.As, f.times)
delete(h.failing, g.As)
}
h.succeeded(g.As)
h.applied[g.As] = appliedEntry{hash: hash, derived: p.Derived, node: g.Node}
h.reenabled(g.As, g.Node)
if reapplying == 0 {
delete(h.lost, g.As)
} else {
wait := h.VerifyEvery << (reapplying - 1)
if wait > maxBackoff || wait <= 0 {
wait = maxBackoff
}
h.lost[g.As] = brake{times: reapplying, nextAt: h.Now().Add(wait)}
if reapplying > 1 {
h.say("%s: next check in %s", g.As, wait.Round(time.Second))
}
}
}
}
// failed counts one more failure in a consumer's unbroken run, and announces the run once it has
// lasted FailingAfter — then again every SayAgainEvery while it lasts.
func (h *Harness) failed(as, node, class, text string) {
now := h.Now()
s := h.trouble[as]
if s == nil {
s = &standing{since: now}
h.trouble[as] = s
}
s.node, s.class, s.text = node, class, text
s.attempts++
if now.Sub(s.since) < h.FailingAfter {
return
}
if !s.saidAt.IsZero() && now.Sub(s.saidAt) < h.SayAgainEvery {
return
}
first := s.saidAt.IsZero()
s.saidAt = now
if first {
h.say("%s: FAILING for %s (%d attempts, %s): %s. Announced as %s; `status` names it until it "+
"succeeds (novox/hq ADR 0224)", as, now.Sub(s.since).Round(time.Second), s.attempts, class, text, EventFailing)
}
h.announce(EventFailing, map[string]any{
"provider": h.Resource, "provider-node": h.Node,
"consumer": as, "node": node,
"class": class, "error": clip(text),
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
})
}
// succeeded ends a consumer's run of failures; one that was announced is announced recovered.
//
// **And the first success for a consumer since this process started is announced too**, failing or
// not: a provider that announced a failure and was restarted has forgotten it, and without this the
// controller would name the consumer failing for ever after it recovered unheard.
func (h *Harness) succeeded(as string) {
if h.trouble[as] == nil && !h.cleared[as] {
h.cleared[as] = true
h.announce(EventRecovered, map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "why": "first-success",
})
return
}
h.cleared[as] = true
h.recovered(as, "")
}
func (h *Harness) recovered(as, why string) {
s := h.trouble[as]
if s == nil {
return
}
delete(h.trouble, as)
if s.saidAt.IsZero() {
return // never announced, so there is nothing to take back
}
if why == "" {
h.say("%s: recovered after %s and %d failed attempt(s)", as, h.Now().Sub(s.since).Round(time.Second), s.attempts)
}
body := map[string]any{
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "node": s.node,
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
}
if why != "" {
body["why"] = why
}
h.announce(EventRecovered, body)
}
func (h *Harness) announce(event string, body map[string]any) {
if h.Announce != nil {
h.Announce(event, body)
}
}
// classOf is an error's class: the adapter's word when it has one, else read from the text.
func (h *Harness) classOf(err error, text string) string {
if c, ok := h.Adapter.(Classifier); ok {
if class := c.Class(err); class != "" {
return class
}
}
return ClassOf(text)
}
// ClassOf reads an error's class from its text — the words the backends the mesh runs use.
func ClassOf(text string) string {
t := strings.ToLower(text)
for _, w := range []string{"invalid_grant", "invalid user credentials", "password authentication failed",
"authentication failed", "unauthorized", " 401"} {
if strings.Contains(t, w) {
return ClassCredentials
}
}
for _, w := range []string{"connection refused", "no such host", "i/o timeout", "deadline exceeded",
"connection reset", "network is unreachable", "no route to host", "eof"} {
if strings.Contains(t, w) {
return ClassUnreachable
}
}
return ClassRefused
}
// clip keeps an announced error to what belongs in a status line.
func clip(text string) string {
const most = 300
if len(text) <= most {
return text
}
return text[:most] + "…"
}
// scrub is an error's text with the consumer's password removed, raw and URL-encoded.
func scrub(err error, password string) string {
text := err.Error()
if password == "" {
return text
}
for _, form := range []string{password, url.QueryEscape(password), url.PathEscape(password)} {
text = strings.ReplaceAll(text, form, "***")
}
return text
}
+154
View File
@@ -0,0 +1,154 @@
package harness
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
type world struct {
t *testing.T
dir string
receives string
now time.Time
h *Harness
a *recorder
said []string
}
func newWorld(t *testing.T) *world {
w := &world{t: t, dir: t.TempDir(), now: time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC), a: &recorder{held: true}}
w.receives = filepath.Join(w.dir, "mesh.json")
w.h = &Harness{Resource: "postgres-database", Receives: w.receives, Adapter: w.a,
Now: func() time.Time { return w.now },
Log: func(f string, a ...any) { w.said = append(w.said, f) }}
return w
}
func (w *world) give(given ...map[string]any) {
for _, g := range given {
secret := filepath.Join(w.dir, g["as"].(string)+".secret")
if err := os.WriteFile(secret, []byte("pw-"+g["as"].(string)+"\n"), 0o600); err != nil {
w.t.Fatal(err)
}
g["secret"] = secret
}
if given == nil {
given = []map[string]any{}
}
raw, _ := json.Marshal(map[string]any{"requirement": "postgres-database", "given": given})
if err := os.WriteFile(w.receives, raw, 0o600); err != nil {
w.t.Fatal(err)
}
}
func TestAConsumerIsCreatedOnceUnderTheMeshsLoginAndPassword(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "mesh_ace_letta", "node": "ace", "values": map[string]any{"name": "letta"}})
w.h.Reconcile(ctx)
w.h.Reconcile(ctx)
if len(w.a.created) != 1 {
t.Fatalf("created %d times", len(w.a.created))
}
p := w.a.created[0]
if p.As != "mesh_ace_letta" || p.Password != "pw-mesh_ace_letta" || p.Consumer != "ace" {
t.Fatalf("%+v", p)
}
}
func TestAConsumerNoLongerAskedForIsRetiredOnceStable(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
w.h.Reconcile(ctx)
w.give(map[string]any{"as": "a"})
w.settle() // five passes and ten minutes
if strings.Join(w.a.removed, ",") != "b" {
t.Fatal(w.a.removed)
}
// Only a file that says nobody asks retires the last one.
w.give()
w.settle()
if strings.Join(w.a.removed, ",") != "b,a" {
t.Fatal(w.a.removed)
}
}
func TestNothingReadIsNotNobodyAsking(t *testing.T) {
for name, content := range map[string]string{
"unreadable": "",
"not JSON": "{",
"no given": `{"requirement": "postgres-database"}`,
"another": `{"requirement": "mssql-database", "given": []}`,
} {
t.Run(name, func(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
if content == "" {
os.Remove(w.receives)
} else {
os.WriteFile(w.receives, []byte(content), 0o600)
}
w.h.Reconcile(ctx)
if len(w.a.removed) != 0 {
t.Fatalf("retired %v on a file it could not use", w.a.removed)
}
})
}
}
func TestALostConsumerIsAppliedAgainAndBraked(t *testing.T) {
w := newWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
w.a.held = false
w.now = w.now.Add(2 * time.Minute)
w.h.Reconcile(ctx)
if len(w.a.created) != 2 {
t.Fatalf("created %d times", len(w.a.created))
}
// Still not held a minute later: applied again, then braked for two minutes.
w.now = w.now.Add(61 * time.Second)
w.h.Reconcile(ctx)
w.now = w.now.Add(61 * time.Second)
w.h.Reconcile(ctx)
if len(w.a.created) != 3 {
t.Fatalf("not braked: created %d times", len(w.a.created))
}
}
func TestAFailingCreateIsRetriedAndSaidOnce(t *testing.T) {
w := newWorld(t)
w.a.failing = &pgErr{"extension \"nope\" refused, password pw-a"}
w.give(map[string]any{"as": "a"})
for i := 0; i < 5; i++ {
w.h.Reconcile(ctx)
}
n := 0
for _, s := range w.said {
if strings.Contains(s, "create failed") {
n++
}
}
if n != 1 {
t.Fatalf("said %d times", n)
}
w.a.failing = nil
w.h.Reconcile(ctx)
if len(w.a.created) != 1 {
t.Fatal("not retried")
}
}
type pgErr struct{ s string }
func (e *pgErr) Error() string { return e.s }
func TestScrubRemovesThePassword(t *testing.T) {
if got := scrub(&pgErr{"bad pw a/b c and a%2Fb+c"}, "a/b c"); strings.Contains(got, "a/b c") || strings.Contains(got, "a%2Fb+c") {
t.Fatal(got)
}
}
+339
View File
@@ -0,0 +1,339 @@
package harness
// A provider handed to the provisioner (novox/hq ADR 0373, ADR 0349 clause 12, issues 780 and 805): where its
// consumers file is set, it runs on the provider helper (package provider) instead of the harness reading the
// controller's grant files. Moved is that path, shared by every provider that keeps a resource per consumer; a module
// says only what differs, in a MovedSpec.
//
// The helper learns who binds the provider from the plans, composes each consumer's entry and reads its credential
// through ConsumersFile, the file this machine's provisioner client fills. It leaves the rest to the two hooks
// here: Make brings the consumer's resource into being exactly as the harness's adapter does, and Gone leaves
// retirement to the harness's guards (retirement.go): the resource is disabled and marked to delete, never removed.
//
// **A consumer not moved is still the grant file's** (issue 937, ADR 0349 clause 17), for a provider whose spec says
// WithGrants: only a moved consumer's plan names its provider, so the controller still gives every other consumer
// through the grant file (MESH_RECEIVES) until R1 (issue 682). Each login it gives is asked for, and one no plan
// names is made and checked by the harness's grant pass exactly as on the grant path; a login a plan names is the
// helper's alone. A grant file that cannot be read is not nobody asking (issue 241): nothing is retired until it is
// read again.
import (
"context"
"encoding/json"
"errors"
"fmt"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// MovedSpec is what differs per provider on the moved path.
type MovedSpec struct {
// Module and Provision are the provider module's name and what it provides, as its manifest says.
Module, Provision string
// Offer is what it serves: its credential form and served values.
Offer provider.Offer
// Login is the identity a consumer is made under at the provider's end (postgres: the entry's User; keycloak: its
// As); an entry with none is not made.
Login func(e provider.Entry) string
// Values are the values the adapter is handed for a consumer; the entry's contribution when not set.
Values func(e provider.Entry) map[string]any
// Noun names what Login is in words ("login", "client id"), Made what is made ("login", "client"), and Backend
// the service in "no longer holds it" ("the backend", "Keycloak").
Noun, Made, Backend string
// WithGrants makes the logins the grant file gives, that no plan names, asked for and made by the harness's grant
// pass (issue 937).
WithGrants bool
}
// Moved is a provider on the provider helper, with the harness's own memory and retirement behind it: what is held,
// what is counted unasked, what waits for a person (retirement.go). The harness is never Run here; the helper's Make
// and Gone and the loop below drive the same state, so the guards are the harness's, not a copy.
type Moved struct {
h *Harness
spec MovedSpec
// FirstRetry is the first wait before a watch asked too early is asked again (issue 943); a second when zero.
FirstRetry time.Duration
mu sync.Mutex // guards assignments and loaded
assignments map[string][]byte
loaded bool
verified map[string]time.Time
}
func newMoved(h *Harness, spec MovedSpec) *Moved {
h.mu.Lock()
h.init()
h.mu.Unlock()
return &Moved{h: h, spec: spec, assignments: map[string][]byte{}, verified: map[string]time.Time{}}
}
// Make brings one consumer's resource into being under its login and the vault's credential, as the harness does:
// at once when it is new or changed, and otherwise only when the backend no longer holds it, which is asked every
// VerifyEvery.
func (m *Moved) Make(ctx context.Context, c provider.Consumer) error {
h, e := m.h, c.Entry
as := m.spec.Login(e)
if as == "" {
return fmt.Errorf("%s has no %s to make", e.Key(), m.spec.Noun)
}
if len(c.Credential) == 0 {
return fmt.Errorf("%s has no credential to make its %s with", e.Key(), m.spec.Made)
}
h.mu.Lock()
defer h.mu.Unlock()
password := string(c.Credential)
values := orEmpty(e.Contributes)
if m.spec.Values != nil {
values = m.spec.Values(e)
}
p := Provision{As: as, Password: password, Values: values, Consumer: e.Consumer}
hash := hashOf(as, password, p.Values, nil)
if was, ok := h.applied[as]; ok && was.hash == hash {
if h.Now().Sub(m.verified[as]) < h.VerifyEvery {
return nil
}
m.verified[as] = h.Now()
hctx, cancel := context.WithTimeout(ctx, h.HoldsTimeout)
held, err := h.Adapter.Holds(hctx, p)
cancel()
if err != nil {
text := scrub(err, password)
h.failed(as, e.Consumer, h.classOf(err, text), text)
return errors.New(text)
}
if held {
h.succeeded(as)
return nil
}
h.say("%s: %s no longer holds it; applying again", as, m.spec.Backend)
}
if err := h.Adapter.Create(ctx, p); err != nil {
text := scrub(err, password)
h.failed(as, e.Consumer, h.classOf(err, text), text)
return errors.New(text)
}
h.succeeded(as)
h.applied[as] = appliedEntry{hash: hash, node: e.Consumer}
m.verified[as] = h.Now()
h.reenabled(as, e.Consumer)
return nil
}
// Gone is the helper asking whether it may forget a consumer that no longer binds. **It never retires**: retiring is
// the harness's rule, the same unasked set over StablePasses passes and StableFor, at most RetireAtOnce at a time
// (retirement.go, ADR 0230), run by retire below. So a consumer missing from one read of the plans loses nothing:
// the helper is told no, and asks again at its next pass, until the login is retired, or asked for again.
func (m *Moved) Gone(_ context.Context, e provider.Entry) error {
h := m.h
as := m.spec.Login(e)
if as == "" {
return nil // refused: nothing was made for it
}
// The plans and the grant file as they are now, not as the last retirement pass read them: with either not
// read, nothing is let go.
want, read := m.asked(h.Node)
h.mu.Lock()
defer h.mu.Unlock()
asked := want[as]
if m.spec.WithGrants {
_, granted, ok := m.granted(want)
read = read && ok
asked = asked || granted[as]
}
_, held := h.applied[as]
_, retired := h.r.retired[as]
switch {
case !read:
case asked:
return nil // another holder of the same login still binds: nothing to retire
case retired, !held && h.r.seeded:
return nil
}
return fmt.Errorf("%s is kept until its retirement is stable (%d passes and %s) and within the bound, or a "+
"person approves it", as, h.StablePasses, h.StableFor)
}
// plan is as much of an assignment as is read to learn who binds the provider.
type plan struct {
Node string `json:"node"`
Module string `json:"module"`
Provisioning *struct {
Provisions []struct {
Provision string `json:"provision"`
Provider string `json:"provider"`
ProviderModule string `json:"provider-module"`
// As is who the consumer is at its provider's end, its slug first (issue 788).
As string `json:"as"`
} `json:"provisions"`
} `json:"provisioning"`
}
// asked is the logins the plans bind the provider for, and whether they could all be read: a plan that is not the
// controller's, or no read yet, is not an answer, so nothing is counted unasked from it (novox/hq issue 241).
func (m *Moved) asked(node string) (map[string]bool, bool) {
m.mu.Lock()
defer m.mu.Unlock()
if !m.loaded || len(m.assignments) == 0 {
return nil, false
}
want := map[string]bool{}
for key, raw := range m.assignments {
var a plan
if err := json.Unmarshal(raw, &a); err != nil || a.Node+"."+a.Module != key {
return nil, false
}
if a.Provisioning == nil {
continue
}
for _, b := range a.Provisioning.Provisions {
if b.Provider == node && b.ProviderModule == m.spec.Module && b.Provision == m.spec.Provision {
as := b.As
if as == "" {
as = provider.ConsumerIdentity(a.Node, a.Module) // a plan that names none
}
want[as] = true
}
}
}
return want, true
}
// granted is what the grant file gives that no plan names (issue 937), and every login it gives; and whether it
// could be read. The caller holds h.mu. With no grant file named there is nothing given, and that is a read.
func (m *Moved) granted(planned map[string]bool) ([]contribution, map[string]bool, bool) {
h := m.h
if h.Receives == "" {
return nil, map[string]bool{}, true
}
given := h.readContributions()
if given == nil {
return nil, nil, false
}
var own []contribution
logins := map[string]bool{}
for _, g := range given {
logins[g.As] = true
if !planned[g.As] {
own = append(own, g)
}
}
return own, logins, true
}
// retire is one pass of the harness's retirement over what the plans bind and the grant file gives now; the logins
// only the grant file gives are made and checked first, as the grant path does.
func (m *Moved) retire(ctx context.Context) {
h := m.h
want, ok := m.asked(h.Node)
h.mu.Lock()
defer h.mu.Unlock()
var given []contribution
if ok && m.spec.WithGrants {
var granted map[string]bool
given, granted, ok = m.granted(want)
for as := range granted {
want[as] = true
}
}
if !ok {
// Not a result: the passes that must agree before anything is retired start again.
h.r.key, h.r.count = "", 0
return
}
h.seed(ctx, want)
h.apply(ctx, given)
h.retireUnasked(ctx, want)
h.r.lastWant = want
for as := range h.failing {
if !want[as] {
delete(h.failing, as)
}
}
for as := range h.trouble {
if _, held := h.applied[as]; !want[as] && !held {
h.recovered(as, "no longer asked for")
}
}
}
// watch keeps the plans as the bus holds them, from the first read on.
func (m *Moved) watch(ctx context.Context, bus provider.Bus) error {
err := bus.Watch(ctx, provider.AssignmentsBucket, func(key string, value []byte) {
m.mu.Lock()
defer m.mu.Unlock()
if value == nil {
delete(m.assignments, key)
} else {
m.assignments[key] = append([]byte(nil), value...)
}
})
if err == nil {
m.mu.Lock()
m.loaded = true
m.mu.Unlock()
}
return err
}
// Run answers the consumers through the helper and retires through the harness's rules, until ctx ends.
//
// The bus is reached through the runtime that launched this bundle, which answers only once Serve runs, and Run starts
// beside Serve: a watch asked too early is asked again, after a wait that doubles from FirstRetry up to a minute,
// until it is answered or ctx ends, so a start that loses the race never leaves the consumers unanswered (issue 943).
func (m *Moved) Run(ctx context.Context, p *provider.Provider, bus provider.Bus) error {
wait := m.FirstRetry
if wait <= 0 {
wait = time.Second
}
for tries := 1; ; tries++ {
err := m.watch(ctx, bus)
if err == nil {
break
}
if tries == 4 || tries%20 == 0 {
m.h.say("the plans cannot be watched yet (%d tries), asking again: %v", tries, err)
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
wait = min(wait*2, time.Minute)
}
go func() {
for {
m.retire(ctx)
select {
case <-ctx.Done():
return
case <-time.After(m.h.Every):
}
}
}()
return p.Run(ctx)
}
// NewMoved is a provider handed to the provisioner: the helper answering its consumers, and the harness (never run)
// whose retirement and tools it shares. consumersFile is where its machine's client fills its consumers. Run the
// result with Moved.Run, passing the provider and the bus returned with it.
func NewMoved(consumersFile string, h *Harness, spec MovedSpec) (*Moved, *provider.Provider, provider.Bus, error) {
if consumersFile == "" {
return nil, nil, nil, errors.New("no consumers' file")
}
run := newMoved(h, spec)
bus := provider.StateBus(spec.Module)
return run, provider.New(provider.Options{
Node: h.Node,
Module: spec.Module,
Offers: map[string]provider.Offer{spec.Provision: spec.Offer},
Bus: bus,
Vault: provider.ConsumersFile{Paths: []string{consumersFile}},
Make: run.Make,
Gone: run.Gone,
Every: h.Every,
Logf: func(f string, a ...any) { h.say(f, a...) },
}), bus, nil
}
+480
View File
@@ -0,0 +1,480 @@
package harness
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/provider"
)
const (
ProvisionName = "postgres-database"
servedPort = 5432
)
// postgresSpec is what a postgres-like provider says: a login per consumer, and the grant file's logins.
var postgresSpec = MovedSpec{Module: "postgres", Provision: ProvisionName,
Offer: provider.Offer{Credential: provider.PerConsumer, Serves: map[string]any{"port": servedPort}},
Login: func(e provider.Entry) string { return e.User },
Noun: "login", Made: "login", Backend: "the backend", WithGrants: true}
// bucketBus is the helper's bus over in-memory buckets.
type bucketBus struct {
mu sync.Mutex
buckets map[string]map[string][]byte
}
func (b *bucketBus) Watch(_ context.Context, bucket string, handle func(string, []byte)) error {
b.mu.Lock()
defer b.mu.Unlock()
for k, v := range b.buckets[bucket] {
handle(k, v)
}
return nil
}
func (b *bucketBus) Put(_ context.Context, bucket, key string, value []byte) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.buckets[bucket] == nil {
b.buckets[bucket] = map[string][]byte{}
}
b.buckets[bucket][key] = append([]byte(nil), value...)
return nil
}
func (b *bucketBus) Delete(_ context.Context, bucket, key string) error {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.buckets[bucket], key)
return nil
}
// bind is one consumer's assignment, whose plan names this machine's postgres as the provider and carries what the
// consumer contributes (issue 776).
func (b *bucketBus) bind(node, module string, contributes map[string]any) {
b.bindNamed("postgres", node, module, contributes, nil)
}
// bindNamed is bind to a provider module, with the names the controller composed for the consumer (issue 792).
func (b *bucketBus) bindNamed(provides, node, module string, contributes map[string]any, names map[string]string) {
plan, _ := json.Marshal(map[string]any{"node": node, "module": module, "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox", "provider-module": provides,
"contributes": contributes, "names": names}}}})
if err := b.Put(context.Background(), provider.AssignmentsBucket, node+"."+module, plan); err != nil {
panic(err)
}
}
// consumersFile writes the file the machine's client fills: one credential per consumer.
func consumersFile(t *testing.T, bound map[[2]string]string) string {
var list []provider.ConsumerCredential
for k, password := range bound {
list = append(list, provider.ConsumerCredential{Consumer: k[0], Module: k[1],
Credential: provider.PairName("novox", ProvisionName, k[0], k[1], ""), Value: password})
}
raw, _ := json.Marshal(list)
at := filepath.Join(t.TempDir(), "consumers.json")
if err := os.WriteFile(at, raw, 0o600); err != nil {
t.Fatal(err)
}
return at
}
// movedWorld is postgres on the helper over an in-memory bus, with a recorder for a server and a clock to move.
type movedWorld struct {
t *testing.T
a *recorder
bus *bucketBus
h *Harness
spec MovedSpec
run *Moved
p *provider.Provider
now time.Time
file string
}
func newMovedWorld(t *testing.T, bound map[[2]string]string) *movedWorld {
return newMovedWorldOf(t, bound, postgresSpec)
}
func newMovedWorldOf(t *testing.T, bound map[[2]string]string, spec MovedSpec) *movedWorld {
w := &movedWorld{t: t, spec: spec, a: &recorder{}, bus: &bucketBus{buckets: map[string]map[string][]byte{}},
now: time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)}
w.file = consumersFile(t, bound)
w.h = &Harness{Resource: ProvisionName, Adapter: w.a, Node: "novox", Now: func() time.Time { return w.now },
Log: func(string, ...any) {}}
var bus provider.Bus
w.run, w.p, bus, _ = NewMoved(w.file, w.h, spec)
_ = bus
if err := w.run.watch(ctx, w.bus); err != nil {
t.Fatal(err)
}
return w
}
// pass is one helper pass and one retirement pass, five seconds after the last.
func (w *movedWorld) pass() []provider.Outcome {
w.now = w.now.Add(5 * time.Second)
// the plans as the bus holds them now, as the watch would hand them
w.run.assignments = map[string][]byte{}
for k, v := range w.bus.buckets[provider.AssignmentsBucket] {
w.run.assignments[k] = v
}
// the helper reads its buckets from the in-memory bus
w.p = provider.New(provider.Options{Node: "novox", Module: w.spec.Module, Bus: w.bus,
Vault: provider.ConsumersFile{Paths: []string{w.file}},
Offers: map[string]provider.Offer{ProvisionName: w.spec.Offer},
Make: w.run.Make, Gone: w.run.Gone, Logf: func(string, ...any) {}})
out, err := w.p.Pass(ctx)
if err != nil {
w.t.Fatal(err)
}
w.run.retire(ctx)
return out
}
func bindings(n int) (map[[2]string]string, []string) {
bound := map[[2]string]string{}
var logins []string
for i := 1; i <= n; i++ {
module := fmt.Sprintf("app%d", i)
bound[[2]string{"ace", module}] = "pw-" + module
logins = append(logins, provider.ConsumerIdentity("ace", module))
}
return bound, logins
}
// The consumers' file makes the same calls on the adapter as the equivalent grant files do (the harness).
func TestTheConsumersFileMakesWhatTheGrantFilesMake(t *testing.T) {
letta := provider.ConsumerIdentity("ace", "letta")
plain := provider.ConsumerIdentity("ace", "plain")
// Today: the grant files, through the harness.
g := newWorld(t)
g.give(
map[string]any{"as": letta, "node": "ace", "values": map[string]any{"name": "letta", "extensions": []any{"vector"}}},
map[string]any{"as": plain, "node": "ace", "values": map[string]any{"name": "plain"}},
)
g.h.Reconcile(ctx)
// The consumers' file, through the provider helper.
w := newMovedWorld(t, map[[2]string]string{{"ace", "letta"}: "pw-" + letta, {"ace", "plain"}: "pw-" + plain})
w.bus.bind("ace", "letta", map[string]any{"name": "letta", "extensions": []any{"vector"}})
w.bus.bind("ace", "plain", map[string]any{"name": "plain"})
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
byAs := map[string]Provision{}
for _, c := range w.a.created {
byAs[c.As] = c
}
if len(w.a.created) != 2 {
t.Fatalf("created %d, want 2", len(w.a.created))
}
for _, want := range g.a.created {
c := byAs[want.As]
if c.As != want.As || c.Password != want.Password || c.Consumer != want.Consumer ||
!reflect.DeepEqual(orEmpty(c.Values), orEmpty(want.Values)) {
t.Fatalf("made %+v, the grant files make %+v", c, want)
}
}
// Unchanged, and held: nothing is made again, within a minute or after it.
w.a.held = true
w.pass()
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 2 {
t.Fatalf("an unchanged pass created again: %d", len(w.a.created))
}
// No longer held: made again.
w.a.held = false
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 4 {
t.Fatalf("not made again: %d", len(w.a.created))
}
}
// One read without a consumer retires nothing: it takes the harness's stable passes and time, and then the entry
// goes and the login is retired, never dropped.
func TestAConsumerMissingFromOneReadRetiresNothing(t *testing.T) {
bound, logins := bindings(2)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.bus.bind("ace", "app2", nil)
w.pass()
// One read without app2.
saved := w.bus.buckets[provider.AssignmentsBucket]["ace.app2"]
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app2")
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v on one read", w.a.removed)
}
if _, kept := w.bus.buckets[provider.BucketOf("postgres")]["ace.app2.postgres-database"]; !kept {
t.Fatal("the entry went on one read")
}
// It is back: nothing was lost, and the entry never left the bucket.
w.bus.Put(ctx, provider.AssignmentsBucket, "ace.app2", saved)
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
if len(w.a.removed) != 0 || len(w.bus.buckets[provider.BucketOf("postgres")]) != 2 {
t.Fatalf("retired %v, entries %d", w.a.removed, len(w.bus.buckets[provider.BucketOf("postgres")]))
}
// Gone for good: not retired before StablePasses and StableFor, then retired and its entry removed.
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app2")
for i := 0; i < 6; i++ {
w.pass()
}
if len(w.a.removed) != 0 {
t.Fatalf("retired %v before the time was out", w.a.removed)
}
w.now = w.now.Add(11 * time.Minute)
w.pass()
w.pass()
if !reflect.DeepEqual(w.a.removed, []string{logins[1]}) || len(w.a.deleted) != 0 {
t.Fatalf("retired %v, deleted %v", w.a.removed, w.a.deleted)
}
if _, kept := w.bus.buckets[provider.BucketOf("postgres")]["ace.app2.postgres-database"]; kept {
t.Fatal("the entry stayed after the login was retired")
}
}
// More than three at once waits for a person, as the harness's bound says.
func TestMoreThanTheBoundWaitsForAPerson(t *testing.T) {
bound, _ := bindings(5)
w := newMovedWorld(t, bound)
for i := 1; i <= 5; i++ {
w.bus.bind("ace", fmt.Sprintf("app%d", i), nil)
}
w.pass()
for i := 1; i <= 4; i++ {
w.bus.Delete(ctx, provider.AssignmentsBucket, fmt.Sprintf("ace.app%d", i))
}
for i := 0; i < 6; i++ {
w.pass()
}
w.now = w.now.Add(11 * time.Minute)
w.pass()
w.pass()
if len(w.a.removed) != 0 || w.h.r.waiting == nil {
t.Fatalf("retired %v, waiting %v", w.a.removed, w.h.r.waiting)
}
}
// A plan list that was never read is not nobody asking.
func TestNoPlansReadRetiresNothing(t *testing.T) {
bound, _ := bindings(1)
w := newMovedWorld(t, bound)
w.bus.bind("ace", "app1", nil)
w.pass()
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app1")
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(20 * time.Minute)
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v with no plans read", w.a.removed)
}
}
// A consumer whose credential is not in the file yet is waited for: nothing is made for it.
func TestAConsumerWithoutACredentialIsWaitedFor(t *testing.T) {
w := newMovedWorld(t, map[[2]string]string{})
w.bus.bind("ace", "letta", map[string]any{"name": "letta"})
outcomes := w.pass()
if len(outcomes) != 1 || outcomes[0].Status != provider.Waiting || len(w.a.created) != 0 {
t.Fatalf("%+v, created %v", outcomes, w.a.created)
}
}
// Where the file is not there, postgres takes the grant directory, as before; where it is, the helper.
// The grant path restarts itself when the consumers file is first filled.
// A consumer whose module has a slug keeps the login its plan names, the slug first (issue 788): the adapter makes it
// under that login, and the retirement counts it as asked for, so a pass retires nothing.
func TestASluggedConsumerKeepsTheLoginItsPlanNames(t *testing.T) {
agent := provider.ConsumerIdentity("ace", "agent")
w := newMovedWorld(t, map[[2]string]string{{"ace", "claude-code"}: "pw-agent"})
plan, _ := json.Marshal(map[string]any{"node": "ace", "module": "claude-code", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
"provider-module": "postgres", "as": agent}}}})
if err := w.bus.Put(ctx, provider.AssignmentsBucket, "ace.claude-code", plan); err != nil {
t.Fatal(err)
}
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
if len(w.a.created) != 1 || w.a.created[0].As != agent {
t.Fatalf("made %+v, want one under %s", w.a.created, agent)
}
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v: the plan's login was counted unasked", w.a.removed)
}
}
// A provider whose spec names the plan's `as` as the login and no grant file (the keycloak shape) makes the consumer
// under that name with the values its spec composes, and a grant file that cannot be read does not hold up its
// retirement pass.
func TestASpecWithoutGrantsMakesUnderThePlansAsAndIgnoresTheGrantFile(t *testing.T) {
a := &recorder{}
bus := &bucketBus{buckets: map[string]map[string][]byte{}}
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
h := &Harness{Resource: ProvisionName, Adapter: a, Node: "novox", Now: func() time.Time { return now },
Receives: filepath.Join(t.TempDir(), "unreadable-or-missing.json"), Log: func(string, ...any) {}}
spec := MovedSpec{Module: "keycloak", Provision: ProvisionName,
Offer: provider.Offer{Credential: provider.PerConsumer},
Login: func(e provider.Entry) string { return e.As }, Noun: "client id", Made: "client", Backend: "Keycloak",
Values: func(e provider.Entry) map[string]any { return map[string]any{"label": e.Contributes["label"]} }}
file := consumersFile(t, map[[2]string]string{{"ace", "wiki"}: "pw-wiki"})
run, _, _, err := NewMoved(file, h, spec)
if err != nil {
t.Fatal(err)
}
plan, _ := json.Marshal(map[string]any{"node": "ace", "module": "wiki", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
"provider-module": "keycloak", "as": "wiki-client", "contributes": map[string]any{"label": "Wiki"}}}}})
_ = bus.Put(ctx, provider.AssignmentsBucket, "ace.wiki", plan)
if err := run.watch(ctx, bus); err != nil {
t.Fatal(err)
}
p := provider.New(provider.Options{Node: "novox", Module: "keycloak", Bus: bus,
Vault: provider.ConsumersFile{Paths: []string{file}},
Offers: map[string]provider.Offer{ProvisionName: spec.Offer}, Make: run.Make, Gone: run.Gone,
Logf: func(string, ...any) {}})
out, err := p.Pass(ctx)
if err != nil || len(out) != 1 || out[0].Status != provider.Answered {
t.Fatalf("%v %+v", err, out)
}
run.retire(ctx)
if len(a.created) != 1 || a.created[0].As != "wiki-client" || a.created[0].Password != "pw-wiki" ||
a.created[0].Values["label"] != "Wiki" {
t.Fatalf("made %+v", a.created)
}
}
const testIssuer = "https://id.example.org/realms/Novox"
var (
grafanaContributes = map[string]any{"label": "grafana", "endpoint": "web", "callback": "/login/generic_oauth"}
grafanaNames = map[string]string{"name": "grafana.example.org", "internal-name": "grafana.home.internal"}
)
// keycloakSpec is what a keycloak-like provider says: the client id is the plan's `as`, the values are what the
// consumer contributes with the controller's names over them, and no grant file is consulted.
var keycloakSpec = MovedSpec{Module: "keycloak", Provision: ProvisionName,
Offer: provider.Offer{Credential: provider.PerConsumer, Serves: map[string]any{"issuer": testIssuer,
"token-path": "/protocol/openid-connect/token"}},
Login: func(e provider.Entry) string { return e.As },
Values: func(e provider.Entry) map[string]any {
values := map[string]any{}
for k, v := range e.Contributes {
values[k] = v
}
for k, name := range e.Names {
values[k] = name
}
return values
},
Noun: "client id", Made: "client", Backend: "Keycloak"}
// The consumers' file makes the same call on the adapter as the equivalent grant file does (the harness), for a
// provider of the keycloak shape: the same client id, secret and values; and the entry tells the consumer what the
// provider serves.
func TestTheConsumersFileMakesWhatTheGrantFileMakesForTheKeycloakShape(t *testing.T) {
grafana := provider.ConsumerIdentity("home", "grafana")
values := map[string]any{"name": grafanaNames["name"], "internal-name": grafanaNames["internal-name"]}
for k, v := range grafanaContributes {
values[k] = v
}
g := newWorld(t)
g.give(map[string]any{"as": grafana, "node": "home", "values": values})
g.h.Reconcile(ctx)
w := newMovedWorldOf(t, map[[2]string]string{{"home", "grafana"}: "pw-" + grafana}, keycloakSpec)
w.bus.bindNamed("keycloak", "home", "grafana", grafanaContributes, grafanaNames)
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
if len(w.a.created) != 1 || len(g.a.created) != 1 {
t.Fatalf("created %d through the helper, %d through the harness, want 1 each", len(w.a.created), len(g.a.created))
}
c, want := w.a.created[0], g.a.created[0]
if c.As != want.As || c.Password != want.Password || c.Consumer != want.Consumer ||
!reflect.DeepEqual(orEmpty(c.Values), orEmpty(want.Values)) {
t.Fatalf("made %+v, the grant file makes %+v", c, want)
}
var entry provider.Entry
if err := json.Unmarshal(w.bus.buckets[provider.BucketOf("keycloak")]["home.grafana."+ProvisionName], &entry); err != nil {
t.Fatal(err)
}
if entry.Values["issuer"] != testIssuer || entry.Values["token-path"] != "/protocol/openid-connect/token" {
t.Fatalf("served %v", entry.Values)
}
w.a.held = true
w.pass()
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 1 {
t.Fatalf("an unchanged pass created again: %d", len(w.a.created))
}
w.a.held = false
w.now = w.now.Add(2 * time.Minute)
w.pass()
if len(w.a.created) != 2 {
t.Fatalf("not made again: %d", len(w.a.created))
}
}
// A consumer whose plan names its client id (a slugged module) keeps it, and the retirement counts it as asked for.
func TestAConsumerNamedByItsPlanKeepsItsClientId(t *testing.T) {
agent := provider.ConsumerIdentity("home", "dashboard")
w := newMovedWorldOf(t, map[[2]string]string{{"home", "grafana-2"}: "pw"}, keycloakSpec)
plan, _ := json.Marshal(map[string]any{"node": "home", "module": "grafana-2", "provisioning": map[string]any{
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox", "provider-module": "keycloak",
"as": agent, "contributes": grafanaContributes, "names": grafanaNames}}}})
if err := w.bus.Put(ctx, provider.AssignmentsBucket, "home.grafana-2", plan); err != nil {
t.Fatal(err)
}
for _, o := range w.pass() {
if o.Status != provider.Answered {
t.Fatalf("%+v", o)
}
}
if len(w.a.created) != 1 || w.a.created[0].As != agent {
t.Fatalf("made %+v, want one under %s", w.a.created, agent)
}
for i := 0; i < 8; i++ {
w.pass()
}
w.now = w.now.Add(15 * time.Minute)
w.pass()
w.pass()
if len(w.a.removed) != 0 {
t.Fatalf("retired %v: the plan's client id was counted unasked", w.a.removed)
}
}
+623
View File
@@ -0,0 +1,623 @@
package harness
// What becomes of a consumer the mesh no longer asks for (novox/hq ADR 0230 — the operator's model,
// decided 2026-10-06; it replaces ADR 0229's withdrawal brake, which let one consumer go every hour).
//
// A consumer — a login, a client, a key, and the data behind it — is in one of three states:
//
// 1. ACTIVE.
// 2. RETIRED. The mesh stopped asking for it: its access is disabled, reversibly, and its login and
// data are marked "to delete" with when and why. Nothing is deleted. Asked for again, the ordinary
// create re-enables it at once, as it was.
// 3. DELETED, only through `cleanup delete`, a person's act, which this provider executes because it
// owns its backend.
//
// **Stable removals.** A consumer is retired only once the same set of consumers has gone unasked BOTH
// in StablePasses (5) consecutive passes that read the file AND for at least StableFor (10 minutes)
// since the first pass that saw it. Five passes alone are twenty-five seconds — shorter than a
// controller restart, a store reconnecting or a file half written. A pass that could not read the file
// is not a result and starts both again; so does a different set. Additions and changes are never
// delayed.
//
// **Too many is a person.** A stable set of more than RetireAtOnce (3), or — where more than one is
// held — of more than RetireFraction (half) of those held, retires nothing: it WAITS, said in the
// journal and announced `provisioner.retirement` `waiting` (again every SayAgainEvery), which the
// controller raises as an urgent condition, until `retire approve <node> <module>` or `retire reject`.
// An unassignment the controller made itself is no exception: many changes at once means a person is
// at work, and a person confirms once. On 2026-10-04 one misread file withdrew seven consumers at once
// (issue 241); under this rule that is a question, not an act.
//
// **The backend remembers, not this process.** What is retired, since when and why is read from the
// backend's own mark (Adapter.Inventory), so a restart forgets nothing; a consumer the backend holds
// active that the mesh no longer asks for is retired by the same rules after a restart as before one.
// A consumer found disabled without the mark — withdrawn before this record — is ADOPTED as retired:
// marked, said, announced `adopted`, and its clock starts then.
//
// **A rejection lives as long as this process.** Rejected, the set is kept active and not asked about
// again while it stays the same set; a restart asks again — loudly, never silently.
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// EventRetirement is the one event a provider says about retirement, its `change` saying what
// happened. The controller derives the permission to emit it for every module that receives
// contributions, as it does the standing events (novox/hq ADR 0224, 0230).
const EventRetirement = "provisioner.retirement"
// The changes EventRetirement carries.
const (
ChangeWaiting = "waiting" // a stable set over the bound waits for a person
ChangeSettled = "settled" // a waiting or rejected set ended without being retired
ChangeApproved = "approved" // a person approved the waiting (or rejected) set
ChangeRejected = "rejected" // a person kept the waiting set active
ChangeRetired = "retired" // consumers disabled and marked to delete
ChangeReenabled = "reenabled" // a retired consumer asked for again, enabled as it was
ChangeDeleted = "deleted" // a retired consumer deleted, by a person
ChangeAdopted = "adopted" // found disabled without the mark, now retired on record
)
// StableFor is the least time the same unasked set must hold before it is retired (novox/hq ADR 0230):
// longer than a controller restart, a store reconnecting or a file half written.
const StableFor = 10 * time.Minute
// KindConsumer is a retired consumer. A backend may list other things set aside for deletion under a
// word of its own (postgres: a database renamed aside).
const KindConsumer = "consumer"
// Retired is one thing a provider holds retired, as its backend's mark says.
type Retired struct {
Consumer string
// Node is the consumer's machine, where the mark records it.
Node string
// RetiredAt is when it was retired; zero for one found disabled without the mesh's mark.
RetiredAt time.Time
Why string
// SizeBytes is what deleting it would free; -1 when the backend cannot say.
SizeBytes int64
Kind string
}
// Inventory is what a backend holds that the mesh made.
type Inventory struct {
Active []string
Retired []Retired
// Sizes is what each active consumer keeps on the backend, in bytes, where the backend can say: what
// the controller tells an empty replacement of a consumer's data by (novox/hq ADR 0233).
Sizes map[string]int64
}
// retirement is the loop's memory of the sets it is counting, waiting on and was told to keep.
type retirement struct {
key string // the unasked set, joined
count int // consecutive passes that saw it
firstSeen time.Time
waiting *waitingSet
rejected *rejectedSet
retired map[string]retiredEntry
lastWant map[string]bool
seeded bool
seedSaid string
// noneSaid is whether this process has said that nothing waits: an earlier process may have left a set
// waiting that this one never knew (issue 937).
noneSaid bool
}
type waitingSet struct {
set []string
since time.Time
saidAt time.Time
held int
}
type rejectedSet struct {
set []string
by, why string
at time.Time
}
type retiredEntry struct {
node string
at time.Time
why string
}
// seed reads what the backend holds, once per process: an active consumer the mesh no longer asks
// for is held, so it is retired by the same rules as one this process made; one found disabled
// without the mark is adopted as retired.
func (h *Harness) seed(ctx context.Context, want map[string]bool) {
if h.r.seeded {
return
}
inv, err := h.Adapter.Inventory(ctx)
if err != nil {
if said := err.Error(); said != h.r.seedSaid {
h.say("cannot list what the backend holds (%v); a consumer the mesh stopped asking for while this "+
"provider was down is not retired until it can", err)
h.r.seedSaid = said
}
return
}
h.r.seeded = true
for _, as := range inv.Active {
if _, held := h.applied[as]; !held && !want[as] {
// No hash: asked for again, it is applied again, which is harmless and marks it.
h.applied[as] = appliedEntry{}
h.say("%s: held by the backend and no longer asked for; retired once that holds for %d passes "+
"and %s (novox/hq ADR 0230)", as, h.StablePasses, h.StableFor)
}
}
now := h.Now()
for _, r := range inv.Retired {
if r.Kind != "" && r.Kind != KindConsumer {
continue
}
if !r.RetiredAt.IsZero() {
h.r.retired[r.Consumer] = retiredEntry{node: r.Node, at: r.RetiredAt, why: r.Why}
continue
}
if want[r.Consumer] {
continue // asked for: this pass's create enables it
}
why := "found disabled without the mesh's mark — withdrawn before retirement was recorded " +
"(novox/hq ADR 0230); retired as found"
if err := h.Adapter.Retire(ctx, r.Consumer, nil, why, now); err != nil {
h.say("%s: found disabled and could not be marked retired, will try at the next start: %v", r.Consumer, err)
continue
}
h.r.retired[r.Consumer] = retiredEntry{node: r.Node, at: now, why: why}
h.say("%s: ADOPTED as retired — %s", r.Consumer, why)
h.announce(EventRetirement, h.retirementBody(ChangeAdopted, []map[string]any{
{"consumer": r.Consumer, "node": r.Node, "retired_at": stamp(now), "why": why, "size_bytes": r.SizeBytes},
}, map[string]any{"why": why}))
}
}
// retireUnasked counts the passes that saw the same set no longer asked for and, once it is stable,
// retires it — or, past the bound, waits for a person.
func (h *Harness) retireUnasked(ctx context.Context, want map[string]bool) {
var unasked []string
for as := range h.applied {
if !want[as] {
unasked = append(unasked, as)
}
}
sort.Strings(unasked)
key := strings.Join(unasked, "\x00")
now := h.Now()
switch {
case len(unasked) == 0:
h.settle("every consumer held is asked for again")
h.r.key, h.r.count = "", 0
return
case key != h.r.key:
if h.r.waiting != nil || h.r.rejected != nil {
h.settle("the set the mesh no longer asks for changed")
}
h.r.key, h.r.count, h.r.firstSeen = key, 1, now
h.say("no longer asked for: %s; retired once the same set holds for %d passes and %s",
strings.Join(unasked, ", "), h.StablePasses, h.StableFor)
default:
h.r.count++
}
if h.r.rejected != nil || h.r.count < h.StablePasses || now.Sub(h.r.firstSeen) < h.StableFor {
return
}
if h.overTheBound(len(unasked), len(h.applied)) {
h.wait(unasked)
return
}
why := fmt.Sprintf("the mesh stopped asking for it: the same result in %d consecutive passes over %s, from %s",
h.r.count, now.Sub(h.r.firstSeen).Round(time.Second), stamp(h.r.firstSeen))
h.retire(ctx, unasked, why, nil)
}
// overTheBound says retiring n of the held consumers at once needs a person.
func (h *Harness) overTheBound(n, held int) bool {
if n == 0 {
return false
}
return n > h.RetireAtOnce || (held > 1 && float64(n) > h.RetireFraction*float64(held))
}
func (h *Harness) bound(held int) string {
return fmt.Sprintf("more than %d at once, or more than %.0f%% of the %d held", h.RetireAtOnce,
h.RetireFraction*100, held)
}
// wait holds a stable set over the bound for a person, said once and announced again every
// SayAgainEvery so a controller that missed the first hears the next.
func (h *Harness) wait(set []string) {
now := h.Now()
w := h.r.waiting
if w == nil {
w = &waitingSet{set: set, since: now, held: len(h.applied)}
h.r.waiting = w
h.say("RETIREMENT WAITS FOR A PERSON: the mesh no longer asks for %d of the %d consumer(s) this provider "+
"holds (%s), %s. Nothing is retired; each stays active until `retire approve %s <module>` or "+
"`retire reject` (novox/hq ADR 0230)", len(set), w.held, strings.Join(set, ", "), h.bound(w.held), h.Node)
} else if now.Sub(w.saidAt) < h.SayAgainEvery {
return
}
w.saidAt = now
h.announce(EventRetirement, h.retirementBody(ChangeWaiting, h.named(set), map[string]any{
"held": w.held, "bound": h.bound(w.held), "since": stamp(w.since),
}))
}
// settle ends a waiting or rejected set that the mesh's own answer made moot.
func (h *Harness) settle(why string) {
var set []string
switch {
case h.r.waiting != nil:
set = h.r.waiting.set
case h.r.rejected != nil:
set = h.r.rejected.set
case h.r.noneSaid:
return
default:
// A restart forgets a set an earlier process waited on, and so would never say it ended: the
// controller's condition would ask a person to approve what nobody waits on any more (issue 937).
// The first time this process finds every consumer held asked for, it says so once, with no set.
h.r.noneSaid = true
h.say("nothing waits for a person: %s", why)
h.announce(EventRetirement, h.retirementBody(ChangeSettled, []map[string]any{}, map[string]any{"why": why}))
return
}
h.r.waiting, h.r.rejected = nil, nil
h.say("retirement of %s settled without retiring: %s", strings.Join(set, ", "), why)
h.announce(EventRetirement, h.retirementBody(ChangeSettled, h.named(set), map[string]any{"why": why}))
}
// retire disables and marks each consumer of set; one that fails stays held and is tried again once
// its set is stable again.
func (h *Harness) retire(ctx context.Context, set []string, why string, extra map[string]any) []string {
now := h.Now()
held := len(h.applied)
var done []string
var said []map[string]any
for _, as := range set {
was, ok := h.applied[as]
if !ok {
continue
}
if err := h.Adapter.Retire(ctx, as, was.derived, why, now); err != nil {
h.say("%s: retiring failed, will try again: %v", as, err)
continue
}
delete(h.applied, as)
delete(h.lost, as)
delete(h.failing, as)
h.r.retired[as] = retiredEntry{node: was.node, at: now, why: why}
h.recovered(as, "retired")
h.say("%s: RETIRED — its access disabled and its data kept, marked to delete (%s). Asked for again "+
"it is re-enabled as it was; it is deleted only by `cleanup delete` (novox/hq ADR 0230)", as, why)
done = append(done, as)
said = append(said, map[string]any{"consumer": as, "node": was.node, "retired_at": stamp(now), "why": why})
}
h.r.key, h.r.count, h.r.waiting, h.r.rejected = "", 0, nil, nil
if len(done) > 0 {
body := map[string]any{"held": held, "why": why}
for k, v := range extra {
body[k] = v
}
h.announce(EventRetirement, h.retirementBody(ChangeRetired, said, body))
}
return done
}
// reenabled says a retired consumer was asked for again and its create enabled it.
func (h *Harness) reenabled(as, node string) {
e, was := h.r.retired[as]
if !was {
return
}
delete(h.r.retired, as)
h.say("%s: asked for again — re-enabled as it was, its mark to delete cleared (retired %s: %s)", as,
stamp(e.at), e.why)
h.announce(EventRetirement, h.retirementBody(ChangeReenabled, []map[string]any{
{"consumer": as, "node": node, "retired_at": stamp(e.at), "why": e.why},
}, nil))
}
// Approve retires exactly the set waiting (or the set rejected) — a person's confirmation.
func (h *Harness) Approve(ctx context.Context, consumers []string, why, by, via string) ([]string, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
if strings.TrimSpace(why) == "" {
return nil, errors.New("approve: say why")
}
set := sorted(consumers)
var held []string
switch {
case h.r.waiting != nil && same(set, h.r.waiting.set):
held = h.r.waiting.set
case h.r.rejected != nil && same(set, h.r.rejected.set):
held = h.r.rejected.set
default:
return nil, fmt.Errorf("approve: %s is not the set waiting here — %s", orNone(set), h.waitingWords())
}
h.say("retirement of %s APPROVED by %s: %s", strings.Join(held, ", "), orSomebody(by), why)
h.announce(EventRetirement, h.retirementBody(ChangeApproved, h.named(held),
map[string]any{"why": why, "by": by, "via": via}))
reason := fmt.Sprintf("the mesh stopped asking for it; approved by %s: %s", orSomebody(by), why)
return h.retire(ctx, held, reason, map[string]any{"by": by, "via": via}), nil
}
// Reject keeps the waiting set active; it is not asked about again while it stays the same set.
func (h *Harness) Reject(consumers []string, why, by, via string) ([]string, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
if strings.TrimSpace(why) == "" {
return nil, errors.New("reject: say why")
}
set := sorted(consumers)
if h.r.waiting == nil || !same(set, h.r.waiting.set) {
return nil, fmt.Errorf("reject: %s is not the set waiting here — %s", orNone(set), h.waitingWords())
}
h.r.rejected = &rejectedSet{set: h.r.waiting.set, by: by, why: why, at: h.Now()}
h.r.waiting = nil
h.say("retirement of %s REJECTED by %s: %s. Kept active, and not asked about again while the mesh goes on "+
"not asking for exactly these (until this provider restarts)", strings.Join(set, ", "), orSomebody(by), why)
h.announce(EventRetirement, h.retirementBody(ChangeRejected, h.named(set),
map[string]any{"why": why, "by": by, "via": via}))
return set, nil
}
// DeleteRetired deletes one retired consumer, by a person's word: never an active one, never one the
// mesh asks for.
func (h *Harness) DeleteRetired(ctx context.Context, consumer, why, by, via string) (int64, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
if strings.TrimSpace(why) == "" {
return 0, errors.New("delete: say why")
}
if h.r.lastWant[consumer] {
return 0, fmt.Errorf("delete: the mesh asks for %s — it is not retired", consumer)
}
if _, held := h.applied[consumer]; held {
return 0, fmt.Errorf("delete: %s is active here — only a retired consumer is deleted", consumer)
}
inv, err := h.Adapter.Inventory(ctx)
if err != nil {
return 0, fmt.Errorf("delete: what the backend holds cannot be read, so nothing is deleted: %w", err)
}
var found *Retired
for i := range inv.Retired {
if inv.Retired[i].Consumer == consumer {
found = &inv.Retired[i]
}
}
if found == nil {
return 0, fmt.Errorf("delete: %s is not retired here — only a retired consumer is deleted", consumer)
}
freed, err := h.Adapter.Delete(ctx, *found)
if err != nil {
return 0, err
}
delete(h.r.retired, consumer)
h.say("%s: DELETED by %s: %s (retired %s: %s; %d bytes freed)", consumer, orSomebody(by), why,
stampOr(found.RetiredAt), found.Why, freed)
h.announce(EventRetirement, h.retirementBody(ChangeDeleted, []map[string]any{{
"consumer": consumer, "node": found.Node, "retired_at": stampOr(found.RetiredAt), "why": found.Why,
"size_bytes": found.SizeBytes, "kind": kindOf(*found),
}}, map[string]any{"why": why, "by": by, "via": via, "freed_bytes": freed}))
return freed, nil
}
// Retirement is what a person (and the controller's `cleanup list` and its probe) asks: what is
// held, waiting, rejected and retired here.
func (h *Harness) Retirement(ctx context.Context) (map[string]any, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.init()
inv, err := h.Adapter.Inventory(ctx)
if err != nil {
return nil, err
}
var held []string
for as := range h.applied {
held = append(held, as)
}
sort.Strings(held)
retired := []map[string]any{}
for _, r := range inv.Retired {
retired = append(retired, map[string]any{"consumer": r.Consumer, "node": r.Node,
"retired_at": stampOr(r.RetiredAt), "why": r.Why, "size_bytes": r.SizeBytes, "kind": kindOf(r)})
}
sizes := map[string]int64{}
for _, as := range held {
if size, ok := inv.Sizes[as]; ok && size >= 0 {
sizes[as] = size
}
}
out := map[string]any{"resource": h.Resource, "node": h.Node, "held": orEmptyList(held), "held_sizes": sizes,
"stable_passes": h.StablePasses, "stable_for_seconds": int(h.StableFor.Seconds()), "bound": h.bound(len(h.applied)), "waiting": nil, "rejected": nil,
"retired": retired}
if w := h.r.waiting; w != nil {
out["waiting"] = map[string]any{"consumers": h.named(w.set), "since": stamp(w.since), "held": w.held}
}
if r := h.r.rejected; r != nil {
out["rejected"] = map[string]any{"consumers": h.named(r.set), "by": r.by, "why": r.why, "at": stamp(r.at)}
}
return out, nil
}
func (h *Harness) waitingWords() string {
switch {
case h.r.waiting != nil:
return "waiting: " + strings.Join(h.r.waiting.set, ", ")
case h.r.rejected != nil:
return "nothing waits; rejected and kept: " + strings.Join(h.r.rejected.set, ", ")
}
return "nothing waits for a person here"
}
// named is a set with each consumer's machine, as the events and the tools say it.
func (h *Harness) named(set []string) []map[string]any {
out := make([]map[string]any, 0, len(set))
for _, as := range set {
out = append(out, map[string]any{"consumer": as, "node": h.applied[as].node})
}
return out
}
func (h *Harness) retirementBody(change string, consumers []map[string]any, extra map[string]any) map[string]any {
body := map[string]any{"provider": h.Resource, "provider-node": h.Node, "change": change,
"consumers": consumers, "at": stamp(h.Now())}
for k, v := range extra {
body[k] = v
}
return body
}
// RetirementTools are the four tools every provider serves for retirement, the same names in every
// module: the controller's `retire` and `cleanup` verbs ask them on the provider's machine.
func RetirementTools(h *Harness) []stdio.Tool {
if h == nil {
return nil
}
ask := func() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), 2*time.Minute)
}
who := map[string]any{
"why": map[string]any{"type": "string", "description": "why — required, kept in the hand-act log"},
"by": map[string]any{"type": "string", "description": "who decided"},
"via": map[string]any{"type": "string", "description": "mesh-controller when asked through its verbs"},
}
withSet := map[string]any{"consumers": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
"description": "the set, exactly as provisioner_retirement names it"}}
for k, v := range who {
withSet[k] = v
}
withOne := map[string]any{
"consumer": map[string]any{"type": "string", "description": "the retired consumer to delete"},
"confirm": map[string]any{"type": "string", "description": "the consumer's name again, to say this is meant"},
}
for k, v := range who {
withOne[k] = v
}
return []stdio.Tool{
{Name: "provisioner_retirement",
Description: "What this provider holds, what waits for a person to approve its retirement, what was rejected, " +
"and every retired consumer with when, why and its size (novox/hq ADR 0230).",
Run: func(map[string]any) (any, error) {
ctx, cancel := ask()
defer cancel()
return h.Retirement(ctx)
}},
{Name: "provisioner_retire_approve",
Description: "Retire exactly the set waiting for a person (or the set rejected earlier): access disabled, data " +
"kept and marked to delete. Use the controller's `retire approve`, which records the hand act.",
Input: withSet,
Run: func(args map[string]any) (any, error) {
ctx, cancel := ask()
defer cancel()
done, err := h.Approve(ctx, strs(args["consumers"]), argString(args, "why"), argString(args, "by"), argString(args, "via"))
if err != nil {
return nil, err
}
return map[string]any{"retired": orEmptyList(done)}, nil
}},
{Name: "provisioner_retire_reject",
Description: "Keep the set waiting for a person active. Use the controller's `retire reject`.",
Input: withSet,
Run: func(args map[string]any) (any, error) {
kept, err := h.Reject(strs(args["consumers"]), argString(args, "why"), argString(args, "by"), argString(args, "via"))
if err != nil {
return nil, err
}
return map[string]any{"kept": kept}, nil
}},
{Name: "provisioner_delete",
Description: "Delete one RETIRED consumer — its login and its data — for good. Refused for anything active or " +
"asked for. Use the controller's `cleanup delete`, which records the hand act.",
Input: withOne,
Run: func(args map[string]any) (any, error) {
consumer := argString(args, "consumer")
if consumer == "" || argString(args, "confirm") != consumer {
return nil, errors.New("delete: name the consumer, and repeat it in confirm")
}
ctx, cancel := ask()
defer cancel()
freed, err := h.DeleteRetired(ctx, consumer, argString(args, "why"), argString(args, "by"), argString(args, "via"))
if err != nil {
return nil, err
}
return map[string]any{"deleted": consumer, "freed_bytes": freed}, nil
}},
}
}
func strs(v any) []string {
list, _ := v.([]any)
out := []string{}
for _, x := range list {
if s, ok := x.(string); ok && s != "" {
out = append(out, s)
}
}
return out
}
func sorted(list []string) []string {
out := append([]string(nil), list...)
sort.Strings(out)
return out
}
func same(a, b []string) bool {
return strings.Join(sorted(a), "\x00") == strings.Join(sorted(b), "\x00")
}
func orNone(set []string) string {
if len(set) == 0 {
return "an empty set"
}
return strings.Join(set, ", ")
}
func orSomebody(by string) string {
if by == "" {
return "a person"
}
return by
}
func orEmptyList(list []string) []string {
if list == nil {
return []string{}
}
return list
}
func kindOf(r Retired) string {
if r.Kind == "" {
return KindConsumer
}
return r.Kind
}
func stamp(t time.Time) string { return t.UTC().Format(time.RFC3339) }
func stampOr(t time.Time) string {
if t.IsZero() {
return ""
}
return stamp(t)
}
func argString(args map[string]any, key string) string {
s, _ := args[key].(string)
return s
}
+571
View File
@@ -0,0 +1,571 @@
package harness
// Retirement (novox/hq ADR 0230), as the shared loop does it: a consumer the mesh stops asking for is
// retired only after the same result in five consecutive passes, too many at once wait for a person,
// asked for again it is re-enabled, and only a person deletes. The same file in every Go provider.
import (
"context"
"errors"
"fmt"
"os"
"strings"
"testing"
"time"
)
// recorder is a backend that remembers what it holds, active and retired, as a real one's mark does.
type recorder struct {
created []Provision
removed []string // retired, in order
deleted []string
held bool
failing error
holdsErr error
retErr error
inv Inventory
invErr error
}
func (r *recorder) Create(_ context.Context, p Provision) error {
if r.failing != nil {
return r.failing
}
r.created = append(r.created, p)
r.inv.Retired = withoutRetired(r.inv.Retired, p.As)
if !listHas(r.inv.Active, p.As) {
r.inv.Active = append(r.inv.Active, p.As)
}
return nil
}
func (r *recorder) Retire(_ context.Context, as string, _ map[string]any, why string, at time.Time) error {
if r.retErr != nil {
return r.retErr
}
r.removed = append(r.removed, as)
r.inv.Active = without(r.inv.Active, as)
r.inv.Retired = append(withoutRetired(r.inv.Retired, as),
Retired{Consumer: as, RetiredAt: at, Why: why, SizeBytes: 42, Kind: KindConsumer})
return nil
}
func (r *recorder) Holds(context.Context, Provision) (bool, error) {
if r.holdsErr != nil {
return false, r.holdsErr
}
return r.held, nil
}
func (r *recorder) Inventory(context.Context) (Inventory, error) { return r.inv, r.invErr }
func (r *recorder) Delete(_ context.Context, x Retired) (int64, error) {
r.deleted = append(r.deleted, x.Consumer)
r.inv.Retired = withoutRetired(r.inv.Retired, x.Consumer)
return x.SizeBytes, nil
}
func listHas(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
func without(list []string, s string) []string {
var out []string
for _, x := range list {
if x != s {
out = append(out, x)
}
}
return out
}
func withoutRetired(list []Retired, s string) []Retired {
var out []Retired
for _, x := range list {
if x.Consumer != s {
out = append(out, x)
}
}
return out
}
// holding gives the provider n consumers c1…cn and applies them.
func holding(w *world, n int) []map[string]any {
var given []map[string]any
for i := 1; i <= n; i++ {
given = append(given, map[string]any{"as": fmt.Sprintf("c%d", i), "node": "anchor"})
}
w.give(given...)
w.h.Reconcile(ctx)
return given
}
// pass is one reconcile five seconds after the last.
func (w *world) pass() {
w.now = w.now.Add(5 * time.Second)
w.h.Reconcile(ctx)
}
func retirements(said []announced) []string {
var out []string
for _, a := range said {
if a.event == EventRetirement {
out = append(out, a.body["change"].(string))
}
}
return out
}
func noneWaits(a announced) bool {
c, _ := a.body["consumers"].([]map[string]any)
return a.body["change"] == ChangeSettled && len(c) == 0
}
func lastRetirement(t *testing.T, said []announced) map[string]any {
t.Helper()
for i := len(said) - 1; i >= 0; i-- {
if said[i].event == EventRetirement {
return said[i].body
}
}
t.Fatal("nothing about retirement was announced")
return nil
}
func namesOf(body map[string]any) string {
var out []string
for _, c := range body["consumers"].([]map[string]any) {
out = append(out, c["consumer"].(string))
}
return strings.Join(out, ",")
}
// settle passes every five seconds until the same answer has held for StableFor, and one pass more.
func (w *world) settle() { w.passes(StableFor + 5*time.Second) }
func TestATransientEmptyListForFourPassesRetiresNothing(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 1)
w.give()
for i := 0; i < 4; i++ {
w.pass()
}
w.give(given...)
w.pass()
if len(w.a.removed) != 0 || len(retirements(*said)) != 0 {
t.Fatalf("four passes retired %v and said %v", w.a.removed, retirements(*said))
}
}
// Five identical passes are twenty-five seconds — shorter than a controller restart. Both must hold:
// five passes AND ten minutes of the same answer (novox/hq ADR 0230).
func TestFivePassesInTwentyFiveSecondsRetireNothingTenMinutesDo(t *testing.T) {
w, said := standingWorld(t)
holding(w, 1)
w.give()
for i := 0; i < 5; i++ {
w.pass()
}
if len(w.a.removed) != 0 || len(retirements(*said)) != 0 {
t.Fatalf("five passes in 25 s retired %v", w.a.removed)
}
w.passes(StableFor - 30*time.Second)
if len(w.a.removed) != 0 {
t.Fatalf("retired before the set held %s: %v", StableFor, w.a.removed)
}
w.passes(time.Minute)
if strings.Join(w.a.removed, ",") != "c1" {
t.Fatalf("the same set held %s and was not retired: %v", StableFor, w.a.removed)
}
// Ten minutes in one slow pass are not five passes.
w2 := newWorld(t)
holding(w2, 1)
w2.give()
w2.pass()
w2.now = w2.now.Add(StableFor)
w2.pass()
if len(w2.a.removed) != 0 {
t.Fatalf("two passes ten minutes apart retired %v", w2.a.removed)
}
w2.passes(15 * time.Second)
if len(w2.a.removed) != 1 {
t.Fatalf("five passes over ten minutes did not retire: %v", w2.a.removed)
}
}
func TestAStableSetIsRetiredAndAskedAgainReEnables(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 3)
w.give(given[:2]...)
w.settle()
if strings.Join(w.a.removed, ",") != "c3" {
t.Fatalf("retired %v", w.a.removed)
}
body := lastRetirement(t, *said)
if body["change"] != ChangeRetired || namesOf(body) != "c3" || body["held"] != 3 || body["provider-node"] != "anchor" ||
!strings.Contains(body["why"].(string), "consecutive passes over 10m") {
t.Fatalf("%v", body)
}
if len(w.a.inv.Retired) != 1 || w.a.inv.Retired[0].Consumer != "c3" {
t.Fatalf("the backend does not hold it retired: %+v", w.a.inv)
}
// Asked for again: the ordinary create, on the first pass, and said.
created := len(w.a.created)
w.give(given...)
w.pass()
if len(w.a.created) != created+1 || w.a.created[created].As != "c3" {
t.Fatalf("not created again: %v", w.a.created)
}
if body := lastRetirement(t, *said); body["change"] != ChangeReenabled || namesOf(body) != "c3" {
t.Fatalf("%v", body)
}
if len(w.a.inv.Retired) != 0 {
t.Fatalf("still marked retired: %+v", w.a.inv.Retired)
}
}
func TestAnUnreadablePassStartsTheCountAgain(t *testing.T) {
w := newWorld(t)
holding(w, 1)
w.give()
w.passes(StableFor - time.Minute)
os.WriteFile(w.receives, []byte("{"), 0o600)
w.pass()
w.give()
w.passes(StableFor - time.Minute)
if len(w.a.removed) != 0 {
t.Fatalf("an unreadable pass counted: %v", w.a.removed)
}
w.passes(2 * time.Minute)
if len(w.a.removed) != 1 {
t.Fatalf("not retired after ten minutes of readable passes: %v", w.a.removed)
}
}
func TestADifferentSetStartsTheCountAgain(t *testing.T) {
w := newWorld(t)
given := holding(w, 5)
w.give(given[:4]...)
w.passes(StableFor - time.Minute)
w.give(given[:3]...)
w.passes(StableFor - time.Minute)
if len(w.a.removed) != 0 {
t.Fatalf("a changed set kept its count: %v", w.a.removed)
}
w.passes(2 * time.Minute)
if strings.Join(w.a.removed, ",") != "c4,c5" {
t.Fatalf("%v", w.a.removed)
}
}
func TestAdditionsAndChangesAreNeverDelayed(t *testing.T) {
w := newWorld(t)
holding(w, 1)
w.give(map[string]any{"as": "c1", "node": "anchor"}, map[string]any{"as": "c2"})
w.pass()
if len(w.a.created) != 2 || w.a.created[1].As != "c2" {
t.Fatalf("an addition waited: %v", w.a.created)
}
w.give(map[string]any{"as": "c1", "node": "anchor", "values": map[string]any{"name": "rotated"}}, map[string]any{"as": "c2"})
w.pass()
if len(w.a.created) != 3 || w.a.created[2].As != "c1" {
t.Fatalf("a change waited: %v", w.a.created)
}
}
func TestTooManyWaitsForAPersonAndApproveRetiresExactlyThatSet(t *testing.T) {
w, said := standingWorld(t)
holding(w, 4)
w.give()
w.passes(15 * time.Minute)
if len(w.a.removed) != 0 {
t.Fatalf("four of four were retired without a person: %v", w.a.removed)
}
if got := strings.Join(retirements(*said), ","); got != ChangeWaiting {
t.Fatalf("said %q, want one waiting", got)
}
body := lastRetirement(t, *said)
if namesOf(body) != "c1,c2,c3,c4" || body["held"] != 4 || body["bound"] == "" {
t.Fatalf("%v", body)
}
if !strings.Contains(strings.Join(w.said, "\n"), "RETIREMENT WAITS FOR A PERSON") {
t.Fatal("the wait was not said")
}
// Said again every quarter of an hour while it waits.
w.passes(11 * time.Minute)
if got := strings.Join(retirements(*said), ","); got != "waiting,waiting" {
t.Fatalf("%q", got)
}
if _, err := w.h.Approve(ctx, []string{"c1", "c2"}, "tidy", "operator", "mesh-controller"); err == nil {
t.Fatal("approved a set that is not the one waiting")
}
if _, err := w.h.Approve(ctx, []string{"c4", "c3", "c2", "c1"}, "", "operator", ""); err == nil {
t.Fatal("approved without a why")
}
done, err := w.h.Approve(ctx, []string{"c4", "c3", "c2", "c1"}, "the old machine is gone", "operator", "mesh-controller")
if err != nil || strings.Join(done, ",") != "c1,c2,c3,c4" || strings.Join(w.a.removed, ",") != "c1,c2,c3,c4" {
t.Fatal(done, err, w.a.removed)
}
changes := retirements(*said)
if strings.Join(changes[len(changes)-2:], ",") != "approved,retired" {
t.Fatalf("%v", changes)
}
if b := lastRetirement(t, *said); b["by"] != "operator" || b["via"] != "mesh-controller" ||
!strings.Contains(b["why"].(string), "the old machine is gone") {
t.Fatalf("%v", b)
}
// Nothing more waits.
w.passes(time.Minute)
if r, _ := w.h.Retirement(ctx); r["waiting"] != nil || len(r["retired"].([]map[string]any)) != 4 {
t.Fatalf("%v", r)
}
}
func TestRejectedIsKeptAndNotAskedAgainUntilTheSetChanges(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 4)
w.give()
w.settle()
if _, err := w.h.Reject([]string{"c1"}, "no", "operator", ""); err == nil {
t.Fatal("rejected a set that is not the one waiting")
}
kept, err := w.h.Reject([]string{"c1", "c2", "c3", "c4"}, "a person is moving them", "operator", "mesh-controller")
if err != nil || len(kept) != 4 {
t.Fatal(kept, err)
}
w.passes(time.Hour)
if len(w.a.removed) != 0 {
t.Fatalf("a rejected set was retired: %v", w.a.removed)
}
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected" {
t.Fatalf("asked again after a rejection: %q", got)
}
r, _ := w.h.Retirement(ctx)
if r["rejected"] == nil || r["waiting"] != nil {
t.Fatalf("%v", r)
}
// One of them asked for again: a different answer, so the rejection is settled and counting
// starts over — three of four is over the bound again, and waits again.
w.give(given[0])
w.pass()
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected,settled" {
t.Fatalf("%q", got)
}
w.settle()
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected,settled,waiting" {
t.Fatalf("%q", got)
}
// A rejected set can still be approved later.
w2, _ := standingWorld(t)
holding(w2, 4)
w2.give()
w2.settle()
w2.h.Reject([]string{"c1", "c2", "c3", "c4"}, "wait", "operator", "")
if done, err := w2.h.Approve(ctx, []string{"c1", "c2", "c3", "c4"}, "now", "operator", ""); err != nil || len(done) != 4 {
t.Fatal(done, err)
}
}
func TestAWaitingSetAskedForAgainSettles(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 4)
w.give()
w.settle()
w.give(given...)
w.pass()
if got := strings.Join(retirements(*said), ","); got != "waiting,settled" || len(w.a.removed) != 0 {
t.Fatalf("%q %v", got, w.a.removed)
}
if _, err := w.h.Approve(ctx, []string{"c1", "c2", "c3", "c4"}, "late", "operator", ""); err == nil {
t.Fatal("approved a set that no longer waits")
}
}
func TestDeleteRemovesOnlyThatRetiredConsumer(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 5)
w.give(given[:3]...)
w.settle()
if strings.Join(w.a.removed, ",") != "c4,c5" {
t.Fatalf("%v", w.a.removed)
}
if _, err := w.h.DeleteRetired(ctx, "c1", "tidy", "operator", ""); err == nil {
t.Fatal("deleted an active consumer")
}
if _, err := w.h.DeleteRetired(ctx, "c5", "", "operator", ""); err == nil {
t.Fatal("deleted without a why")
}
if _, err := w.h.DeleteRetired(ctx, "nobody", "tidy", "operator", ""); err == nil {
t.Fatal("deleted something not retired")
}
freed, err := w.h.DeleteRetired(ctx, "c5", "the experiment is over", "operator", "mesh-controller")
if err != nil || freed != 42 || strings.Join(w.a.deleted, ",") != "c5" {
t.Fatal(freed, err, w.a.deleted)
}
if b := lastRetirement(t, *said); b["change"] != ChangeDeleted || namesOf(b) != "c5" || b["freed_bytes"] != int64(42) {
t.Fatalf("%v", b)
}
r, _ := w.h.Retirement(ctx)
if left := r["retired"].([]map[string]any); len(left) != 1 || left[0]["consumer"] != "c4" {
t.Fatalf("%v", r)
}
// Retired and asked for again before anybody deleted it: refused, it is the mesh's again.
w.give(given[:4]...)
w.pass()
if _, err := w.h.DeleteRetired(ctx, "c4", "tidy", "operator", ""); err == nil {
t.Fatal("deleted a consumer the mesh asks for")
}
}
func TestTheBound(t *testing.T) {
h := &Harness{}
h.init()
for _, c := range []struct{ n, held int }{{1, 1}, {1, 2}, {1, 3}, {3, 7}, {3, 6}, {2, 5}} {
if h.overTheBound(c.n, c.held) {
t.Errorf("%d of %d waits for a person", c.n, c.held)
}
}
for _, c := range []struct{ n, held int }{{2, 2}, {2, 3}, {4, 7}, {4, 10}, {7, 7}} {
if !h.overTheBound(c.n, c.held) {
t.Errorf("%d of %d is retired without a person", c.n, c.held)
}
}
}
func TestARestartRetiresWhatTheBackendHoldsUnaskedAndAdoptsWhatItFindsDisabled(t *testing.T) {
w, said := standingWorld(t)
w.a.inv = Inventory{Active: []string{"kept", "orphan"}, Retired: []Retired{
{Consumer: "locked-before", SizeBytes: 7, Kind: KindConsumer},
{Consumer: "old_deleted_20261005", RetiredAt: time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC), Kind: "set-aside-database"},
}}
w.give(map[string]any{"as": "kept"})
w.h.Reconcile(ctx)
if b := lastRetirement(t, *said); b["change"] != ChangeAdopted || namesOf(b) != "locked-before" {
t.Fatalf("%v", b)
}
if strings.Join(w.a.removed, ",") != "locked-before" {
t.Fatalf("adopting did not mark it: %v", w.a.removed)
}
w.settle()
if strings.Join(w.a.removed, ",") != "locked-before,orphan" {
t.Fatalf("an orphan the backend holds was not retired: %v", w.a.removed)
}
}
func TestABackendThatCannotBeListedIsSaidAndAskedAgain(t *testing.T) {
w := newWorld(t)
w.a.invErr = errors.New("connection refused")
holding(w, 1)
if !strings.Contains(strings.Join(w.said, "\n"), "cannot list what the backend holds") {
t.Fatal(w.said)
}
w.a.invErr = nil
w.a.inv = Inventory{Active: []string{"c1", "orphan"}}
w.pass()
w.settle()
if strings.Join(w.a.removed, ",") != "orphan" {
t.Fatalf("%v", w.a.removed)
}
}
func TestTheToolsAnswer(t *testing.T) {
w, _ := standingWorld(t)
holding(w, 4)
w.give()
w.settle()
tools := map[string]func(map[string]any) (any, error){}
for _, tool := range RetirementTools(w.h) {
tools[tool.Name] = tool.Run
}
if len(tools) != 4 {
t.Fatalf("%d tools", len(tools))
}
got, err := tools["provisioner_retirement"](nil)
if err != nil || got.(map[string]any)["waiting"] == nil {
t.Fatal(got, err)
}
set := []any{"c1", "c2", "c3", "c4"}
if _, err := tools["provisioner_retire_approve"](map[string]any{"consumers": set}); err == nil {
t.Fatal("approved without a why")
}
if _, err := tools["provisioner_retire_approve"](map[string]any{"consumers": set, "why": "gone", "by": "operator"}); err != nil {
t.Fatal(err)
}
if _, err := tools["provisioner_delete"](map[string]any{"consumer": "c1", "why": "gone"}); err == nil {
t.Fatal("deleted without confirm")
}
if _, err := tools["provisioner_delete"](map[string]any{"consumer": "c1", "confirm": "c1", "why": "gone"}); err != nil {
t.Fatal(err)
}
if strings.Join(w.a.deleted, ",") != "c1" {
t.Fatal(w.a.deleted)
}
}
func TestAFailingConsumerRetiredIsSaidRecovered(t *testing.T) {
w, said := standingWorld(t)
given := holding(w, 2)
w.a.held = false
w.a.failing = errors.New("boom")
w.passes(7 * time.Minute)
w.give(given[0])
w.settle()
recovered := false
for _, a := range *said {
if a.event == EventRecovered && a.body["consumer"] == "c2" && a.body["why"] == "retired" {
recovered = true
}
}
if !recovered {
t.Fatalf("%v", *said)
}
}
// everyWord is a world that keeps every word the provider announces.
func everyWord(t *testing.T) (*world, *[]announced) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) { said = append(said, announced{e, b}) }
w.h.Node = "anchor"
return w, &said
}
// A restarted provider that finds every consumer it holds asked for says once that nothing waits, so the controller
// clears a waiting condition an earlier process raised and this one never knew (issue 937); it says it no more.
func TestARestartedProviderSaysOnceThatNothingWaits(t *testing.T) {
w, said := everyWord(t)
holding(w, 2)
w.pass()
w.pass()
var words []announced
for _, a := range *said {
if a.event == EventRetirement {
words = append(words, a)
}
}
if len(words) != 1 || !noneWaits(words[0]) || words[0].body["provider-node"] != w.h.Node {
t.Fatalf("said %+v, want one settled with no set", words)
}
}
// A provider whose own set waits does not say that nothing waits.
func TestAProviderThatWaitsDoesNotSayNothingWaits(t *testing.T) {
w, said := everyWord(t)
w.a.inv.Active = []string{"c1", "c2", "c3", "c4", "c5"}
w.give(map[string]any{"as": "c1", "node": "anchor"})
w.settle()
for _, a := range *said {
if a.event == EventRetirement && noneWaits(a) {
t.Fatalf("said nothing waits while %v waited", w.h.r.waiting)
}
}
if w.h.r.waiting == nil {
t.Fatal("nothing waited: the test does not hold the case")
}
}
+193
View File
@@ -0,0 +1,193 @@
package harness
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224): not on the first
// failure, which may be a restart; after FailingAfter of failures with no success between; again
// every SayAgainEvery while it lasts; and recovered on the first success, or when the consumer goes.
import (
"errors"
"os"
"strings"
"testing"
"time"
)
type announced struct {
event string
body map[string]any
}
func standingWorld(t *testing.T) (*world, *[]announced) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) {
// The first success since start, and the first word that nothing waits (issue 937), are their own
// tests'; every other test reads past them.
if c, _ := b["consumers"].([]map[string]any); b["why"] != "first-success" &&
!(e == EventRetirement && b["change"] == ChangeSettled && len(c) == 0) {
said = append(said, announced{e, b})
}
}
w.h.Node = "anchor"
return w, &said
}
// passes reconciles every five seconds for d, as Run would.
func (w *world) passes(d time.Duration) {
for end := w.now.Add(d); w.now.Before(end); w.now = w.now.Add(5 * time.Second) {
w.h.Reconcile(ctx)
}
}
func events(said []announced) string {
var out []string
for _, a := range said {
out = append(out, a.event)
}
return strings.Join(out, ",")
}
func TestAConsumerFailedForMinutesIsAnnouncedNamingItAndTheClass(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New(`token request failed: 401 {"error":"invalid_grant","error_description":"Invalid user credentials"}`)
w.give(map[string]any{"as": "mesh_home_grafana", "node": "home-server"})
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("announced before FailingAfter: %v", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("want one %s, got %q", EventFailing, events(*said))
}
b := (*said)[0].body
if b["consumer"] != "mesh_home_grafana" || b["node"] != "home-server" || b["class"] != ClassCredentials ||
b["provider"] != "postgres-database" || b["provider-node"] != "anchor" || b["attempts"].(int) < 60 {
t.Fatalf("%v", b)
}
// Said again while it lasts, not every pass.
w.passes(14 * time.Minute)
if events(*said) != EventFailing {
t.Fatalf("repeated too soon: %q", events(*said))
}
w.passes(2 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("not repeated: %q", events(*said))
}
// The first success takes it back.
w.a.failing = nil
w.passes(5 * time.Second)
if events(*said) != EventFailing+","+EventFailing+","+EventRecovered {
t.Fatalf("no recovery: %q", events(*said))
}
if (*said)[2].body["consumer"] != "mesh_home_grafana" {
t.Fatal((*said)[2].body)
}
}
func TestOneSuccessBetweenFailuresStartsTheRunAgain(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("connection refused")
w.give(map[string]any{"as": "a"})
w.passes(4 * time.Minute)
w.a.failing = nil
w.passes(5 * time.Second)
w.give(map[string]any{"as": "a", "values": map[string]any{"name": "changed"}})
w.a.failing = errors.New("connection refused")
w.passes(4 * time.Minute)
if len(*said) != 0 {
t.Fatalf("two runs of four minutes are not one of eight: %q", events(*said))
}
}
// The check that failed for a day on 2026-10-05: clients already made, every minute's check refused
// at the token. A check that cannot be asked is a failure too.
func TestACheckThatKeepsFailingIsAFailureToo(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
w.h.Reconcile(ctx)
w.a.holdsErr = errors.New("401 invalid_grant")
w.passes(7 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassCredentials {
t.Fatalf("%q %v", events(*said), *said)
}
w.a.holdsErr = nil
w.passes(time.Minute + 5*time.Second)
if events(*said) != EventFailing+","+EventRecovered {
t.Fatalf("%q", events(*said))
}
}
func TestAnUnreadableSecretIsAnnouncedAsSuch(t *testing.T) {
w, said := standingWorld(t)
w.give(map[string]any{"as": "a"})
os.Remove(w.dir + "/a.secret")
w.passes(6 * time.Minute)
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassSecret {
t.Fatalf("%q %v", events(*said), *said)
}
}
func TestAConsumerNoLongerAskedForIsNoLongerFailing(t *testing.T) {
w, said := standingWorld(t)
w.a.failing = errors.New("boom")
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
w.passes(6 * time.Minute)
if events(*said) != EventFailing+","+EventFailing {
t.Fatalf("%q", events(*said))
}
w.give(map[string]any{"as": "a"})
w.passes(5 * time.Second)
last := (*said)[len(*said)-1]
if last.event != EventRecovered || last.body["consumer"] != "b" || last.body["why"] != "no longer asked for" {
t.Fatalf("%v", *said)
}
}
func TestAnErrorIsClassedByItsWords(t *testing.T) {
for text, want := range map[string]string{
`Keycloak token request failed: 401 {"error":"invalid_grant"}`: ClassCredentials,
`FATAL: password authentication failed for user "postgres"`: ClassCredentials,
`dial tcp 127.0.0.1:5432: connect: connection refused`: ClassUnreachable,
`context deadline exceeded`: ClassUnreachable,
`extension "nope" is not available`: ClassRefused,
} {
if got := ClassOf(text); got != want {
t.Errorf("%s: %s, want %s", text, got, want)
}
}
}
func TestAnAdapterThatClassesItsOwnErrorsIsBelieved(t *testing.T) {
w, said := standingWorld(t)
w.h.Adapter = classing{w.a}
w.a.failing = errors.New("anything")
w.give(map[string]any{"as": "a"})
w.passes(6 * time.Minute)
if (*said)[0].body["class"] != "its-own" {
t.Fatal((*said)[0].body)
}
}
type classing struct{ *recorder }
func (classing) Class(error) string { return "its-own" }
// A provider restarted after announcing a failure has forgotten it; its first success for each
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
func TestTheFirstSuccessSinceStartIsAnnouncedOnce(t *testing.T) {
w := newWorld(t)
var said []announced
w.h.Announce = func(e string, b map[string]any) {
if e != EventRetirement {
said = append(said, announced{e, b})
}
}
w.give(map[string]any{"as": "a"})
w.passes(3 * time.Minute)
if events(said) != EventRecovered || said[0].body["why"] != "first-success" || said[0].body["consumer"] != "a" {
t.Fatalf("%v", said)
}
}
+769
View File
@@ -0,0 +1,769 @@
// Package provider is a providing module's half of a binding: it answers the provider's consumers itself, over the
// bus (novox/hq ADR 0349 clause 12, issue 678): the provider module, not the controller, makes each bound
// consumer's entry.
//
// **Who binds the provider.** Each node's provisioner client records the needs of the modules handed to it from
// their plans, and the controller names in each plan, per required provision and local name, the provider that
// resolution chose: `provisioning.provisions` in `<node>.<module>` of the assignments mirror (issue 760). A recorded
// need names no provider (issue 677's note 7), so the plan's entry is where a bind is seen, as the vault sees it
// (mesh-vault pair.go). A consumer binds this provider when an entry names this machine and this module, and
// unbinds when the entry, or the assignment, goes. A consumer still filled the old way has no plan and is not seen
// here; the controller answers it as before (ADR 0349 clause 17).
//
// **One entry per consumer, in the provider's own bucket.** For each bind the helper composes the consumer's
// served values from the provider's offer (its config already composed into it, ADR 0202 §7) and the consumer's
// identity (`${consumer:as}`, `${consumer:as:dns}`, ADR 0202 §1), and writes one entry, `<node>.<module>.
// <provision>[.<local>]` of `<provider module>_bound`: the one statement both ends read. It never carries a
// secret. It names the credential the vault keeps for the pair, which the consumer's client asks the vault for.
//
// **The credential is the vault's** (ADR 0349 clause 8). The helper mints nothing and reads nothing of the
// provider's own secrets: it reads the value from the vault and hands it to the provider's adapter, which makes the
// login its software keeps. An offer whose software holds one credential (ADR 0158) is the vault's one value per
// provider and provision, `provision/<provider node>/<provision>`, with the provider's one user name; an offer with
// a login per consumer (ADR 0048, ADR 0158 §11, the default) is each consumer's own value,
// `pair/<provider node>/<provision>/<node>/<module>[/<local>]`, with the consumer's identity as its login (issue
// 677's notes 12 and 13). A credential the vault does not hold yet is waited for: nothing is made and no entry is
// written, so a consumer is never told of a login its provider has not made. A provider in the tool runner never asks
// the vault itself: its Vault is ConsumersFile, the file its machine's provisioner client fills (ADR 0373).
//
// **A multi-holder consumer gets no derived value** (ADR 0202 §10 as ADR 0349 clause 12 changes it): a consumer
// bound under a local name keeps several holders of the provision, each with its own login, while what the
// provider derives names one; its entry refuses it, naming both ends, and its client reports it.
//
// The helper speaks to the bus and the vault through the two interfaces below, so a provider wires it to its
// runtime and a test to a stand-in. Standard library only.
package provider
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// AssignmentsBucket is the controller's assignments mirror, `<node>.<module>`, whose value carries the plan of a
// module handed to the provisioner.
const AssignmentsBucket = "mesh-controller_assignments"
// BucketOf is a providing module's own bucket of entries, one per consumer: `<module>_bound`, the module's state
// `bound`.
func BucketOf(module string) string { return module + "_bound" }
// Bus is what the helper needs of the bus: a key-value watch and the provider's writes to its own bucket.
type Bus interface {
// Watch hands handle every current key and value of bucket, then each change as it comes, a removed key with a
// nil value; it returns once the current values are handed, and the watch lasts until ctx ends.
Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error
Put(ctx context.Context, bucket, key string, value []byte) error
Delete(ctx context.Context, bucket, key string) error
}
// ConditionalBus is a Bus that can also delete a key only while it still holds what the caller read. The helper uses
// it when the Bus offers it, so a pass that lets a consumer go never removes an entry another instance of the
// provider wrote after the pass read it; a Bus without it deletes unconditionally. DeleteIf returns an error that
// is stdio.ErrChanged when the key holds something else now.
type ConditionalBus interface {
Bus
DeleteIf(ctx context.Context, bucket, key string, held []byte) error
}
// Vault is how the provider reads a credential the vault keeps, by its name, opened.
type Vault interface {
Value(ctx context.Context, name string) ([]byte, error)
}
// CredentialForm is which credential an offer's consumers present.
type CredentialForm int
const (
// PerConsumer is a login of each consumer's own (ADR 0048, ADR 0158 §11): the default.
PerConsumer CredentialForm = iota
// Shared is the one credential the provider's software holds, shared with every consumer (ADR 0158).
Shared
// NoCredential is an offer whose consumers present none.
NoCredential
)
// Offer is what the provider serves one provision with.
type Offer struct {
// Serves are the values served to each consumer, the provider's config already composed into them; a string
// may name the consumer's identity, `${consumer:as}` or `${consumer:as:dns}`.
Serves map[string]any
// Credential is the form of the credential its consumers present. A consumer's plan entry that names a
// credential form of its own wins over it (the plan's `credential-form`).
Credential CredentialForm
// User is the provider's one user name, for a Shared credential.
User string
}
// Entry is one consumer's entry in the provider's bucket: the one statement both ends read. Never a secret.
type Entry struct {
Provider string `json:"provider"`
ProviderModule string `json:"provider-module"`
Provision string `json:"provision"`
Consumer string `json:"consumer"`
Module string `json:"module"`
Local string `json:"local,omitempty"`
// As is the consumer's identity, the mesh's (ADR 0202 §1).
As string `json:"as,omitempty"`
// User is the login the consumer presents: its identity, or the provider's one user name for a shared credential.
User string `json:"user,omitempty"`
// Values are the served values, composed for this consumer: what `${bound:<provision>:<key>}` reads.
Values map[string]any `json:"values,omitempty"`
// VaultName is the name under which the vault keeps this consumer's credential, which the consumer's client asks
// the vault for: a name, never the credential. Its key is not "credential" because the runtime's state guard
// refuses any key ending in that word, whatever the value (issue 945).
VaultName string `json:"vault-name,omitempty"`
// Refused, when set, is why the consumer is not served, naming both ends.
Refused string `json:"refused,omitempty"`
// Contributes is what the consumer's module tells its provider about this provision (a database's name, its
// extensions), from its plan's entry (issue 776). Handed to Make and Gone; never written to the bucket.
Contributes map[string]any `json:"-"`
// At and From are where the consumer's plan says the provider is and which machine it is (issue 788): the values
// today's binding file gives that consumer. Written into Values as `at` and `from`; never alone to the bucket.
At, From string `json:"-"`
// Names are the names the controller composed for this consumer on this provision (issue 792): its own name, its
// internal name, and one of each per local name it contributes under, keyed as its binding file names them.
// Written into Values over anything the provider serves under the key; never alone to the bucket.
Names map[string]string `json:"-"`
// CredentialForm and Name are from the consumer's plan entry (issue 825): the form of the credential the
// controller chose for this bind (`shared`, `pair`, `issued` or `given`) and the holder's name. When set, they
// win over the offer's form. Never written to the bucket.
CredentialForm, Name string `json:"-"`
}
// Key is the entry's key in the provider's bucket: `<node>.<module>.<provision>`, and `.<local>` for a bind under a
// local name.
func (e Entry) Key() string {
k := e.Consumer + "." + e.Module + "." + e.Provision
if e.Local != "" {
k += "." + e.Local
}
return k
}
// Consumer is what the provider's adapter makes: the consumer's entry and the credential the vault keeps for it.
type Consumer struct {
Entry Entry
Credential []byte
}
// Options are the provider's.
type Options struct {
// Node and Module are this provider: the machine it runs on and the module serving.
Node, Module string
// Offers are the provisions it serves, by name.
Offers map[string]Offer
Bus Bus
Vault Vault
// Make brings a consumer into being in the provider's software under the vault's credential. Asked on every
// pass for every consumer answered, so it must leave one that is already as asked alone; a rotated credential
// reaches the software this way.
Make func(ctx context.Context, c Consumer) error
// Gone, when set, hears that a consumer no longer binds the provider, before its entry is removed; an error
// keeps the entry, and it is heard again on the next pass. Retiring what was kept for it is the provider's
// (ADR 0230).
Gone func(ctx context.Context, e Entry) error
// Identity is the consumer's identity for an entry whose plan names none; ConsumerIdentity of the node and the
// module when not set.
Identity func(node, module string) string
// Every is how often a pass runs without a change watched; a minute when not set.
Every time.Duration
// Logf says what a pass did that differs from the last; the standard log when not set.
Logf func(format string, args ...any)
}
// Provider answers its consumers.
type Provider struct {
opts Options
mu sync.Mutex
assignments map[string][]byte
entries map[string][]byte
said map[string]string
poke chan struct{}
}
// New is a provider answering its consumers as opts say.
func New(opts Options) *Provider {
if opts.Identity == nil {
opts.Identity = ConsumerIdentity
}
if opts.Every <= 0 {
opts.Every = time.Minute
}
if opts.Logf == nil {
opts.Logf = func(f string, a ...any) { fmt.Printf("[provider] "+f+"\n", a...) }
}
return &Provider{opts: opts, assignments: map[string][]byte{}, entries: map[string][]byte{},
said: map[string]string{}, poke: make(chan struct{}, 1)}
}
// What a pass did for one bind.
const (
Answered = "answered"
Refused = "refused"
Waiting = "waiting"
Failed = "failed"
Removed = "removed"
// Superseded is an entry no bind asks for any more that another instance of the provider wrote after this pass
// read it: it is left standing.
Superseded = "superseded"
)
// Outcome is what a pass did for one consumer's entry.
type Outcome struct {
Key string `json:"key"`
Status string `json:"status"`
Why string `json:"why,omitempty"`
}
// Run watches the binds and the provider's own entries, and passes once at start, on every change and every
// Every, until ctx ends.
func (p *Provider) Run(ctx context.Context) error {
if err := p.watch(ctx, BucketOf(p.opts.Module), p.entries); err != nil {
return fmt.Errorf("watching %s: %w", BucketOf(p.opts.Module), err)
}
if err := p.watch(ctx, AssignmentsBucket, p.assignments); err != nil {
return fmt.Errorf("watching %s: %w", AssignmentsBucket, err)
}
tick := time.NewTicker(p.opts.Every)
defer tick.Stop()
for {
p.mu.Lock()
assignments, entries := clone(p.assignments), clone(p.entries)
p.mu.Unlock()
if _, err := p.pass(ctx, assignments, entries); err != nil {
p.say("pass", err.Error())
}
select {
case <-ctx.Done():
return ctx.Err()
case <-p.poke:
case <-tick.C:
}
}
}
func (p *Provider) watch(ctx context.Context, bucket string, into map[string][]byte) error {
return p.opts.Bus.Watch(ctx, bucket, func(key string, value []byte) {
p.mu.Lock()
if value == nil {
delete(into, key)
} else {
into[key] = append([]byte(nil), value...)
}
p.mu.Unlock()
if bucket == AssignmentsBucket {
select {
case p.poke <- struct{}{}:
default:
}
}
})
}
// Pass reads the binds and the provider's entries from the bus and answers once: each bind of this provider gets
// its entry, and each entry of this provider no bind asks for is removed. An assignment that cannot be read, or that
// is not the one its key names, is skipped and said (an outcome of status Failed under the assignment's key, and a
// line of the log), and the entries of its consumer are left as they are, since a bind missed would read as an
// unbind; the pass goes on for the rest. Run passes by itself; Pass is for a provider that passes when it chooses.
//
// Over StateBus each call to Pass leaves two watches open for the life of the process, silenced once the call has
// ended (see StateBus): a provider that passes often should use Run, which watches once.
func (p *Provider) Pass(ctx context.Context) ([]Outcome, error) {
watchCtx, cancel := context.WithCancel(ctx)
defer cancel()
var mu sync.Mutex
read := func(bucket string) (map[string][]byte, error) {
got := map[string][]byte{}
err := p.opts.Bus.Watch(watchCtx, bucket, func(key string, value []byte) {
mu.Lock()
defer mu.Unlock()
if value == nil {
delete(got, key)
} else {
got[key] = append([]byte(nil), value...)
}
})
mu.Lock()
defer mu.Unlock()
return clone(got), err
}
entries, err := read(BucketOf(p.opts.Module))
if err != nil {
return nil, fmt.Errorf("reading %s: %w", BucketOf(p.opts.Module), err)
}
assignments, err := read(AssignmentsBucket)
if err != nil {
return nil, fmt.Errorf("reading %s: %w", AssignmentsBucket, err)
}
return p.pass(ctx, assignments, entries)
}
// planned is as much of an assignment as the helper reads: lenient, since the plan is the client's to act on.
type planned struct {
Node string `json:"node"`
Module string `json:"module"`
Provisioning *struct {
Provisions []struct {
Provision string `json:"provision"`
Local string `json:"local"`
Provider string `json:"provider"`
ProviderModule string `json:"provider-module"`
Contributes map[string]any `json:"contributes"`
At string `json:"at"`
From string `json:"from"`
As string `json:"as"`
Names map[string]string `json:"names"`
CredentialForm string `json:"credential-form"`
Name string `json:"name"`
} `json:"provisions"`
} `json:"provisioning"`
}
func (p *Provider) pass(ctx context.Context, assignments, entries map[string][]byte) ([]Outcome, error) {
var binds []Entry
var out []Outcome
unknown := map[string]bool{} // consumers whose assignment was skipped: their entries are not judged
for _, key := range sortedKeys(assignments) {
if !Standing(assignments[key]) {
continue // ended or pending: read as a removed key (issue 1098)
}
var a planned
if err := json.Unmarshal(assignments[key], &a); err != nil || a.Node+"."+a.Module != key {
o := Outcome{Key: key, Status: Failed, Why: fmt.Sprintf("the assignment %s is not the controller's, so "+
"who it binds is not known; it is skipped, and its entries are left as they are", key)}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
unknown[key] = true
continue
}
if a.Provisioning == nil {
continue
}
for _, b := range a.Provisioning.Provisions {
if b.Provider != p.opts.Node || b.ProviderModule != p.opts.Module {
continue
}
binds = append(binds, Entry{Provider: p.opts.Node, ProviderModule: p.opts.Module,
Provision: b.Provision, Consumer: a.Node, Module: a.Module, Local: b.Local,
Contributes: b.Contributes, At: b.At, From: b.From, As: b.As, Names: b.Names,
CredentialForm: b.CredentialForm, Name: b.Name})
}
}
want := map[string]bool{}
for _, b := range binds {
key := b.Key()
want[key] = true
o := p.answer(ctx, b, entries[key])
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
}
for _, key := range sortedKeys(entries) {
if want[key] || unknown[consumerOf(key)] {
continue
}
var e Entry
if err := json.Unmarshal(entries[key], &e); err != nil || e.Provider != p.opts.Node ||
e.ProviderModule != p.opts.Module {
continue // another machine's instance of this provider wrote it, or nothing of this helper's
}
if p.opts.Gone != nil {
if err := p.opts.Gone(ctx, e); err != nil {
o := Outcome{Key: key, Status: Failed, Why: "no longer bound, and the provider could not let it go: " +
err.Error()}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
continue
}
}
if err := p.remove(ctx, key, entries[key]); err != nil {
if errors.Is(err, stdio.ErrChanged) {
o := Outcome{Key: key, Status: Superseded, Why: "another instance of the provider wrote its entry " +
"since this pass read it, so it stands"}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
continue
}
o := Outcome{Key: key, Status: Failed, Why: "no longer bound, and its entry could not be removed: " +
err.Error()}
out = append(out, o)
p.say(key, o.Status+" "+o.Why)
continue
}
out = append(out, Outcome{Key: key, Status: Removed})
p.say(key, Removed)
p.mu.Lock()
delete(p.said, key)
p.mu.Unlock()
}
return out, nil
}
// consumerOf is the `<node>.<module>` an entry's key starts with: its first two parts.
func consumerOf(key string) string {
parts := strings.SplitN(key, ".", 3)
if len(parts) < 3 {
return key
}
return parts[0] + "." + parts[1]
}
// remove deletes an entry this instance read as held, and only while it is still as read when the Bus can say so.
func (p *Provider) remove(ctx context.Context, key string, held []byte) error {
if c, ok := p.opts.Bus.(ConditionalBus); ok {
return c.DeleteIf(ctx, BucketOf(p.opts.Module), key, held)
}
return p.opts.Bus.Delete(ctx, BucketOf(p.opts.Module), key)
}
// answer makes one bind's entry and writes it when it differs from what the bucket holds.
func (p *Provider) answer(ctx context.Context, e Entry, have []byte) Outcome {
key := e.Key()
offer, offered := p.opts.Offers[e.Provision]
switch {
case !offered:
e.Refused = fmt.Sprintf("%s on %s is bound to %s on %s for %s, which %s does not serve", e.Module,
e.Consumer, e.ProviderModule, e.Provider, e.Provision, e.ProviderModule)
case e.Local != "" && derives(offer.Serves):
e.Refused = fmt.Sprintf("%s on %s keeps several holders of %s (this one is %q), and %s on %s derives %s for "+
"each consumer from its login: each holder has a login of its own and a consumer is told one value per "+
"requirement, so the two ends would name different things (novox/hq ADR 0202 §10, ADR 0349 clause 12)",
e.Module, e.Consumer, e.Provision, e.Local, e.ProviderModule, e.Provider,
strings.Join(derivedKeys(offer.Serves), ", "))
}
if e.Refused != "" {
return p.write(ctx, e, have, Outcome{Key: key, Status: Refused, Why: e.Refused})
}
// The plan's identity is the controller's, slug first; Identity only answers for an entry without one.
if e.As == "" {
e.As = p.opts.Identity(e.Consumer, e.Module)
}
values, err := ServedTo(offer.Serves, e.As)
if err != nil {
e.Refused = fmt.Sprintf("%s on %s serving %s to %s on %s: %v", e.ProviderModule, e.Provider, e.Provision,
e.Module, e.Consumer, err)
e.As = ""
return p.write(ctx, e, have, Outcome{Key: key, Status: Refused, Why: e.Refused})
}
if e.At != "" || e.From != "" {
if values == nil {
values = map[string]any{}
}
if e.At != "" {
values["at"] = e.At
}
if e.From != "" {
values["from"] = e.From
}
}
if len(e.Names) > 0 {
if values == nil {
values = map[string]any{}
}
for key, name := range e.Names {
values[key] = name
}
}
e.Values = values
var credential []byte
// The plan's form wins; the offer's is the default.
// An offer that declares no credential has none whatever the plan says: the controller writes "pair" for it.
form := e.CredentialForm
if offer.Credential == NoCredential {
form = ""
} else if form == "" {
form = map[CredentialForm]string{Shared: "shared", PerConsumer: "pair"}[offer.Credential]
}
switch form {
case "shared":
e.User = offer.User
e.VaultName = SharedName(e.Provider, e.Provision)
case "pair":
e.User = e.As
e.VaultName = PairName(e.Provider, e.Provision, e.Consumer, e.Module, e.Local)
case "issued", "given":
// A per-holder value, as mesh-vault names it (perholder.go).
e.User = e.As
e.VaultName = HolderName(e.Provider, e.Provision, e.Consumer, e.Module)
}
if e.VaultName != "" {
credential, err = p.opts.Vault.Value(ctx, e.VaultName)
if err != nil {
return Outcome{Key: key, Status: Waiting, Why: fmt.Sprintf("the vault's %s: %v", e.VaultName, err)}
}
}
if p.opts.Make != nil {
if err := p.opts.Make(ctx, Consumer{Entry: e, Credential: credential}); err != nil {
why := err.Error()
why = redact(why, credential)
return Outcome{Key: key, Status: Failed, Why: "the provider could not make it: " + why}
}
}
return p.write(ctx, e, have, Outcome{Key: key, Status: Answered})
}
// redact takes a credential out of text in the forms it can come back in: as it is, in base64 (standard and URL,
// padded and not) and as a JSON string says it (escapes included).
func redact(text string, credential []byte) string {
if len(credential) == 0 {
return text
}
quoted, _ := json.Marshal(string(credential))
forms := []string{string(credential), base64.StdEncoding.EncodeToString(credential),
base64.RawStdEncoding.EncodeToString(credential), base64.URLEncoding.EncodeToString(credential),
base64.RawURLEncoding.EncodeToString(credential), string(quoted[1 : len(quoted)-1])}
sort.SliceStable(forms, func(i, j int) bool { return len(forms[i]) > len(forms[j]) })
for _, f := range forms {
text = strings.ReplaceAll(text, f, "<credential>")
}
return text
}
func (p *Provider) write(ctx context.Context, e Entry, have []byte, o Outcome) Outcome {
body, err := json.Marshal(e)
if err != nil {
return Outcome{Key: o.Key, Status: Failed, Why: err.Error()}
}
if sameJSON(have, body) {
return o
}
if err := p.opts.Bus.Put(ctx, BucketOf(p.opts.Module), o.Key, body); err != nil {
return Outcome{Key: o.Key, Status: Failed, Why: "its entry could not be written: " + err.Error()}
}
return o
}
// say logs what a key's pass said when it differs from the last.
func (p *Provider) say(key, what string) {
p.mu.Lock()
same := p.said[key] == what
p.said[key] = what
p.mu.Unlock()
if !same {
p.opts.Logf("%s: %s", key, strings.TrimSpace(what))
}
}
// SharedName is the vault's one value per provider and provision (ADR 0158, issue 677).
func SharedName(provider, provision string) string { return "provision/" + provider + "/" + provision }
// HolderName is the vault's value of one holder's own credential, issued or given for it (mesh-vault perholder.go).
func HolderName(provider, provision, node, module string) string {
return "provision/" + provider + "/" + provision + "/" + node + "/" + module
}
// PairName is the vault's value of one consumer's own login (ADR 0048, ADR 0158 §11, issue 677's note 12).
func PairName(provider, provision, node, module, local string) string {
n := "pair/" + provider + "/" + provision + "/" + node + "/" + module
if local != "" {
n += "/" + local
}
return n
}
// identityUnusable is every character a consumer's identity does not hold (mesh-controller internal/catalogue
// identity.go, the one derivation).
var identityUnusable = regexp.MustCompile(`[^a-z0-9_]+`)
// ConsumerIdentity is what one module on one machine is called wherever it authenticates: `mesh_<node>_<module>`,
// lower case, anything but letters, digits and `_` written `_` (mesh-controller ConsumerIdentity). module is the
// identity source: the module's slug when it declares one, otherwise its name (ADR 0049).
func ConsumerIdentity(node, module string) string {
clean := func(s string) string {
return strings.Trim(identityUnusable.ReplaceAllString(strings.ToLower(s), "_"), "_")
}
return "mesh_" + clean(node) + "_" + clean(module)
}
// consumerFact is `${consumer:<fact>}` or `${consumer:<fact>:<alphabet>}`.
var consumerFact = regexp.MustCompile(`\$\{consumer:([a-z][a-z0-9-]*)(?::([a-z][a-z0-9-]*))?\}`)
// ServedTo fills an offer's served values for one consumer whose identity is as (ADR 0202 §1): `${consumer:as}` as
// it is, `${consumer:as:dns}` with `-` for `_`. Only strings carry placeholders; any other fact or alphabet is
// refused, never left standing.
func ServedTo(serves map[string]any, as string) (map[string]any, error) {
if serves == nil {
return nil, nil
}
out := make(map[string]any, len(serves))
for _, key := range sortedKeys(serves) {
text, ok := serves[key].(string)
if !ok || !strings.Contains(text, "${consumer:") {
out[key] = serves[key]
continue
}
var failed error
out[key] = consumerFact.ReplaceAllStringFunc(text, func(match string) string {
parts := consumerFact.FindStringSubmatch(match)
switch {
case parts[1] == "as" && parts[2] == "":
return as
case parts[1] == "as" && parts[2] == "dns":
return strings.ReplaceAll(as, "_", "-")
}
if failed == nil {
failed = fmt.Errorf("the value served as %q says %s, and the mesh states only ${consumer:as} and "+
"${consumer:as:dns}", key, match)
}
return match
})
if failed != nil {
return nil, failed
}
}
return out, nil
}
func derives(serves map[string]any) bool { return len(derivedKeys(serves)) > 0 }
func derivedKeys(serves map[string]any) []string {
var out []string
for _, k := range sortedKeys(serves) {
if text, ok := serves[k].(string); ok && strings.Contains(text, "${consumer:") {
out = append(out, k)
}
}
return out
}
func clone(m map[string][]byte) map[string][]byte {
out := make(map[string][]byte, len(m))
for k, v := range m {
out[k] = v
}
return out
}
func sameJSON(a, b []byte) bool {
if a == nil || b == nil {
return false
}
var x, y any
if json.Unmarshal(a, &x) != nil || json.Unmarshal(b, &y) != nil {
return false
}
ja, _ := json.Marshal(x)
jb, _ := json.Marshal(y)
return string(ja) == string(jb)
}
func sortedKeys[V any](m map[string]V) []string {
if len(m) == 0 {
return nil
}
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// StateBus is the Bus of a provider that runs as a module bundle over the runtime's stdio: the provider's own bucket,
// BucketOf(module), is the module's state `bound`, and AssignmentsBucket is the state it reads,
// `mesh-controller.assignments` (both declared in the module's manifest). Any other bucket is refused.
//
// A stdio state watch lasts as long as the process and cannot be ended (stdio.KeptState.Watch), so Watch honours the
// Bus contract by silencing instead: once ctx has ended the handler is never called again, though the runtime keeps
// sending changes that are dropped. A Pass, which watches twice per call, is therefore correct, only the dropped
// changes of its two spent watches remain.
func StateBus(module string) Bus { return stateBus{module: module} }
type stateBus struct{ module string }
func (b stateBus) state(bucket string) (stdio.KeptState, error) {
switch bucket {
case BucketOf(b.module):
return stdio.State("bound"), nil
case AssignmentsBucket:
return stdio.State("mesh-controller.assignments"), nil
}
return stdio.KeptState{}, fmt.Errorf("%s is not a bucket this provider keeps or reads", bucket)
}
func (b stateBus) Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error {
s, err := b.state(bucket)
if err != nil {
return err
}
return s.Watch("", func(ch stdio.StateChange) error {
if ctx.Err() != nil {
return nil
}
if ch.Op == "delete" {
handle(ch.Key, nil)
} else {
handle(ch.Key, ch.Value)
}
return nil
})
}
func (b stateBus) Put(_ context.Context, bucket, key string, value []byte) error {
s, err := b.state(bucket)
if err != nil {
return err
}
_, err = s.Put(key, json.RawMessage(value))
return err
}
// DeleteIf removes the key only while it still holds what the caller read; it reads the key first, so another
// writer's change in that short while can still be lost.
func (b stateBus) DeleteIf(_ context.Context, bucket, key string, held []byte) error {
s, err := b.state(bucket)
if err != nil {
return err
}
cur, err := s.Get(key)
if err != nil {
return err
}
if cur == nil {
return nil
}
if !sameJSON(cur.Value, held) {
return fmt.Errorf("%s is not as it was read: %w", key, stdio.ErrChanged)
}
return s.Delete(key)
}
func (b stateBus) Delete(_ context.Context, bucket, key string) error {
s, err := b.state(bucket)
if err != nil {
return err
}
return s.Delete(key)
}
// Standing is whether an assignment's value is a standing assignment: false when `ended` is set (the module was
// taken off its node, novox/hq ADR 0408 clause 4) or `state` is not empty (a pending one). Such a value is read
// exactly as a removed key. Every other field, known or not, changes nothing (issue 1098).
func Standing(value []byte) bool {
var v struct {
Ended json.RawMessage `json:"ended"`
State json.RawMessage `json:"state"`
}
if json.Unmarshal(value, &v) != nil {
return true // not decided here: the parse that follows refuses it
}
return !fieldSet(v.Ended) && !fieldSet(v.State)
}
// fieldSet is whether a field carries something: not absent, null or empty.
func fieldSet(f json.RawMessage) bool {
s := strings.TrimSpace(string(f))
return s != "" && s != "null" && s != `""`
}
+803
View File
@@ -0,0 +1,803 @@
package provider
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"reflect"
"slices"
"strings"
"sync"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// standIn is a stand-in bus: buckets of keys and values, and watches that hand the current values first and then
// every change, as the bus's key-value watch does.
type standIn struct {
mu sync.Mutex
buckets map[string]map[string][]byte
watchers map[string][]func(string, []byte)
}
func newStandIn() *standIn {
return &standIn{buckets: map[string]map[string][]byte{}, watchers: map[string][]func(string, []byte){}}
}
func (s *standIn) Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error {
s.mu.Lock()
current := map[string][]byte{}
for k, v := range s.buckets[bucket] {
current[k] = v
}
s.watchers[bucket] = append(s.watchers[bucket], handle)
s.mu.Unlock()
for _, k := range sortedKeys(current) {
handle(k, current[k])
}
return nil
}
func (s *standIn) Put(_ context.Context, bucket, key string, value []byte) error {
s.mu.Lock()
if s.buckets[bucket] == nil {
s.buckets[bucket] = map[string][]byte{}
}
s.buckets[bucket][key] = append([]byte(nil), value...)
ws := slices.Clone(s.watchers[bucket])
s.mu.Unlock()
for _, w := range ws {
w(key, value)
}
return nil
}
func (s *standIn) Delete(_ context.Context, bucket, key string) error {
s.mu.Lock()
delete(s.buckets[bucket], key)
ws := slices.Clone(s.watchers[bucket])
s.mu.Unlock()
for _, w := range ws {
w(key, nil)
}
return nil
}
// DeleteIf removes the key only while it still holds what the caller read.
func (s *standIn) DeleteIf(ctx context.Context, bucket, key string, held []byte) error {
s.mu.Lock()
cur := s.buckets[bucket][key]
s.mu.Unlock()
if !bytes.Equal(cur, held) {
return fmt.Errorf("%s: %w", key, stdio.ErrChanged)
}
return s.Delete(ctx, bucket, key)
}
func (s *standIn) get(bucket, key string) []byte {
s.mu.Lock()
defer s.mu.Unlock()
return s.buckets[bucket][key]
}
func (s *standIn) keys(bucket string) []string {
s.mu.Lock()
defer s.mu.Unlock()
return sortedKeys(s.buckets[bucket])
}
// everything is every value the bus holds, for a test that no secret is in a bucket.
func (s *standIn) everything() []byte {
s.mu.Lock()
defer s.mu.Unlock()
var all []byte
for _, b := range s.buckets {
for k, v := range b {
all = append(all, k...)
all = append(all, v...)
}
}
return all
}
// vault is a stand-in vault: a value per name, or nothing yet.
type vault struct {
mu sync.Mutex
values map[string]string
asked []string
}
func (v *vault) Value(_ context.Context, name string) ([]byte, error) {
v.mu.Lock()
defer v.mu.Unlock()
v.asked = append(v.asked, name)
val, ok := v.values[name]
if !ok {
return nil, fmt.Errorf("the vault holds no %s yet", name)
}
return []byte(val), nil
}
// maker records what the provider's adapter was asked to make, with the credential it was handed.
type maker struct {
mu sync.Mutex
made map[string]string
// contributed is what each consumer's entry carried of its plan's contribution.
contributed map[string]map[string]any
// as is the identity each consumer's entry was made under.
as map[string]string
gone []string
fail error
calls int
}
func (m *maker) Make(_ context.Context, c Consumer) error {
m.mu.Lock()
defer m.mu.Unlock()
m.calls++
if m.fail != nil {
return m.fail
}
if m.made == nil {
m.made = map[string]string{}
}
m.made[c.Entry.Key()] = string(c.Credential)
if m.contributed == nil {
m.contributed = map[string]map[string]any{}
}
m.contributed[c.Entry.Key()] = c.Entry.Contributes
if m.as == nil {
m.as = map[string]string{}
}
m.as[c.Entry.Key()] = c.Entry.As
return nil
}
func (m *maker) Gone(_ context.Context, e Entry) error {
m.mu.Lock()
defer m.mu.Unlock()
m.gone = append(m.gone, e.Key())
return nil
}
// assign puts a consumer's assignment with its plan's provisions into the stand-in mirror.
func assign(t *testing.T, bus *standIn, node, module string, provisions ...map[string]string) {
t.Helper()
ps := []map[string]string{}
ps = append(ps, provisions...)
raw, err := json.Marshal(map[string]any{"node": node, "module": module, "assigned": "2026-10-15T00:00:00Z",
"provisioning": map[string]any{"templates": map[string]string{"source": "x", "digest": "y"},
"provisions": ps}})
if err != nil {
t.Fatal(err)
}
if err := bus.Put(context.Background(), AssignmentsBucket, node+"."+module, raw); err != nil {
t.Fatal(err)
}
}
func bind(provision, local, provider, providerModule string) map[string]string {
b := map[string]string{"provision": provision, "provider": provider, "provider-module": providerModule}
if local != "" {
b["local"] = local
}
return b
}
func entryOf(t *testing.T, bus *standIn, key string) Entry {
t.Helper()
raw := bus.get(BucketOf("postgres"), key)
if raw == nil {
t.Fatalf("no entry %s in %s; it holds %v", key, BucketOf("postgres"), bus.keys(BucketOf("postgres")))
}
var e Entry
if err := json.Unmarshal(raw, &e); err != nil {
t.Fatalf("entry %s: %v", key, err)
}
return e
}
// postgres is a provider of postgres-database on novox with a login per consumer (ADR 0048, ADR 0158 §11), its
// served values derived from each consumer's identity (ADR 0202 §1).
func postgres(bus *standIn, v *vault, m *maker) *Provider {
return New(Options{
Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Gone: m.Gone,
Offers: map[string]Offer{
"postgres-database": {Credential: PerConsumer, Serves: map[string]any{
"host": "postgres.novox.mesh", "port": 5432, "database": "${consumer:as}"}},
},
Logf: func(string, ...any) {},
})
}
// A provider answers each consumer that binds it with an entry of its own in its own bucket: composed from its
// served values and the consumer's identity, naming the credential the vault keeps for that consumer, and never
// the credential itself; its adapter is handed each consumer's own value read from the vault.
func TestEachConsumerGetsAnEntryOfItsOwnAndTheVaultsValue(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/letta": "letta-secret-1",
"pair/novox/postgres-database/g14/umami": "umami-secret-2",
}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.postgres-database",
"g14.umami.postgres-database"}) {
t.Fatalf("entries %v", got)
}
e := entryOf(t, bus, "ace.letta.postgres-database")
if e.Provider != "novox" || e.ProviderModule != "postgres" || e.Consumer != "ace" || e.Module != "letta" ||
e.As != "mesh_ace_letta" || e.User != "mesh_ace_letta" ||
e.VaultName != "pair/novox/postgres-database/ace/letta" || e.Refused != "" {
t.Fatalf("entry %+v", e)
}
if e.Values["database"] != "mesh_ace_letta" || e.Values["host"] != "postgres.novox.mesh" ||
e.Values["port"] != float64(5432) {
t.Fatalf("values %v", e.Values)
}
if m.made["ace.letta.postgres-database"] != "letta-secret-1" ||
m.made["g14.umami.postgres-database"] != "umami-secret-2" {
t.Fatalf("made %v", m.made)
}
if bytes.Contains(bus.everything(), []byte("secret-")) {
t.Fatal("a credential is in a bucket")
}
for _, o := range out {
if o.Status != Answered {
t.Fatalf("outcome %+v", o)
}
}
}
// A provider whose offer shares one credential (ADR 0158) names the vault's one value for the provision, and its
// one user name, to every consumer.
func TestASharedCredentialIsTheVaultsOneValueAndTheProvidersOneUser(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"provision/novox/sonarr-api": "one-value"}}
assign(t, bus, "ace", "overseerr", bind("sonarr-api", "", "novox", "sonarr"))
assign(t, bus, "g14", "homepage", bind("sonarr-api", "", "novox", "sonarr"))
p := New(Options{Node: "novox", Module: "sonarr", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"sonarr-api": {Credential: Shared, User: "sonarr",
Serves: map[string]any{"url": "http://sonarr.novox.mesh:8989"}}}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
for _, key := range []string{"ace.overseerr.sonarr-api", "g14.homepage.sonarr-api"} {
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("sonarr"), key), &e); err != nil {
t.Fatalf("%s: %v", key, err)
}
if e.VaultName != "provision/novox/sonarr-api" || e.User != "sonarr" || e.Values["url"] == nil {
t.Fatalf("%s: %+v", key, e)
}
if m.made[key] != "one-value" {
t.Fatalf("%s made with %q", key, m.made[key])
}
}
if bytes.Contains(bus.everything(), []byte("one-value")) {
t.Fatal("the shared credential is in a bucket")
}
}
// A consumer that keeps several holders of one provision is served no derived value: its entry refuses it, naming
// both ends, so its client can report it (ADR 0349 clause 12, ADR 0202 §10); nothing is made for it.
func TestAMultiHolderConsumerIsRefusedADerivedValueNamingBothEnds(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta/archive": "x-secret"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "archive", "novox", "postgres"))
out, err := postgres(bus, v, m).Pass(context.Background())
if err != nil {
t.Fatal(err)
}
e := entryOf(t, bus, "ace.letta.postgres-database.archive")
for _, says := range []string{"letta on ace", "postgres on novox", "postgres-database", "archive"} {
if !strings.Contains(e.Refused, says) {
t.Fatalf("the refusal does not name %q: %s", says, e.Refused)
}
}
if e.Values != nil || e.VaultName != "" {
t.Fatalf("a refused entry carries values: %+v", e)
}
if m.calls != 0 {
t.Fatal("the adapter made a refused consumer")
}
if len(out) != 1 || out[0].Status != Refused {
t.Fatalf("outcomes %+v", out)
}
}
// A consumer bound under a local name is served when the offer derives nothing from it: its credential is its own
// pair row under that local name.
func TestALocalNameIsServedWhenNothingIsDerived(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/redis-cache/ace/letta/queue": "q-secret"}}
assign(t, bus, "ace", "letta", bind("redis-cache", "queue", "novox", "redis"))
p := New(Options{Node: "novox", Module: "redis", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"redis-cache": {Credential: PerConsumer, Serves: map[string]any{"port": 6379}}}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("redis"), "ace.letta.redis-cache.queue"), &e); err != nil {
t.Fatal(err)
}
if e.Refused != "" || e.VaultName != "pair/novox/redis-cache/ace/letta/queue" || e.Local != "queue" {
t.Fatalf("entry %+v", e)
}
if m.made["ace.letta.redis-cache.queue"] != "q-secret" {
t.Fatalf("made %v", m.made)
}
}
// A consumer that unbinds loses its entry and the adapter hears it is gone; an entry another machine's instance of
// the same provider wrote is not this one's to remove.
func TestAnUnbindRemovesTheEntryAndLeavesAnotherProvidersAlone(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
other, _ := json.Marshal(Entry{Provider: "shanks", ProviderModule: "postgres", Provision: "postgres-database",
Consumer: "g14", Module: "gitea"})
_ = bus.Put(context.Background(), BucketOf("postgres"), "g14.gitea.postgres-database", other)
p := postgres(bus, v, m)
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"g14.gitea.postgres-database"}) {
t.Fatalf("entries left %v", got)
}
if !slices.Equal(m.gone, []string{"ace.letta.postgres-database"}) {
t.Fatalf("gone %v", m.gone)
}
if len(out) != 1 || out[0].Status != Removed {
t.Fatalf("outcomes %+v", out)
}
}
// A bind whose credential the vault does not hold yet is waited for: nothing is made and no entry is written, so a
// consumer is never told of a login its provider has not made; once the vault holds it, it is answered.
func TestACredentialTheVaultDoesNotHoldYetIsWaitedFor(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if len(out) != 1 || out[0].Status != Waiting || !strings.Contains(out[0].Why, "pair/novox/postgres-database/ace/letta") {
t.Fatalf("outcomes %+v", out)
}
if bus.keys(BucketOf("postgres")) != nil || m.calls != 0 {
t.Fatal("a consumer was answered before its credential was there")
}
v.values["pair/novox/postgres-database/ace/letta"] = "late"
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
if entryOf(t, bus, "ace.letta.postgres-database").Refused != "" || m.made["ace.letta.postgres-database"] != "late" {
t.Fatal("not answered once the vault held the value")
}
}
// Only the binds naming this provider, on this machine, are its own; a bind naming it for a provision it does not
// offer is refused, naming both ends.
func TestOnlyBindsNamingThisProviderAreAnswered(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "shanks", "postgres"),
bind("mongodb-database", "", "novox", "postgres"))
assign(t, bus, "ace", "other", bind("postgres-database", "", "novox", "mssql"))
out, err := postgres(bus, v, m).Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.mongodb-database"}) {
t.Fatalf("entries %v", got)
}
e := entryOf(t, bus, "ace.letta.mongodb-database")
if !strings.Contains(e.Refused, "postgres on novox") || !strings.Contains(e.Refused, "letta on ace") {
t.Fatalf("refusal %q", e.Refused)
}
if len(out) != 1 || m.calls != 0 {
t.Fatalf("outcomes %+v, %d made", out, m.calls)
}
}
// An assignment that cannot be read is skipped and said, by its key and why; the rest of the pass goes on (issue 825,
// finding 3). Its consumer's own entry is left alone, since a bind missed would read as an unbind.
func TestAnUnreadableAssignmentIsSkippedAndSaidAndTheRestGoesOn(t *testing.T) {
for name, bad := range map[string]string{"not json": "{not json",
"another key": `{"node":"g14","module":"umami"}`} {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s",
"pair/novox/postgres-database/g14/umami": "u"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
_ = bus.Put(context.Background(), AssignmentsBucket, "ace.letta", []byte(bad))
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
var logged []string
p.opts.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
out, err := p.Pass(context.Background())
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if bus.get(BucketOf("postgres"), "g14.umami.postgres-database") == nil {
t.Fatalf("%s: the good consumer was not answered", name)
}
if bus.get(BucketOf("postgres"), "ace.letta.postgres-database") == nil || len(m.gone) != 0 {
t.Fatalf("%s: the entry of the unreadable assignment's consumer was removed", name)
}
said := false
for _, o := range out {
if o.Key == "ace.letta" && o.Status == Failed && o.Why != "" {
said = true
}
}
if !said || !strings.Contains(strings.Join(logged, "\n"), "ace.letta") {
t.Fatalf("%s: the skipped assignment is not said: %+v, %q", name, out, logged)
}
}
}
// An adapter that fails is said with the credential taken out of its words, and the entry is not written.
func TestAFailedMakeIsSaidWithoutTheCredential(t *testing.T) {
bus, m := newStandIn(), &maker{fail: errors.New(`password "hunter2-secret" rejected`)}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "hunter2-secret"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
var logged []string
p := postgres(bus, v, m)
p.opts.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if len(out) != 1 || out[0].Status != Failed || strings.Contains(out[0].Why, "hunter2") {
t.Fatalf("outcomes %+v", out)
}
if strings.Contains(strings.Join(logged, "\n"), "hunter2") || len(logged) == 0 {
t.Fatalf("logged %q", logged)
}
if bus.keys(BucketOf("postgres")) != nil {
t.Fatal("an entry was written for a consumer the adapter could not make")
}
}
// Run watches the binds on the stand-in bus: a consumer that binds is answered, and one that unbinds is removed,
// without a pass asked by hand.
func TestRunAnswersBindsAndUnbindsAsTheyAreWatched(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
p := postgres(bus, v, m)
p.opts.Every = time.Hour
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- p.Run(ctx) }()
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
waitFor(t, func() bool { return bus.get(BucketOf("postgres"), "ace.letta.postgres-database") != nil })
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
waitFor(t, func() bool { return bus.get(BucketOf("postgres"), "ace.letta.postgres-database") == nil })
cancel()
if err := <-done; err != nil && !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
}
func waitFor(t *testing.T, ok func() bool) {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal("not within 3s")
}
time.Sleep(5 * time.Millisecond)
}
}
// The consumer's identity is the mesh's (ADR 0202 §1): mesh_<node>_<module>, lower case, `_` for anything else,
// and its DNS form writes `-` for `_`.
func TestTheIdentityIsTheMeshsAndItsDNSForm(t *testing.T) {
if got := ConsumerIdentity("Home.Server", "my-app"); got != "mesh_home_server_my_app" {
t.Fatal(got)
}
got, err := ServedTo(map[string]any{"bucket": "b-${consumer:as:dns}", "n": 3}, "mesh_ace_letta")
if err != nil || got["bucket"] != "b-mesh-ace-letta" || got["n"] != 3 {
t.Fatalf("%v %v", got, err)
}
if _, err := ServedTo(map[string]any{"x": "${consumer:node}"}, "mesh_ace_letta"); err == nil {
t.Fatal("a fact the mesh does not state was filled")
}
}
// A consumer's plan entry may carry what its module contributes to the provider (issue 776): the helper hands it to
// the provider's adapter with the consumer, and does not write it to the bucket.
func TestAConsumersContributionIsHandedToTheProviderPerConsumer(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/nextcloud": "a", "pair/novox/postgres-database/g14/umami": "b"}}
raw := `{"node":"ace","module":"nextcloud","provisioning":{"templates":{"source":"x","digest":"y"},` +
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
`"contributes":{"name":"nextcloud","extensions":["pg_trgm"]}}]}}`
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.nextcloud", []byte(raw)); err != nil {
t.Fatal(err)
}
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
t.Fatal(err)
}
want := map[string]any{"name": "nextcloud", "extensions": []any{"pg_trgm"}}
if got := m.contributed["ace.nextcloud.postgres-database"]; !reflect.DeepEqual(got, want) {
t.Fatalf("the adapter was handed the contribution %v, want %v", got, want)
}
if got := m.contributed["g14.umami.postgres-database"]; got != nil {
t.Fatalf("a consumer contributing nothing was handed %v", got)
}
if bytes.Contains(bus.get(BucketOf("postgres"), "ace.nextcloud.postgres-database"), []byte("pg_trgm")) {
t.Fatal("a contribution is in the provider's bucket")
}
}
// A consumer's plan entry carries where its provider is, which machine, and who the consumer is (issue 788): the
// entry's values carry `at` and `from` as the binding file gives them, and its `as` is the plan's, so a module with a
// slug keeps the login it has today. An entry of a plan without them is as it was.
func TestAConsumersEntryCarriesAtFromAndTheAsOfItsPlan(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/claude-code": "a", "pair/novox/postgres-database/g14/umami": "b"}}
raw := `{"node":"ace","module":"claude-code","provisioning":{"templates":{"source":"x","digest":"y"},` +
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
`"at":"10.7.0.1","from":"novox","as":"mesh_ace_agent"}]}}`
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.claude-code", []byte(raw)); err != nil {
t.Fatal(err)
}
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
t.Fatal(err)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.claude-code.postgres-database"), &e); err != nil {
t.Fatal(err)
}
if e.Values["at"] != "10.7.0.1" || e.Values["from"] != "novox" || e.Values["port"] == nil {
t.Fatalf("the entry's values are %v", e.Values)
}
if e.As != "mesh_ace_agent" || e.User != "mesh_ace_agent" || m.as["ace.claude-code.postgres-database"] != "mesh_ace_agent" {
t.Fatalf("the entry is %+v, made as %q", e, m.as["ace.claude-code.postgres-database"])
}
var plain Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "g14.umami.postgres-database"), &plain); err != nil {
t.Fatal(err)
}
if _, has := plain.Values["at"]; has || plain.As != "mesh_g14_umami" {
t.Fatalf("an entry of a plan without them changed: %+v", plain)
}
}
// A consumer's plan entry may carry the names the controller composed for it (issue 792): each is copied into the
// entry's values under its key, over anything the provider serves under it. An entry without them is as it was.
func TestAConsumersEntryCarriesTheNamesOfItsPlan(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{
"pair/novox/postgres-database/ace/gitea": "a", "pair/novox/postgres-database/g14/umami": "b"}}
raw := `{"node":"ace","module":"gitea","provisioning":{"templates":{"source":"x","digest":"y"},` +
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
`"names":{"name":"git.example.org","internal-name":"git.mesh","name-web":"web.example.org"}}]}}`
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.gitea", []byte(raw)); err != nil {
t.Fatal(err)
}
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
t.Fatal(err)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.gitea.postgres-database"), &e); err != nil {
t.Fatal(err)
}
if e.Values["name"] != "git.example.org" || e.Values["internal-name"] != "git.mesh" ||
e.Values["name-web"] != "web.example.org" || e.Values["port"] == nil {
t.Fatalf("the entry's values are %v", e.Values)
}
var plain Entry
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "g14.umami.postgres-database"), &plain); err != nil {
t.Fatal(err)
}
if _, has := plain.Values["name"]; has {
t.Fatalf("an entry of a plan without names changed: %+v", plain)
}
}
// refusedByTheStateGuard mirrors the key rule of the runtime's state guard (mesh-tools node-tools internal/bus
// state.go): a JSON key whose lowercased name, with `-`, `_` and spaces taken out, ends in one of these is refused.
func refusedByTheStateGuard(key string) bool {
k := strings.NewReplacer("-", "", "_", "", " ", "").Replace(strings.ToLower(key))
for _, end := range []string{"password", "passwd", "secret", "token", "credential", "credentials",
"authorization", "apikey", "privatekey", "accesskey", "cookie"} {
if strings.HasSuffix(k, end) {
return true
}
}
return false
}
func keysOf(v any, into *[]string) {
switch x := v.(type) {
case map[string]any:
for k, c := range x {
*into = append(*into, k)
keysOf(c, into)
}
case []any:
for _, c := range x {
keysOf(c, into)
}
}
}
// Every entry the helper writes passes the state guard's key rule (issue 945), and names where the vault keeps the
// credential under `vault-name`, for a login per consumer and for a shared one.
func TestAnEntryHasNoKeyTheStateGuardRefuses(t *testing.T) {
for name, form := range map[string]CredentialForm{"pair": PerConsumer, "shared": Shared} {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1",
"provision/novox/postgres-database": "s2"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"postgres-database": {Credential: form, User: "shared-user"}}})
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
var doc map[string]any
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.letta.postgres-database"), &doc); err != nil {
t.Fatal(err)
}
var keys []string
keysOf(doc, &keys)
for _, k := range keys {
if refusedByTheStateGuard(k) {
t.Fatalf("%s: the entry carries the key %q, which the state guard refuses", name, k)
}
}
want := map[string]string{"pair": "pair/novox/postgres-database/ace/letta",
"shared": "provision/novox/postgres-database"}[name]
if doc["vault-name"] != want {
t.Fatalf("%s: vault-name is %v, want %s", name, doc["vault-name"], want)
}
}
}
// The plan's credential form and holder name win over the offer's; the offer's form is the default (issue 825,
// finding 4).
func TestThePlansCredentialFormWinsOverTheOffers(t *testing.T) {
run := func(offer CredentialForm, plan map[string]string) Entry {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "p",
"provision/novox/postgres-database": "s", "provision/novox/postgres-database/ace/letta": "h"}}
b := bind("postgres-database", "", "novox", "postgres")
for k, val := range plan {
b[k] = val
}
assign(t, bus, "ace", "letta", b)
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
Offers: map[string]Offer{"postgres-database": {Credential: offer, User: "shared-user"}}})
out, err := p.Pass(context.Background())
if err != nil || len(out) != 1 || out[0].Status != Answered {
t.Fatalf("%v %+v", err, out)
}
return entryOf(t, bus, "ace.letta.postgres-database")
}
if e := run(PerConsumer, map[string]string{"credential-form": "shared"}); e.VaultName !=
"provision/novox/postgres-database" || e.User != "shared-user" {
t.Fatalf("shared over pair: %+v", e)
}
if e := run(Shared, map[string]string{"credential-form": "pair"}); e.VaultName !=
"pair/novox/postgres-database/ace/letta" || e.User != "mesh_ace_letta" {
t.Fatalf("pair over shared: %+v", e)
}
for _, form := range []string{"issued", "given"} {
if e := run(PerConsumer, map[string]string{"credential-form": form, "name": "main"}); e.VaultName !=
"provision/novox/postgres-database/ace/letta" || e.User != "mesh_ace_letta" {
t.Fatalf("%s holder: %+v", form, e)
}
}
if e := run(Shared, nil); e.VaultName != "provision/novox/postgres-database" {
t.Fatalf("the offer's form is the default: %+v", e)
}
}
// An instance deletes only an entry it wrote, and only while the entry is as it read it: one that another instance of
// the provider wrote after the consumer moved stands (issue 825, finding 5).
func TestAnEntryAnotherInstanceWroteAfterTheMoveStands(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
a := postgres(bus, v, m)
if _, err := a.Pass(context.Background()); err != nil {
t.Fatal(err)
}
// The consumer moves to the instance on shanks: A's pass reads its own entry and the unbind, and while it lets
// the consumer go, B writes the consumer's entry.
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
theirs, _ := json.Marshal(Entry{Provider: "shanks", ProviderModule: "postgres", Provision: "postgres-database",
Consumer: "ace", Module: "letta", VaultName: "pair/shanks/postgres-database/ace/letta"})
a.opts.Gone = func(ctx context.Context, e Entry) error {
return bus.Put(ctx, BucketOf("postgres"), e.Key(), theirs)
}
out, err := a.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.get(BucketOf("postgres"), "ace.letta.postgres-database"); !bytes.Equal(got, theirs) {
t.Fatalf("the other instance's entry was removed: %s", got)
}
for _, o := range out {
if o.Status == Removed {
t.Fatalf("outcomes %+v", out)
}
}
}
// A failed Make is said with the credential taken out in the forms it can come back in: as it is, in base64 (standard
// and URL, with and without padding) and as a JSON string says it (issue 825, finding 6).
func TestAFailedMakeIsSaidWithoutTheCredentialInAnyOfItsForms(t *testing.T) {
cred := "pa/ss+w\"rd?>>x"
b := []byte(cred)
jsonQuoted, _ := json.Marshal(cred)
forms := map[string]string{
"raw": cred, "std": base64.StdEncoding.EncodeToString(b), "std unpadded": base64.RawStdEncoding.EncodeToString(b),
"url": base64.URLEncoding.EncodeToString(b), "url unpadded": base64.RawURLEncoding.EncodeToString(b),
"json": string(jsonQuoted[1 : len(jsonQuoted)-1]),
}
if forms["std"] == forms["url"] || forms["json"] == forms["raw"] {
t.Fatal("the test credential does not tell the forms apart")
}
for name, form := range forms {
bus, m := newStandIn(), &maker{fail: errors.New("login failed for " + form + " at the server")}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": cred}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
out, err := postgres(bus, v, m).Pass(context.Background())
if err != nil || len(out) != 1 {
t.Fatalf("%v %+v", err, out)
}
if strings.Contains(out[0].Why, form) || !strings.Contains(out[0].Why, "<credential>") {
t.Fatalf("%s form: %q", name, out[0].Why)
}
}
}
// An offer that declares no credential is answered with none whatever form the plan says (the controller writes
// "pair" for such an offer): no vault name, and the Vault is never asked, so a provider with no Vault works (issue 825,
// finding 4; issue 923).
func TestAnOfferWithNoCredentialIgnoresThePlansForm(t *testing.T) {
bus := newStandIn()
b := bind("route", "", "novox", "route-proxy")
b["credential-form"] = "pair"
assign(t, bus, "ace", "letta", b)
p := New(Options{Node: "novox", Module: "route-proxy", Bus: bus, Logf: func(string, ...any) {},
Offers: map[string]Offer{"route": {Credential: NoCredential}}})
out, err := p.Pass(context.Background())
if err != nil || len(out) != 1 || out[0].Status != Answered {
t.Fatalf("%v %+v", err, out)
}
var e Entry
if err := json.Unmarshal(bus.get(BucketOf("route-proxy"), "ace.letta.route"), &e); err != nil || e.VaultName != "" ||
e.User != "" {
t.Fatalf("%v %+v", err, e)
}
}
+152
View File
@@ -0,0 +1,152 @@
package provider
import (
"bufio"
"context"
"encoding/json"
"io"
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// runtime drives a bundle over pipes as the runtime does: it reads what the bundle says and sends what the runtime would.
type runtime struct {
t *testing.T
lines *bufio.Scanner
in io.Writer
}
func (r *runtime) next() map[string]any {
done := make(chan map[string]any, 1)
go func() {
if r.lines.Scan() {
var m map[string]any
_ = json.Unmarshal(r.lines.Bytes(), &m)
done <- m
}
}()
select {
case m := <-done:
return m
case <-time.After(3 * time.Second):
r.t.Fatal("the bundle said nothing")
return nil
}
}
func (r *runtime) send(m map[string]any) {
b, _ := json.Marshal(m)
_, _ = r.in.Write(append(b, '\n'))
}
func (r *runtime) call(id int, tool string, args map[string]any) {
r.send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "tools/call", "params": map[string]any{"name": tool, "arguments": args}})
}
func serve(t *testing.T, tools []stdio.Tool) *runtime {
t.Setenv(stdio.ServedAs, "keeper")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() { _ = stdio.ServeOn("", tools, inR, outW) }()
t.Cleanup(func() { _ = inW.Close() })
return &runtime{t: t, lines: bufio.NewScanner(outR), in: inW}
}
// The bus of a moved provider maps its own bucket to the state `bound` and the assignments to the state
// `mesh-controller.assignments`, writes only the own bucket's state, and refuses any other bucket.
func TestStateBusMapsTheProvidersBucketsToTheRuntimesStates(t *testing.T) {
bus := StateBus("postgres")
rt := serve(t, []stdio.Tool{
{Name: "watch", Run: func(a map[string]any) (any, error) {
return nil, bus.Watch(context.Background(), a["bucket"].(string), func(string, []byte) {})
}},
{Name: "put", Run: func(map[string]any) (any, error) {
return "put", bus.Put(context.Background(), BucketOf("postgres"), "k", []byte(`{"a":1}`))
}},
{Name: "delete", Run: func(map[string]any) (any, error) {
return "deleted", bus.Delete(context.Background(), BucketOf("postgres"), "k")
}},
})
for i, c := range []struct{ bucket, state string }{
{BucketOf("postgres"), "bound"},
{AssignmentsBucket, "mesh-controller.assignments"},
} {
rt.call(i+1, "watch", map[string]any{"bucket": c.bucket})
w := rt.next()
if w["method"] != "mesh/state.watch" || w["params"].(map[string]any)["state"] != c.state {
t.Fatalf("bucket %s asked %v, want the state %s", c.bucket, w, c.state)
}
rt.send(map[string]any{"jsonrpc": "2.0", "id": w["id"], "result": map[string]any{}})
rt.next()
}
rt.call(3, "put", nil)
p := rt.next()
if p["method"] != "mesh/state.put" || p["params"].(map[string]any)["state"] != "bound" || p["params"].(map[string]any)["key"] != "k" {
t.Fatalf("put asked %v", p)
}
rt.send(map[string]any{"jsonrpc": "2.0", "id": p["id"], "result": map[string]any{"revision": 1}})
rt.next()
rt.call(4, "delete", nil)
d := rt.next()
if d["method"] != "mesh/state.delete" || d["params"].(map[string]any)["state"] != "bound" {
t.Fatalf("delete asked %v", d)
}
rt.send(map[string]any{"jsonrpc": "2.0", "id": d["id"], "result": map[string]any{}})
rt.next()
for _, bucket := range []string{"other_bound", "elsewhere"} {
if err := bus.Watch(context.Background(), bucket, func(string, []byte) {}); err == nil || !strings.Contains(err.Error(), "not a bucket") {
t.Fatalf("a watch of %s gave %v", bucket, err)
}
if err := bus.Put(context.Background(), bucket, "k", nil); err == nil {
t.Fatalf("a put to %s was allowed", bucket)
}
if err := bus.Delete(context.Background(), bucket, "k"); err == nil {
t.Fatalf("a delete in %s was allowed", bucket)
}
}
}
// A watch whose ctx has ended calls its handler no more, though the runtime keeps sending changes: the Bus contract
// says a watch lasts until ctx ends, and a stdio state watch lasts as long as the process.
func TestStateBusWatchCallsItsHandlerNoMoreOnceItsContextEnded(t *testing.T) {
bus := StateBus("postgres")
ctx, cancel := context.WithCancel(context.Background())
handled := make(chan string, 8)
rt := serve(t, []stdio.Tool{
{Name: "start", Run: func(map[string]any) (any, error) {
return "started", bus.Watch(ctx, AssignmentsBucket, func(key string, value []byte) {
handled <- key + "=" + string(value)
})
}},
{Name: "stop", Run: func(map[string]any) (any, error) { cancel(); return "stopped", nil }},
})
change := func(id int, watch any, key, op string) {
rt.send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "mesh/state", "params": map[string]any{"watch": watch,
"change": map[string]any{"state": "mesh-controller.assignments", "key": key, "op": op, "value": "v"}}})
if r := rt.next(); r["id"] != float64(id) || r["error"] != nil {
t.Fatalf("a change was answered %v", r)
}
}
rt.call(1, "start", nil)
w := rt.next()
rt.send(map[string]any{"jsonrpc": "2.0", "id": w["id"], "result": map[string]any{}})
rt.next()
change(50, w["id"], "before", "put")
if got := <-handled; got != `before="v"` {
t.Fatalf("handled %q", got)
}
rt.call(2, "stop", nil)
rt.next()
change(51, w["id"], "after", "put")
change(52, w["id"], "after", "delete")
select {
case got := <-handled:
t.Fatalf("the handler was called after its context ended: %q", got)
case <-time.After(100 * time.Millisecond):
}
}
+90
View File
@@ -0,0 +1,90 @@
package stdio
import (
"encoding/json"
"errors"
"time"
)
// A module's code reads its values at runtime (novox/hq ADR 0349 clause 14, ADR 0370, and the part of the feature 57 design that lets a running module read, rather than be handed in a file, its provisioned values): its config
// values and its bindings' values that are not secret, for what its manifest declares, and watches them. The
// runtime that launched this bundle asks its machine's provisioner client **as this bundle's module** — the module it
// launched, never a name this code passes — and the client answers only what that module declares, only while the
// mesh places it on that machine. A secret is refused while the module's code runs in the tool runner: the vault
// hands a secret only to a caller the bus proves, and the bus proves only the tool runner there, so a module's
// secrets reach it through its files until it runs under a bus account of its own (novox/hq issue 569).
// ProvisionChange is what a watch hands over: the value now, or — once it is no longer answered, the module taken
// off the machine or the key no longer declared — Refused, why.
type ProvisionChange struct {
Kind string `json:"kind"`
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
Refused string `json:"refused,omitempty"`
}
const provisionTimeout = 30 * time.Second
func provisionGet(kind, key string) (json.RawMessage, error) {
c, err := live()
if err != nil {
return nil, err
}
return c.ask("mesh/provision.get", map[string]any{"kind": kind, "key": key}, provisionTimeout)
}
func provisionWatch(kind, key string, handle func(ProvisionChange) error) error {
if handle == nil {
return errors.New("a watch needs a handler")
}
c, err := live()
if err != nil {
return err
}
id := c.newID()
c.mu.Lock()
c.provisioned[id] = handle
c.mu.Unlock()
if _, err := c.askAs(id, "mesh/provision.watch", map[string]any{"kind": kind, "key": key}, provisionTimeout); err != nil {
c.mu.Lock()
delete(c.provisioned, id)
c.mu.Unlock()
return err
}
return nil
}
// Config is one of this module's config values as mesh-config resolved it for this machine, as JSON: the key as
// its manifest declares it under `config`. A key it does not declare is refused, with the reason.
func Config(key string) (json.RawMessage, error) { return provisionGet("config", key) }
// WatchConfig hands handle the config value now and then every change, in order, and returns once the value now is
// handled. The watch lasts as long as the process.
func WatchConfig(key string, handle func(ProvisionChange) error) error {
return provisionWatch("config", key, handle)
}
// Bound is one value, not a secret, of what this module is bound to: the provision as its manifest requires it,
// and the key its provider answers (`${bound:<provision>:<key>}`).
func Bound(provision, key string) (json.RawMessage, error) {
return provisionGet("bound", provision+":"+key)
}
// WatchBound is WatchConfig for a bound value.
func WatchBound(provision, key string, handle func(ProvisionChange) error) error {
return provisionWatch("bound", provision+":"+key, handle)
}
// Secret is one of this module's secrets. Refused, with the reason, while the module's code runs in the tool
// runner (novox/hq ADR 0349 §11, ADR 0370 §2): its secrets reach it through its files.
func Secret(name string) ([]byte, error) {
raw, err := provisionGet("secret", name)
if err != nil {
return nil, err
}
var s string
if err := json.Unmarshal(raw, &s); err != nil {
return nil, errors.New("the runtime answered a secret that is not a string")
}
return []byte(s), nil
}
+176
View File
@@ -0,0 +1,176 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"strings"
"testing"
"time"
)
// A Go bundle reads its values at runtime through the runtime (novox/hq ADR 0349 §14, ADR 0370, issue 685): Config,
// Bound and their watches ask the runtime, which asks as the module; a watch's value now is handled before it
// returns and each change after is named by the watch's id; what the module does not declare and every secret come
// back refused, in the runtime's words; a change for no watch of this bundle is refused.
func TestABundleReadsAndWatchesItsValuesThroughTheRuntime(t *testing.T) {
t.Setenv(ServedAs, "claude-code")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
seen := make(chan string, 16)
watched := make(chan error, 1)
answers := make(chan string, 4)
go func() {
_ = ServeOn("", []Tool{
{Name: "start", Run: func(map[string]any) (any, error) {
go func() {
watched <- WatchConfig("managed_settings", func(c ProvisionChange) error {
seen <- c.Kind + " " + c.Key + " " + string(c.Value) + c.Refused
return nil
})
}()
return "started", nil
}},
{Name: "read", Run: func(a map[string]any) (any, error) {
var v json.RawMessage
var err error
switch a["what"] {
case "config":
v, err = Config(a["key"].(string))
case "bound":
v, err = Bound("mcp-endpoint", "url")
case "secret":
var s []byte
s, err = Secret("gitea-read")
v = json.RawMessage(s)
}
if err != nil {
answers <- "refused: " + err.Error()
} else {
answers <- string(v)
}
return "read", nil
}},
}, inR, outW)
}()
lines := bufio.NewScanner(outR)
next := func() map[string]any {
done := make(chan map[string]any, 1)
go func() {
if lines.Scan() {
var m map[string]any
_ = json.Unmarshal(lines.Bytes(), &m)
done <- m
}
}()
select {
case m := <-done:
return m
case <-time.After(3 * time.Second):
t.Fatal("the bundle said nothing")
return nil
}
}
send := func(m map[string]any) { b, _ := json.Marshal(m); _, _ = inW.Write(append(b, '\n')) }
// asked is the bundle's next ask of the runtime, beside its answers to this tool call and those before.
asked := func(callID int) map[string]any {
t.Helper()
var ask map[string]any
for ask == nil {
m := next()
if m["method"] != nil {
ask = m
} else if id, ok := m["id"].(float64); !ok || id > float64(callID) || m["error"] != nil {
t.Fatalf("said %v", m)
}
}
return ask
}
// The watch: asked of the runtime with the kind and the key and no module, the module being the runtime's word.
send(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": map[string]any{"name": "start"}})
watch := asked(1)
if watch["method"] != "mesh/provision.watch" {
t.Fatalf("asked %v", watch)
}
if p := watch["params"].(map[string]any); p["kind"] != "config" || p["key"] != "managed_settings" || len(p) != 2 {
t.Fatalf("asked to watch %v", p)
}
send(map[string]any{"jsonrpc": "2.0", "id": 50, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
"change": map[string]any{"kind": "config", "key": "managed_settings", "value": map[string]any{"spinnerTipsEnabled": false}}}})
for {
if r := next(); r["id"] == float64(50) {
if r["error"] != nil {
t.Fatalf("the value now was answered %v", r)
}
break
}
}
if s := <-seen; s != `config managed_settings {"spinnerTipsEnabled":false}` {
t.Fatalf("handled %q", s)
}
select {
case err := <-watched:
t.Fatalf("the watch returned before it was answered: %v", err)
default:
}
send(map[string]any{"jsonrpc": "2.0", "id": watch["id"], "result": map[string]any{}})
if err := <-watched; err != nil {
t.Fatal(err)
}
// A change, and then the value no longer answered.
send(map[string]any{"jsonrpc": "2.0", "id": 51, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
"change": map[string]any{"kind": "config", "key": "managed_settings", "value": map[string]any{"spinnerTipsEnabled": true}}}})
if r := next(); r["id"] != float64(51) || r["error"] != nil {
t.Fatalf("a change was answered %v", r)
}
if s := <-seen; s != `config managed_settings {"spinnerTipsEnabled":true}` {
t.Fatalf("handled %q", s)
}
send(map[string]any{"jsonrpc": "2.0", "id": 52, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
"change": map[string]any{"kind": "config", "key": "managed_settings", "refused": "claude-code is not placed on g14"}}})
if r := next(); r["id"] != float64(52) || r["error"] != nil {
t.Fatalf("a refusal was answered %v", r)
}
if s := <-seen; s != `config managed_settings claude-code is not placed on g14` {
t.Fatalf("handled %q", s)
}
// A change for a watch this bundle never asked for is refused.
send(map[string]any{"jsonrpc": "2.0", "id": 53, "method": "mesh/provision", "params": map[string]any{"watch": "bundle-999",
"change": map[string]any{"kind": "config", "key": "x"}}})
if r := next(); r["id"] != float64(53) || !strings.Contains(r["error"].(map[string]any)["message"].(string), "names no watch") {
t.Fatalf("a change for no watch was answered %v", r)
}
// Each reader asks the runtime, and hands back its answer or its refusal.
for i, c := range []struct {
what, key, kind, askedKey string
result any
refusal string
want string
}{
{what: "config", key: "role", kind: "config", askedKey: "role", result: "the build machine", want: `"the build machine"`},
{what: "config", key: "nothing", kind: "config", askedKey: "nothing",
refusal: "claude-code declares no config value nothing", want: "refused: claude-code declares no config value nothing"},
{what: "bound", kind: "bound", askedKey: "mcp-endpoint:url", result: "http://127.0.0.1:9000", want: `"http://127.0.0.1:9000"`},
{what: "secret", kind: "secret", askedKey: "gitea-read",
refusal: "the vault hands a secret only to a caller the bus proves", want: "refused: the vault hands a secret only to a caller the bus proves"},
} {
id := 10 + i
send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "tools/call", "params": map[string]any{"name": "read",
"arguments": map[string]any{"what": c.what, "key": c.key}}})
get := asked(id)
if p := get["params"].(map[string]any); get["method"] != "mesh/provision.get" || p["kind"] != c.kind || p["key"] != c.askedKey || len(p) != 2 {
t.Fatalf("%s: asked %v", c.what, get)
}
if c.refusal != "" {
send(map[string]any{"jsonrpc": "2.0", "id": get["id"], "error": map[string]any{"code": -32000, "message": c.refusal}})
} else {
send(map[string]any{"jsonrpc": "2.0", "id": get["id"], "result": c.result})
}
if got := <-answers; got != c.want {
t.Errorf("%s %s: %q, want %q", c.what, c.key, got, c.want)
}
}
_ = inW.Close()
}
+203
View File
@@ -0,0 +1,203 @@
// Package seal is the mesh's sealed box (novox/hq ADR 0183, ADR 0206): a value sealed to one recipient's
// public key, which only that recipient's private key opens. X25519 for the agreement, HKDF-SHA256 for the
// key, AES-256-GCM for the box — the box the licence manager made and opened before it lived here, byte
// for byte, and the one the SDK's TypeScript `seal` makes and opens.
//
// A box is `{ v: 1, eph, iv, tag, ct }`, every field standard base64; `eph` is the one-time public key as
// SPKI DER. The key is derived with the salt eph-DER ‖ the recipient's raw public key and the info
// "novox-mesh sealed box v1", so a box cannot be re-addressed to another key.
//
// Keys travel as PEM: the public key as SPKI ("PUBLIC KEY"), the private key as PKCS #8 ("PRIVATE KEY").
// Standard library only.
package seal
import (
"crypto/aes"
"crypto/cipher"
"crypto/ecdh"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"encoding/pem"
"errors"
"fmt"
)
// Box is a value sealed to one recipient, as it travels in JSON.
type Box struct {
V int `json:"v"`
Eph string `json:"eph"`
IV string `json:"iv"`
Tag string `json:"tag"`
Ct string `json:"ct"`
}
// KeyPair is a recipient's keypair as the two PEM strings it is kept and sent as.
type KeyPair struct {
PublicKey string `json:"publicKey"`
PrivateKey string `json:"privateKey"`
}
// Version is the only box version Open opens.
const Version = 1
const info = "novox-mesh sealed box v1"
// GenerateKeyPair makes a fresh X25519 keypair, PEM-encoded.
func GenerateKeyPair() (KeyPair, error) {
priv, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
return KeyPair{}, err
}
pubDER, err := x509.MarshalPKIXPublicKey(priv.PublicKey())
if err != nil {
return KeyPair{}, err
}
privDER, err := x509.MarshalPKCS8PrivateKey(priv)
if err != nil {
return KeyPair{}, err
}
return KeyPair{
PublicKey: string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: pubDER})),
PrivateKey: string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: privDER})),
}, nil
}
// Seal seals plaintext to the recipient's PEM public key, with a fresh one-time key and iv.
func Seal(plaintext, recipientPEM string) (Box, error) {
eph, err := ecdh.X25519().GenerateKey(rand.Reader)
if err != nil {
return Box{}, err
}
iv := make([]byte, 12)
if _, err := rand.Read(iv); err != nil {
return Box{}, err
}
return sealWith(plaintext, recipientPEM, eph, iv)
}
// sealWith is Seal with its two sources of randomness given, for the byte-for-byte tests.
func sealWith(plaintext, recipientPEM string, eph *ecdh.PrivateKey, iv []byte) (Box, error) {
recipient, err := publicFromPEM(recipientPEM)
if err != nil {
return Box{}, err
}
secret, err := eph.ECDH(recipient)
if err != nil {
return Box{}, err
}
ephDER, err := x509.MarshalPKIXPublicKey(eph.PublicKey())
if err != nil {
return Box{}, err
}
gcm, err := newGCM(boxKey(secret, ephDER, recipient.Bytes()))
if err != nil {
return Box{}, err
}
out := gcm.Seal(nil, iv, []byte(plaintext), nil)
ct, tag := out[:len(out)-gcm.Overhead()], out[len(out)-gcm.Overhead():]
b64 := base64.StdEncoding.EncodeToString
return Box{V: Version, Eph: b64(ephDER), IV: b64(iv), Tag: b64(tag), Ct: b64(ct)}, nil
}
// Open opens a box with the recipient's PEM private key. It fails for a box sealed to another key, one
// changed in any field, or one of another version.
func Open(box Box, privatePEM string) (string, error) {
if box.V != Version {
return "", fmt.Errorf("a sealed box of version %d, not %d", box.V, Version)
}
priv, err := privateFromPEM(privatePEM)
if err != nil {
return "", err
}
d := base64.StdEncoding.DecodeString
ephDER, err := d(box.Eph)
if err != nil {
return "", fmt.Errorf("the box's eph: %w", err)
}
ephKey, err := x509.ParsePKIXPublicKey(ephDER)
if err != nil {
return "", fmt.Errorf("the box's eph: %w", err)
}
eph, ok := ephKey.(*ecdh.PublicKey)
if !ok || eph.Curve() != ecdh.X25519() {
return "", errors.New("the box's eph is not an X25519 key")
}
secret, err := priv.ECDH(eph)
if err != nil {
return "", err
}
iv, err1 := d(box.IV)
tag, err2 := d(box.Tag)
ct, err3 := d(box.Ct)
if err := errors.Join(err1, err2, err3); err != nil {
return "", fmt.Errorf("the box's fields: %w", err)
}
gcm, err := newGCM(boxKey(secret, ephDER, priv.PublicKey().Bytes()))
if err != nil {
return "", err
}
if len(iv) != gcm.NonceSize() || len(tag) != gcm.Overhead() {
return "", errors.New("the box does not open with this key")
}
plain, err := gcm.Open(nil, iv, append(append([]byte{}, ct...), tag...), nil)
if err != nil {
return "", errors.New("the box does not open with this key")
}
return string(plain), nil
}
func publicFromPEM(p string) (*ecdh.PublicKey, error) {
block, _ := pem.Decode([]byte(p))
if block == nil {
return nil, errors.New("not a PEM public key")
}
k, err := x509.ParsePKIXPublicKey(block.Bytes)
if err != nil {
return nil, err
}
pub, ok := k.(*ecdh.PublicKey)
if !ok || pub.Curve() != ecdh.X25519() {
return nil, errors.New("not an X25519 public key")
}
return pub, nil
}
func privateFromPEM(p string) (*ecdh.PrivateKey, error) {
block, _ := pem.Decode([]byte(p))
if block == nil {
return nil, errors.New("not a PEM private key")
}
k, err := x509.ParsePKCS8PrivateKey(block.Bytes)
if err != nil {
return nil, err
}
priv, ok := k.(*ecdh.PrivateKey)
if !ok || priv.Curve() != ecdh.X25519() {
return nil, errors.New("not an X25519 private key")
}
return priv, nil
}
// boxKey is HKDF-SHA256 (RFC 5869) for one 32-byte block: written out because crypto/hkdf needs Go 1.24
// and this module promises 1.22; the vectors prove it equals the licence manager's hkdf.Key.
func boxKey(secret, ephDER, recipientRaw []byte) []byte {
salt := append(append([]byte{}, ephDER...), recipientRaw...)
extract := hmac.New(sha256.New, salt)
extract.Write(secret)
prk := extract.Sum(nil)
expand := hmac.New(sha256.New, prk)
expand.Write([]byte(info))
expand.Write([]byte{1})
return expand.Sum(nil)
}
func newGCM(key []byte) (cipher.AEAD, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
return cipher.NewGCM(block)
}
+156
View File
@@ -0,0 +1,156 @@
package seal
import (
"crypto/ecdh"
"encoding/base64"
"encoding/json"
"os"
"testing"
)
type fixture struct {
Recipient KeyPair `json:"recipient"`
Sealed []struct {
Plaintext string `json:"plaintext"`
Box Box `json:"box"`
} `json:"sealed"`
Vectors []struct {
Plaintext string `json:"plaintext"`
EphPrivate string `json:"ephPrivate"`
IV string `json:"iv"`
Box Box `json:"box"`
} `json:"vectors"`
}
func readFixture(t *testing.T) fixture {
t.Helper()
raw, err := os.ReadFile("testdata/sealed-by-licence-manager.json")
if err != nil {
t.Fatal(err)
}
var f fixture
if err := json.Unmarshal(raw, &f); err != nil {
t.Fatal(err)
}
if len(f.Sealed) == 0 || len(f.Vectors) == 0 {
t.Fatal("the fixture holds no boxes")
}
return f
}
// Every box the licence manager's own Seal made opens here to its plaintext.
func TestABoxTheLicenceManagerSealedOpens(t *testing.T) {
f := readFixture(t)
for _, s := range f.Sealed {
got, err := Open(s.Box, f.Recipient.PrivateKey)
if err != nil || got != s.Plaintext {
t.Fatalf("opened %q, %v; want %q", got, err, s.Plaintext)
}
}
}
// Given the same one-time key and iv, this package seals byte for byte what the licence manager sealed:
// the reverse direction, so whatever opens the licence manager's boxes opens these.
func TestSealIsByteForByteTheLicenceManagersBox(t *testing.T) {
f := readFixture(t)
for _, v := range f.Vectors {
d := base64.StdEncoding.DecodeString
ephRaw, err1 := d(v.EphPrivate)
iv, err2 := d(v.IV)
if err1 != nil || err2 != nil {
t.Fatal(err1, err2)
}
eph, err := ecdh.X25519().NewPrivateKey(ephRaw)
if err != nil {
t.Fatal(err)
}
got, err := sealWith(v.Plaintext, f.Recipient.PublicKey, eph, iv)
if err != nil {
t.Fatal(err)
}
if got != v.Box {
t.Fatalf("sealed %+v\nthe licence manager sealed %+v", got, v.Box)
}
if plain, err := Open(got, f.Recipient.PrivateKey); err != nil || plain != v.Plaintext {
t.Fatalf("reopened %q, %v", plain, err)
}
}
}
// A box the agent module's TypeScript sealed (the fixture the licence manager keeps) opens here too.
func TestABoxSealedInTypeScriptOpens(t *testing.T) {
raw, err := os.ReadFile("testdata/sealed-by-typescript.json")
if err != nil {
t.Fatal(err)
}
var f struct {
PrivateKey string `json:"privateKey"`
Box Box `json:"box"`
Plaintext string `json:"plaintext"`
}
if err := json.Unmarshal(raw, &f); err != nil {
t.Fatal(err)
}
if got, err := Open(f.Box, f.PrivateKey); err != nil || got != f.Plaintext {
t.Fatalf("opened %q, %v", got, err)
}
}
// What Seal makes opens with its recipient's key and no other, and a fresh box differs every time.
func TestABoxOpensOnlyForItsRecipient(t *testing.T) {
a, err := GenerateKeyPair()
if err != nil {
t.Fatal(err)
}
b, _ := GenerateKeyPair()
box, err := Seal("a token", a.PublicKey)
if err != nil {
t.Fatal(err)
}
if got, err := Open(box, a.PrivateKey); err != nil || got != "a token" {
t.Fatalf("opened %q, %v", got, err)
}
if _, err := Open(box, b.PrivateKey); err == nil {
t.Fatal("a box opened for another key")
}
again, _ := Seal("a token", a.PublicKey)
if again.Eph == box.Eph || again.IV == box.IV {
t.Fatal("two boxes share a one-time key or iv")
}
}
// A box changed in any field, or of another version, does not open.
func TestATamperedBoxDoesNotOpen(t *testing.T) {
f := readFixture(t)
good := f.Sealed[1].Box
flip := func(s string) string {
b, _ := base64.StdEncoding.DecodeString(s)
b[len(b)-1] ^= 1
return base64.StdEncoding.EncodeToString(b)
}
for name, box := range map[string]Box{
"ct": {V: 1, Eph: good.Eph, IV: good.IV, Tag: good.Tag, Ct: flip(good.Ct)},
"tag": {V: 1, Eph: good.Eph, IV: good.IV, Tag: flip(good.Tag), Ct: good.Ct},
"iv": {V: 1, Eph: good.Eph, IV: flip(good.IV), Tag: good.Tag, Ct: good.Ct},
"eph": {V: 1, Eph: flip(good.Eph), IV: good.IV, Tag: good.Tag, Ct: good.Ct},
"version": {V: 2, Eph: good.Eph, IV: good.IV, Tag: good.Tag, Ct: good.Ct},
} {
if _, err := Open(box, f.Recipient.PrivateKey); err == nil {
t.Fatalf("a box with a changed %s opened", name)
}
}
}
// A key that is not X25519 PEM is refused, not misread.
func TestAKeyOfAnotherKindIsRefused(t *testing.T) {
if _, err := Seal("x", "not a key"); err == nil {
t.Fatal("sealed to text")
}
f := readFixture(t)
if _, err := Open(f.Sealed[0].Box, f.Recipient.PublicKey); err == nil {
t.Fatal("opened with a public key")
}
if _, err := Seal("x", f.Recipient.PrivateKey); err == nil {
t.Fatal("sealed to a private key")
}
}
+121
View File
@@ -0,0 +1,121 @@
{
"about": "Made by the claude-licence-manager's seal.go (mesh-catalog feat/57-config-and-provisioning at 8aa37e8), unchanged but for its two sources of randomness, which the vectors fix. The recipient is a test key derived from a fixed label; it guards nothing.",
"recipient": {
"publicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VuAyEAkZXnghyfA1BR5rWtvBLmjdZMCbC99xxF3WKHe1E+3XM=\n-----END PUBLIC KEY-----\n",
"privateKey": "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VuBCIEIHjV954xXPa72FyGUw5PsWCbsHrliVp/7xoAgG9fGs7q\n-----END PRIVATE KEY-----\n"
},
"sealed": [
{
"plaintext": "",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAwN2gt32hv+9VyFh0oQ8oetPBHsZ9dt80DzwPwcB0bHg=",
"iv": "0xHCegxP6o48EGLq",
"tag": "VYQZm/lBZB0YMPUqYVPKaQ==",
"ct": ""
}
},
{
"plaintext": "a grant sealed by the licence manager",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEA54v96LEb8SwT2uJZCa90/2jcLvEbK/BZpbMS6TxAAW0=",
"iv": "xm6Jg4oOnRpRpDGK",
"tag": "cmbOzIV63Nv0lPfpTgP2kg==",
"ct": "InWe8PZ8hkd7F9QaHujjhdiBECKzLAK22d2aIEXwBhtxIowc/Q=="
}
},
{
"plaintext": "sk-ant-api03-test-0123456789abcdefghij\n",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAIXdum2FYLzvLDj/Of2253hH8Lu0szU4tLxJzfXTahDs=",
"iv": "KMPNa60C/Qjhtp0Z",
"tag": "PPa82ZdFyWz9kbLSSGkl3g==",
"ct": "Phu3ZsxIWGlBWbYOGzPopncEbc0VhTAloxzeHs4KDBMFZU9Pet2V"
}
},
{
"plaintext": "ünïcødé ✓ and a NUL \u0000 inside",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAo1WaRyrAJFM6If/cpFgdpZjsC6XebuJtVSugbI+m8RU=",
"iv": "H9+WvM6ZYbCYJj6D",
"tag": "K5p986nSpeItYK1m4bcVPQ==",
"ct": "eyZs3uVeZguNbqSmrMc1rWgxhW3nP0GABH6Ek06/8UuBqw=="
}
},
{
"plaintext": "a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; ",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAzjZdTMu08I0Z+FWXSr9f++EADfqg3qr2FVt+3Q/hxm0=",
"iv": "lTG2VwPA5SwHbZE/",
"tag": "Il6ICxoZgZr9dOgeIZOPrg==",
"ct": "ZWQ8m1nxrqq0oIGRoeNlUu9/JpTMrxlncTCL5cDqsJrx9NjS0nZX292JJ5p0+BYSVLUN+yGUuBOnH860ywvKK7XhPOc0Yo2nq7X2O3OYn9Za4y+HK6dVDM8zhsS8P3Di/MJCwS+AnremVCaj2k3xHL3oc331RBZZ5w2JfgSX3ouwl2+VbIPyJVSDsxnLLoHsiIR7nB0GNCe0fynhVkoIEwC8IXLnL6m3rjjN3YCQDmVP2UoftniLAypd5ioc9GCtpvUL6ktjs82JkvEtFbHtc+YzIjP6ZnjTj/bvbRHkIqN4MDiu+fvGOr0jtGWO24ecGC+6KtrKGOaEYHDqOLaK3c+UcrDsFYC+cuKCdCNhs5UNhYYylVXE648KFa488eFk"
}
}
],
"vectors": [
{
"plaintext": "",
"ephPrivate": "W8iT0SrRDnazANHYsXHC72sM1wzx+/Y8vowREeGGz/A=",
"iv": "DD7WV6Y8hLGvaG0b",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAdCCqlFhxbKBpv1JClXceH43AqEvcaM6SGfyXx05UJUs=",
"iv": "DD7WV6Y8hLGvaG0b",
"tag": "ctb+ZQjx8FyynvT88rfwkQ==",
"ct": ""
}
},
{
"plaintext": "a grant sealed by the licence manager",
"ephPrivate": "lzIbkYmIbkKTYDL4dZiAFUFv2h+GWBiqlQfnsVD2An8=",
"iv": "xz2FIBXNM3scOAJb",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAS3L49p4KrGMo/NmHZq8osoPbk2ZCQWLoNIKnpB4T5hg=",
"iv": "xz2FIBXNM3scOAJb",
"tag": "7dC8bYqzsJ/17EC7y6lhCA==",
"ct": "FEMfh4HMvMwqx4QE4YMMR3d3g5rNuxa/+u6u+i/aRLed3q8kBw=="
}
},
{
"plaintext": "sk-ant-api03-test-0123456789abcdefghij\n",
"ephPrivate": "tAjRQTQHk6nNJLblzdslDbWmyYm8cE6/G4ipXHR0zlE=",
"iv": "BPF1JDQ5mDNhKKr3",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEA1GianODxSXUHYMdUmZLcQztp5ry2/xtwUSeHZWrGFAo=",
"iv": "BPF1JDQ5mDNhKKr3",
"tag": "uiIIFD7LgPmLV3Y6uleSzg==",
"ct": "+maa0GDDOkcyzD0chFQ4a/0NvBrvl23NVqcGv1BEKgmMUenf9HHd"
}
},
{
"plaintext": "ünïcødé ✓ and a NUL \u0000 inside",
"ephPrivate": "QBYUxuk4jwiWKOPEWeE7v6WxY8BDmZZCDwlY1WJ1PSA=",
"iv": "u76JO8baqokfnjcK",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAePakOoCmmXLtnN3EykbpqfdbfXpkg04F2djxnJWH220=",
"iv": "u76JO8baqokfnjcK",
"tag": "sfBPRnwhfP/p7vlsUx+e9w==",
"ct": "B9yc0FbdKYDtDeX1jmKa+DBP8fPN1O9SbeYBDn9b3+Bq2Q=="
}
},
{
"plaintext": "a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; ",
"ephPrivate": "2VYOyzqLrb+zWUhVpHA9aV3vMAyRWlNPEq2ASf60F9M=",
"iv": "0Gv6ejq07LwbM8cY",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEARSOBczrJ6X1MkxT7Mrd1ryDLmXAMBgYIgpfwBjHz1GA=",
"iv": "0Gv6ejq07LwbM8cY",
"tag": "Vm+E0myXgdMwA6knJDg/+w==",
"ct": "tJwZpUXeQYOu2PG1g9AwIV8RhKKDorUHWQSygEus3xHOpuTwh3+D4j2htIerMB/3AKToQYABeZMyEAOzhL2x+Cg06WlUKrggZjvdX5W7X813KQBgzg0YnnCZTKTYlviDWe1E9TZ3aAauhlcgpBaQXUh/akkDiqeq2Ua5ruY6IGH5HgqWWPYJo3b4zkbArhvdP2HJX9Pn1mQa0BtGMinyutyD0am5vZuemdKGYS1Ao00M81sMXYc2CBy9K48Ofw8GapcmYNHsE0s6NSgNZumnVaR8TQRy/oqEdL3Q6z31hAZMzTdS0ckXg1NCgcgXVuzwNov/D4320E3DxFS3v8Z5p74ZkqAmo/1JcCNJwwQyRCWk/4iJFjP1OKEavOGo+bcp"
}
}
]
}
+12
View File
@@ -0,0 +1,12 @@
{
"privateKey": "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VuBCIEIAi2NK/bN+p7cqYUwv/kz72TgLdmJUfOHCDZTrMpQzpS\n-----END PRIVATE KEY-----\n",
"publicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VuAyEARYvD/w+9ah0KWS9T9pd6Ea6CBymUE48vEVk983QPKyY=\n-----END PUBLIC KEY-----\n",
"box": {
"v": 1,
"eph": "MCowBQYDK2VuAyEAIYlSGrJvF8qSjR1aTFWbEQM/NFbZXrarglN0aRLZZ0E=",
"iv": "ABqT/hMukn6+xpjh",
"tag": "POzmsWTPHSn9xLMMJJ9Akg==",
"ct": "m2u0kuQ0PwrsGelM99Z5aBw6FCd6lFISUbwiK5TzzDw4ZrGtsHEvxytoIeFC"
},
"plaintext": "a grant sealed by the TypeScript agent module"
}
+72
View File
@@ -2,6 +2,8 @@ package stdio
import (
"encoding/json"
"errors"
"fmt"
"time"
)
@@ -29,6 +31,16 @@ type StateChange struct {
Current bool `json:"current"`
}
// StateRevision is one value a bucket still keeps for a key: what was written, at which revision, by which
// operation ("put" or "delete", whose value is null) and when (novox/hq ADR 0307).
type StateRevision struct {
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
Op string `json:"op"`
At time.Time `json:"at"`
}
// KeptState is one state this module keeps, or reads.
type KeptState struct{ name string }
@@ -56,6 +68,21 @@ func (s KeptState) Get(key string) (*StateEntry, error) {
return &e, json.Unmarshal(raw, &e)
}
// History is every value the bucket still keeps for key, oldest first — as many as the state declared it
// keeps, deletes included — and empty for a key it holds nothing for (novox/hq ADR 0307). A read, like Get:
// the owner's instances and its readers may ask it, and it writes nothing.
func (s KeptState) History(key string) ([]StateRevision, error) {
raw, err := s.ask("history", map[string]any{"key": key})
if err != nil {
return nil, err
}
revisions := []StateRevision{}
if len(raw) == 0 || string(raw) == "null" {
return revisions, nil
}
return revisions, json.Unmarshal(raw, &revisions)
}
// Put writes a key — the owner's instances only — and returns the revision written.
func (s KeptState) Put(key string, value any) (uint64, error) {
raw, err := s.ask("put", map[string]any{"key": key, "value": value})
@@ -74,6 +101,51 @@ func (s KeptState) Delete(key string) error {
return err
}
// Increment adds one to a counter kept at key and returns the number it now holds: one for a key with no
// value. Read, then written only if the key is still at the revision read (Create when it had none, Update
// when it had one), so of any number of writers each number is handed out once and no increment is lost.
// A write another writer won is read again and tried again, at most attempts times in all; past that the
// error is ErrChanged and nothing was counted. Any other error is returned at once, as is a value that is
// not a whole number (null included): nothing is ever reported counted that was not written. The reverse
// does not hold: an error that is not ErrChanged — a timeout above all — may come after the write landed, so
// that number may be spent and never handed out. A caller tolerates gaps, never a number handed out twice.
// Deleting the key starts the count again at one; a key whose numbers must stay unique is never deleted.
func (s KeptState) Increment(key string, attempts int) (uint64, error) {
if attempts < 1 {
return 0, fmt.Errorf("an increment of %s is tried at least once: %d attempts", key, attempts)
}
for i := 0; i < attempts; i++ {
e, err := s.Get(key)
if err != nil {
return 0, err
}
var next uint64 = 1
if e == nil {
_, err = s.Create(key, next)
} else {
var n uint64
if string(e.Value) == "null" {
return 0, fmt.Errorf("%s holds null, not a counter", key)
}
if err := json.Unmarshal(e.Value, &n); err != nil {
return 0, fmt.Errorf("%s holds %s, not a counter: %w", key, e.Value, err)
}
if n == ^uint64(0) {
return 0, fmt.Errorf("%s is at the largest number a counter holds", key)
}
next = n + 1
_, err = s.Update(key, next, e.Revision)
}
if err == nil {
return next, nil
}
if !errors.Is(err, ErrChanged) {
return 0, err
}
}
return 0, fmt.Errorf("%s was written by another writer first %d times: %w", key, attempts, ErrChanged)
}
// Keys is every key with a current value, sorted.
func (s KeptState) Keys() ([]string, error) {
raw, err := s.ask("keys", map[string]any{})
+278
View File
@@ -0,0 +1,278 @@
package stdio
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"sync"
"testing"
"time"
)
// casRuntime is the runtime's side of mesh/state.get, create and update as the bus does them (novox/hq ADR
// 0201, ADR 0259): one revision per write across the bucket, create refused when the key has a value,
// update refused unless the key is still at the revision named, each refusal answered with CodeChanged.
// refuse, when set, is asked before each write and may refuse it with an error of its own.
type casRuntime struct {
mu sync.Mutex
revision uint64
keys map[string]StateEntry
writes map[string]int
lost int
refuse func(verb string) (code int, message string, refused bool)
}
func (r *casRuntime) answer(m message) map[string]any {
var p struct {
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Revision uint64 `json:"revision"`
}
_ = json.Unmarshal(m.Params, &p)
r.mu.Lock()
defer r.mu.Unlock()
verb := strings.TrimPrefix(m.Method, "mesh/state.")
lost := func() map[string]any {
r.lost++
return map[string]any{"error": map[string]any{"code": CodeChanged, "message": p.Key + ": the key was written by another writer first"}}
}
if verb != "get" && r.refuse != nil {
if code, msg, refused := r.refuse(verb); refused {
return map[string]any{"error": map[string]any{"code": code, "message": msg}}
}
}
e, has := r.keys[p.Key]
switch verb {
case "get":
if !has {
return map[string]any{"result": nil}
}
return map[string]any{"result": e}
case "create":
if has {
return lost()
}
case "update":
if p.Revision == 0 {
return map[string]any{"error": map[string]any{"code": -32000, "message": "an update names the revision it read"}}
}
if !has || e.Revision != p.Revision {
return lost()
}
case "put":
default:
return map[string]any{"error": map[string]any{"code": -32601, "message": "no " + m.Method}}
}
r.revision++
r.keys[p.Key] = StateEntry{Key: p.Key, Value: p.Value, Revision: r.revision}
r.writes[verb]++
return map[string]any{"result": map[string]any{"revision": r.revision}}
}
// counting serves a bundle whose one tool, "count", runs run, against r; it answers the bundle's asks each
// beside the loop, as the runtime does, so two writers' reads and writes interleave. It returns what the
// tool answered.
func counting(t *testing.T, r *casRuntime, run func() (any, error)) map[string]any {
t.Helper()
t.Setenv(ServedAs, "issues")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() {
_ = ServeOn("", []Tool{{Name: "count", Run: func(map[string]any) (any, error) { return run() }}}, inR, outW)
}()
var wmu sync.Mutex
send := func(m map[string]any) {
m["jsonrpc"] = "2.0"
b, _ := json.Marshal(m)
wmu.Lock()
defer wmu.Unlock()
_, _ = inW.Write(append(b, '\n'))
}
done := make(chan map[string]any, 1)
go func() {
lines := bufio.NewScanner(outR)
lines.Buffer(make([]byte, 1<<20), 1<<24)
for lines.Scan() {
var m message
if json.Unmarshal(lines.Bytes(), &m) != nil {
continue
}
if strings.HasPrefix(m.Method, "mesh/state.") {
go func(m message) {
a := r.answer(m)
a["id"] = m.ID
send(a)
}(m)
continue
}
var answer map[string]any
_ = json.Unmarshal(lines.Bytes(), &answer)
if answer["id"] == float64(1) {
done <- answer
return
}
}
}()
send(map[string]any{"id": 1, "method": "tools/call", "params": map[string]any{"name": "count"}})
t.Cleanup(func() { _ = inW.Close() })
select {
case a := <-done:
return a
case <-time.After(60 * time.Second):
t.Fatal("the bundle did not answer the count")
return nil
}
}
func text(t *testing.T, a map[string]any) (string, bool) {
t.Helper()
res, _ := a["result"].(map[string]any)
content, _ := res["content"].([]any)
if len(content) != 1 {
t.Fatalf("answered %v", a)
}
isError, _ := res["isError"].(bool)
return content[0].(map[string]any)["text"].(string), isError
}
// Two writers incrementing one counter a thousand times each end at two thousand: every number handed out
// once, none lost, each writer's numbers rising — read, then written only at the revision read.
func TestTwoWritersIncrementingOneCounterLoseNoUpdate(t *testing.T) {
r := &casRuntime{keys: map[string]StateEntry{}, writes: map[string]int{}}
const each = 1000
got := counting(t, r, func() (any, error) {
var wg sync.WaitGroup
numbers := make([][]uint64, 2)
errs := make([]error, 2)
for w := range numbers {
wg.Add(1)
go func(w int) {
defer wg.Done()
for i := 0; i < each; i++ {
n, err := State("numbers").Increment("issue", 1000)
if err != nil {
errs[w] = err
return
}
numbers[w] = append(numbers[w], n)
}
}(w)
}
wg.Wait()
if err := errors.Join(errs...); err != nil {
return nil, err
}
seen := map[uint64]bool{}
for w, ns := range numbers {
for i, n := range ns {
if seen[n] {
t.Errorf("the number %d was handed out twice", n)
}
seen[n] = true
if i > 0 && n <= ns[i-1] {
t.Errorf("writer %d was handed %d after %d", w, n, ns[i-1])
}
}
}
return len(seen), nil
})
if out, isError := text(t, got); isError || out != "2000" {
t.Fatalf("two writers counted %s (error: %v)", out, isError)
}
var final uint64
_ = json.Unmarshal(r.keys["issue"].Value, &final)
if final != 2*each {
t.Fatalf("the counter ended at %d, not %d: an increment was lost", final, 2*each)
}
if r.lost == 0 {
t.Fatal("the two writers never collided: the test proved nothing about contention")
}
t.Logf("2000 increments, %d writes lost to the other writer and tried again", r.lost)
if r.writes["create"] != 1 || r.writes["update"] != 2*each-1 || r.writes["put"] != 0 {
t.Fatalf("written by %v: one create, then updates at the revision read, never a put", r.writes)
}
}
// A lost write is read again and tried again, never reported counted; past its attempts it is ErrChanged
// with nothing counted; any other refusal, and a value that is no counter, is returned at once.
func TestAnIncrementThatLosesIsTriedAgainAndNeverReportedCounted(t *testing.T) {
t.Run("lost twice then won", func(t *testing.T) {
lost := 0
r := &casRuntime{keys: map[string]StateEntry{"issue": {Key: "issue", Value: json.RawMessage(`41`), Revision: 7}},
writes: map[string]int{}, revision: 7}
r.refuse = func(string) (int, string, bool) {
if lost < 2 {
lost++
// Another writer was first: the counter moved on by one, to 42 and then 43.
r.revision++
r.keys["issue"] = StateEntry{Key: "issue", Value: json.RawMessage(fmt.Sprint(41 + lost)), Revision: r.revision}
return CodeChanged, "issue: the key was written by another writer first", true
}
return 0, "", false
}
got := counting(t, r, func() (any, error) { return State("numbers").Increment("issue", 3) })
if out, isError := text(t, got); isError || out != "44" {
t.Fatalf("answered %s (error: %v), not 44", out, isError)
}
})
t.Run("lost every time", func(t *testing.T) {
tries := 0
r := &casRuntime{keys: map[string]StateEntry{}, writes: map[string]int{}}
r.refuse = func(string) (int, string, bool) {
tries++
return CodeChanged, "issue: the key was written by another writer first", true
}
var err error
got := counting(t, r, func() (any, error) {
n, e := State("numbers").Increment("issue", 5)
err = e
return n, e
})
if _, isError := text(t, got); !isError || !errors.Is(err, ErrChanged) || tries != 5 {
t.Fatalf("after %d tries: %v", tries, err)
}
})
t.Run("refused otherwise", func(t *testing.T) {
tries := 0
r := &casRuntime{keys: map[string]StateEntry{}, writes: map[string]int{}}
r.refuse = func(string) (int, string, bool) {
tries++
return -32000, "issues keeps no state numbers", true
}
var err error
got := counting(t, r, func() (any, error) {
n, e := State("numbers").Increment("issue", 5)
err = e
return n, e
})
if _, isError := text(t, got); !isError || errors.Is(err, ErrChanged) || tries != 1 {
t.Fatalf("after %d tries: %v", tries, err)
}
})
t.Run("no counter", func(t *testing.T) {
r := &casRuntime{keys: map[string]StateEntry{"issue": {Key: "issue", Value: json.RawMessage(`"x"`), Revision: 1}},
writes: map[string]int{}, revision: 1}
got := counting(t, r, func() (any, error) { return State("numbers").Increment("issue", 5) })
if _, isError := text(t, got); !isError || len(r.writes) != 0 {
t.Fatalf("a value that is no counter was counted: %v, writes %v", got, r.writes)
}
})
t.Run("null", func(t *testing.T) {
r := &casRuntime{keys: map[string]StateEntry{"issue": {Key: "issue", Value: json.RawMessage(`null`), Revision: 1}},
writes: map[string]int{}, revision: 1}
got := counting(t, r, func() (any, error) { return State("numbers").Increment("issue", 5) })
if _, isError := text(t, got); !isError || len(r.writes) != 0 {
t.Fatalf("a null was counted from zero: %v, writes %v", got, r.writes)
}
})
t.Run("no attempts", func(t *testing.T) {
r := &casRuntime{keys: map[string]StateEntry{}, writes: map[string]int{}}
got := counting(t, r, func() (any, error) { return State("numbers").Increment("issue", 0) })
if _, isError := text(t, got); !isError || len(r.writes) != 0 {
t.Fatalf("zero attempts counted: %v", got)
}
})
}
+142
View File
@@ -0,0 +1,142 @@
package stdio
import (
"bufio"
"encoding/json"
"io"
"strings"
"sync"
"testing"
"time"
)
// historying serves a bundle whose one tool, "history", runs run, answering each mesh/state ask with
// answer beside the loop, as the runtime does; it returns what the bundle asked.
func historying(t *testing.T, answer func(message) map[string]any, run func()) []message {
t.Helper()
t.Setenv(ServedAs, "issues")
inR, inW := io.Pipe()
outR, outW := io.Pipe()
go func() {
_ = ServeOn("", []Tool{{Name: "history", Run: func(map[string]any) (any, error) { run(); return "done", nil }}}, inR, outW)
}()
var wmu, amu sync.Mutex
var asked []message
send := func(m map[string]any) {
m["jsonrpc"] = "2.0"
b, _ := json.Marshal(m)
wmu.Lock()
defer wmu.Unlock()
_, _ = inW.Write(append(b, '\n'))
}
done := make(chan struct{})
go func() {
lines := bufio.NewScanner(outR)
for lines.Scan() {
var m message
if json.Unmarshal(lines.Bytes(), &m) != nil {
continue
}
if strings.HasPrefix(m.Method, "mesh/state.") {
amu.Lock()
asked = append(asked, m)
amu.Unlock()
a := answer(m)
a["id"] = m.ID
send(a)
continue
}
if m.Method == "" {
close(done)
return
}
}
}()
send(map[string]any{"id": 1, "method": "tools/call", "params": map[string]any{"name": "history"}})
t.Cleanup(func() { _ = inW.Close() })
select {
case <-done:
case <-time.After(10 * time.Second):
t.Fatal("the bundle did not answer")
}
amu.Lock()
defer amu.Unlock()
return asked
}
// History asks mesh/state.history with the state and the key alone, and hands back every kept value oldest
// first, a delete with its null value; nothing kept is an empty history; a refusal is the runtime's error
// (novox/hq ADR 0307).
func TestHistoryIsEveryKeptValueOldestFirst(t *testing.T) {
kept := []map[string]any{
{"key": "401", "value": map[string]any{"status": "open"}, "revision": 4, "op": "put", "at": "2026-10-10T19:46:00Z"},
{"key": "401", "value": map[string]any{"status": "fixing"}, "revision": 9, "op": "put", "at": "2026-10-11T08:01:02Z"},
{"key": "401", "value": nil, "revision": 12, "op": "delete", "at": "2026-10-11T09:30:00+02:00"},
}
var got, none, null []StateRevision
var refused error
asked := historying(t, func(m message) map[string]any {
var p map[string]any
_ = json.Unmarshal(m.Params, &p)
switch {
case p["state"] != "issues":
return map[string]any{"error": map[string]any{"code": -32000, "message": "issues was issued no state " + p["state"].(string)}}
case p["key"] == "401":
return map[string]any{"result": kept}
case p["key"] == "402":
return map[string]any{"result": nil}
}
return map[string]any{"result": []any{}}
}, func() {
var err error
if got, err = State("issues").History("401"); err != nil {
t.Errorf("history of 401: %v", err)
}
if none, err = State("issues").History("999"); err != nil {
t.Errorf("history of 999: %v", err)
}
if null, err = State("issues").History("402"); err != nil {
t.Errorf("history of 402: %v", err)
}
var h []StateRevision
h, refused = State("elsewhere.issues").History("401")
if h != nil {
t.Errorf("a refused history handed back %v", h)
}
})
if len(asked) != 4 {
t.Fatalf("asked %d times, not 4", len(asked))
}
var p map[string]any
_ = json.Unmarshal(asked[0].Params, &p)
if asked[0].Method != "mesh/state.history" || p["state"] != "issues" || p["key"] != "401" || len(p) != 2 {
t.Fatalf("asked %s %v", asked[0].Method, p)
}
if len(got) != 3 {
t.Fatalf("history of 401 is %d values, not 3: %+v", len(got), got)
}
want := []struct {
rev uint64
op string
value string
at time.Time
}{
{4, "put", `{"status":"open"}`, time.Date(2026, 10, 10, 19, 46, 0, 0, time.UTC)},
{9, "put", `{"status":"fixing"}`, time.Date(2026, 10, 11, 8, 1, 2, 0, time.UTC)},
{12, "delete", `null`, time.Date(2026, 10, 11, 7, 30, 0, 0, time.UTC)},
}
for i, w := range want {
r := got[i]
if r.Key != "401" || r.Revision != w.rev || r.Op != w.op || string(r.Value) != w.value || !r.At.Equal(w.at) {
t.Fatalf("value %d is %+v, not %+v", i, r, w)
}
}
if none == nil || len(none) != 0 || null == nil || len(null) != 0 {
t.Fatalf("nothing kept read as %v and %v, not an empty history", none, null)
}
if refused == nil || !strings.Contains(refused.Error(), "issued no state elsewhere.issues") {
t.Fatalf("a state not issued answered %v", refused)
}
}
+33 -5
View File
@@ -39,7 +39,8 @@ type Tool struct {
Description string
// Input is the JSON schema of the arguments; nil is a tool that takes nothing.
Input map[string]any
Run func(args map[string]any) (any, error)
// Run answers one call. Who called is CallerOf(args): the bus user the runtime proved, or Unproven.
Run func(args map[string]any) (any, error)
}
type message struct {
@@ -65,6 +66,8 @@ type channel struct {
subscribed int
// watches are this bundle's state watches by the id each was asked with (ADR 0201).
watches map[string]func(StateChange) error
// provisioned are this bundle's watches of its values, by the id each was asked with (provision.go).
provisioned map[string]func(ProvisionChange) error
// takers and provers are this bundle's seat work and proof handlers, by worker and subject (ADR 0259).
takers map[string]func(Work) error
provers map[string]func(subject string, body json.RawMessage) (any, error)
@@ -168,7 +171,8 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
byName[t.Name] = t
}
c := &channel{enc: json.NewEncoder(out), waiting: map[string]chan message{}, watches: map[string]func(StateChange) error{},
takers: map[string]func(Work) error{}, provers: map[string]func(string, json.RawMessage) (any, error){}}
provisioned: map[string]func(ProvisionChange) error{},
takers: map[string]func(Work) error{}, provers: map[string]func(string, json.RawMessage) (any, error){}}
currentMu.Lock()
current = c
currentMu.Unlock()
@@ -223,6 +227,8 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
var p struct {
Name string `json:"name"`
Arguments map[string]any `json:"arguments"`
// Meta is the runtime's, never the caller's: who called, taken from the subject (CallerOf).
Meta map[string]any `json:"_meta"`
}
_ = json.Unmarshal(m.Params, &p)
t, ok := byName[p.Name]
@@ -232,18 +238,18 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
}
// Beside the loop, never in it: a tool that emits waits for the runtime's answer, which
// arrives on the line this loop reads next — and calls do not queue behind a slow one.
go func(id any, t Tool, args map[string]any) {
go func(id any, t Tool, args map[string]any, caller string) {
if args == nil {
args = map[string]any{}
}
result, err := t.Run(args)
result, err := runAs(t, caller, args)
if err != nil {
c.say(map[string]any{"id": id, "result": map[string]any{"content": []map[string]any{{"type": "text", "text": err.Error()}}, "isError": true}})
return
}
text, _ := json.Marshal(result)
c.say(map[string]any{"id": id, "result": map[string]any{"content": []map[string]any{{"type": "text", "text": string(text)}}}})
}(id, t, p.Arguments)
}(id, t, p.Arguments, callerIn(p.Meta))
case "mesh/event":
// One event the runtime read from this module's consumer (ADR 0198), answered once every
// handler it is for has taken it; an error is answered as one, so the runtime does not
@@ -294,6 +300,28 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
}
c.say(map[string]any{"id": id, "result": map[string]any{}})
}(id, *p.Change)
case "mesh/provision":
// One change to a value of this module's that this bundle watches (provision.go), named by the
// watch's id, answered once its handler has taken it. Beside the loop, like a tool call.
var p struct {
Watch string `json:"watch"`
Change *ProvisionChange `json:"change"`
}
_ = json.Unmarshal(m.Params, &p)
c.mu.Lock()
handle := c.provisioned[p.Watch]
c.mu.Unlock()
if handle == nil || p.Change == nil {
c.say(map[string]any{"id": id, "error": map[string]any{"code": -32602, "message": "mesh/provision names no watch of this bundle"}})
continue
}
go func(id any, change ProvisionChange) {
if err := handle(change); err != nil {
c.say(map[string]any{"id": id, "error": map[string]any{"code": -32000, "message": err.Error()}})
return
}
c.say(map[string]any{"id": id, "result": map[string]any{}})
}(id, *p.Change)
case "mesh/work":
// One piece of a seat's work this bundle takes (novox/hq ADR 0259): taken by an answer that is
// not an error, offered again after one. Beside the loop, like a tool call.
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@novox/mesh-sdk",
"version": "0.1.13",
"version": "0.1.14",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@novox/mesh-sdk",
"version": "0.1.13",
"version": "0.1.14",
"devDependencies": {
"@types/node": "^22.0.0",
"typescript": "^5.6.0"
+5 -3
View File
@@ -1,18 +1,20 @@
{
"name": "@novox/mesh-sdk",
"version": "0.1.13",
"version": "0.1.14",
"description": "The stable spine a Novox Mesh module's own code builds against.",
"type": "module",
"exports": {
".": "./dist/index.js",
"./contracts": "./dist/contracts/index.js",
"./provisioner": "./dist/provisioner/index.js",
"./provider-loop": "./dist/provider-loop/index.js",
"./provisioner": "./dist/provider-loop/index.js",
"./tools": "./dist/tools/index.js",
"./messaging": "./dist/messaging/index.js",
"./events": "./dist/events/index.js",
"./primitives": "./dist/primitives/index.js",
"./stdio": "./dist/stdio/index.js",
"./state": "./dist/state/index.js"
"./state": "./dist/state/index.js",
"./seal": "./dist/seal/index.js"
},
"files": [
"dist"
+5 -2
View File
@@ -1,13 +1,16 @@
// @novox/mesh-sdk — the stable spine a module's own code builds against (novox/hq ADR 0039).
//
// Import the area you need directly (`@novox/mesh-sdk/tools`, `/provisioner`, …) or the whole
// Import the area you need directly (`@novox/mesh-sdk/tools`, `/provider-loop`, …) or the whole
// surface from here. Everything specific to one module — its API client, its tool implementations,
// its create-a-resource adapter — lives in the module, never here.
export * from "./contracts/index.js";
export * as tools from "./tools/index.js";
export * as provisioner from "./provisioner/index.js";
export * as providerLoop from "./provider-loop/index.js";
/** @deprecated The provider loop's name before design 57 (novox/hq issue 678): `providerLoop`. */
export * as provisioner from "./provider-loop/index.js";
export * as messaging from "./messaging/index.js";
export * as events from "./events/index.js";
export * as primitives from "./primitives/index.js";
export * as state from "./state/index.js";
export * as seal from "./seal/index.js";
@@ -1,3 +1,13 @@
// The provider loop: the reconcile harness every provider on the controller's grant files shares.
// Called `provisioner` until design 57 (novox/hq issue 678): that word is mesh-provisioner's, which
// places every value a module needs (ADR 0349), so the loop is named for what it is. The old path
// `@novox/mesh-sdk/provisioner` and the names `runProvisioner`, `provisionerTools`, `Provisioner` and
// `ProvisionerOptions` stay as aliases of these until the providers that import them move to the Go
// helper that answers consumers over the bus (mesh-sdk go/provider, design 57 B2 and B3) and this loop
// retires with the grant files. The events (`provisioner.failing`, `.recovered`, `.retirement`) and the
// tool names (`provisioner_retirement`, …) are what the controller and the operator read on the bus,
// and keep their names.
//
// The reconcile harness every provider shares. The loop below is identical for postgres, redis,
// minio and umami: read the contributions the mesh delivered, bring each consumer's resource into
// being through the provider's adapter under the login and password the mesh minted, and retire
@@ -10,10 +20,20 @@
// does not hand one back — the consumer already receives its copy through the mesh's own channel.
// This is why the old $MESH_SEAL_KEY, the symmetric seal, and the *.grant.json / *.credential files
// are gone: they described a second, contradictory credential path the mesh does not have.
//
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
// whose create, check or secret has failed without one success in between for `failingAfterMs` is
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
// again every `sayAgainEveryMs` while it lasts; the first success after that is
// `provisioner.recovered`, as is a consumer the mesh stopped asking for, and the first success for
// each consumer since this process started. The controller keeps the newest per provider and consumer
// and `status` names it. Until novox/hq issue 272 only the Go copy of this loop (postgres, keycloak)
// said it; every provider on this one failed its consumers in its journal alone.
import { readFile } from "node:fs/promises";
import type { ToolDefinition } from "../contracts/index.js";
import { emit } from "../events/index.js";
import { registerModuleTools } from "../tools/index.js";
/** One consumer's resource to bring into being — everything the mesh derived and delivered. */
@@ -86,6 +106,9 @@ export interface Adapter {
/** Optional: every consumer the backend holds ACTIVE that the mesh made, read at start, so one the
* mesh stopped asking for while this process was down is still retired (by the same rules). */
held?(): Promise<string[]>;
/** Optional: the class of one of the adapter's own errors (novox/hq ADR 0224), when its code knows
* better than the error's words; "" or undefined falls back to `classOf` of the text. */
classify?(err: unknown): string | undefined;
}
/** One retired consumer, as `provisioner_retirement` answers it. */
@@ -127,9 +150,9 @@ export interface Act {
readonly via?: string;
}
/** The running provisioner: a stop function, as before, that also answers and takes the person's
/** The running provider loop: a stop function, as before, that also answers and takes the person's
* acts on retirement (novox/hq ADR 0230). */
export type Provisioner = (() => void) & {
export type ProviderLoop = (() => void) & {
retirement(): Promise<Retirement>;
approve(consumers: readonly string[], act: Act): Promise<{ retired: string[] }>;
reject(consumers: readonly string[], act: Act): Promise<{ kept: string[] }>;
@@ -139,7 +162,22 @@ export type Provisioner = (() => void) & {
/** The one event a provider says about retirement (novox/hq ADR 0230). */
export const RETIREMENT_EVENT = "provisioner.retirement";
export interface ProvisionerOptions {
/** The events a provider's standing toward one consumer is announced as (novox/hq ADR 0224). The
* controller derives the permission to emit them for every module that receives contributions. */
export const FAILING_EVENT = "provisioner.failing";
export const RECOVERED_EVENT = "provisioner.recovered";
/** The classes a failing consumer is announced with: what a person reading `status` needs to know
* before reading the journal (novox/hq ADR 0224). An adapter may say better (`classify`). */
export const CLASS_CREDENTIALS = "credentials-rejected";
export const CLASS_UNREACHABLE = "unreachable";
export const CLASS_SECRET = "secret-unreadable";
export const CLASS_REFUSED = "refused";
/** How long a consumer fails without one success before it is announced failing (ADR 0224). */
export const FAILING_AFTER_MS = 5 * 60_000;
export interface ProviderLoopOptions {
/** The contributions file the mesh writes for this resource (the module's `receives` path).
* Defaults to $MESH_RECEIVES. */
receives?: string;
@@ -165,10 +203,16 @@ export interface ProvisionerOptions {
/** Nor a set larger than this share of the consumers held, where more than one is held. Defaults
* to 0.5. */
retireFraction?: number;
/** How often a waiting retirement is said again, in ms. Defaults to 15 minutes. */
/** How often a waiting retirement, and a consumer still failing, is said again, in ms. Defaults to
* 15 minutes. */
sayAgainEveryMs?: number;
/** Publishes one of the provider's events (`provisioner.retirement`); usually the SDK's `emit`.
* Without it the provider's retirement is said in its journal only. */
/** How long, in ms, a consumer fails without one success before it is announced
* `provisioner.failing` (novox/hq ADR 0224). Defaults to FAILING_AFTER_MS, five minutes. */
failingAfterMs?: number;
/** Publishes one of the provider's events (`provisioner.failing`, `provisioner.recovered`,
* `provisioner.retirement`). Defaults to the SDK's `emit`, so every provider says them on the bus
* without asking (novox/hq issue 272); a failed publish is said in the journal and never fails
* the act it reports. */
announce?: (event: string, body: Record<string, unknown>) => Promise<void> | void;
/** The machine this provider runs on, said in each event. Defaults to $MESH_NODE. */
node?: string;
@@ -216,7 +260,7 @@ interface Contribution {
* `retireFraction` of those held where more than one is held, is not retired without a person: the
* provider waits for approve or reject. RETIRED → DELETED only through `provisioner_delete`.
*/
export function runProvisioner(resource: string, adapter: Adapter, opts: ProvisionerOptions = {}): Provisioner {
export function runProviderLoop(resource: string, adapter: Adapter, opts: ProviderLoopOptions = {}): ProviderLoop {
const receives = opts.receives ?? envOrThrow("MESH_RECEIVES");
const everyMs = opts.everyMs ?? 5000;
const verifyEveryMs = opts.verifyEveryMs ?? 60_000;
@@ -227,6 +271,8 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
const retireAtOnce = opts.retireAtOnce ?? 3;
const retireFraction = opts.retireFraction ?? 0.5;
const sayAgainEveryMs = opts.sayAgainEveryMs ?? 15 * 60_000;
const failingAfterMs = opts.failingAfterMs ?? FAILING_AFTER_MS;
const announce = opts.announce ?? ((event: string, body: Record<string, unknown>) => emit(event, body));
const node = opts.node ?? process.env.MESH_NODE ?? "";
const say = (line: string): void => console.error(`[provisioner:${resource}] ${line}`);
// Retirement (novox/hq ADR 0230). The unasked set last seen and in how many consecutive readable
@@ -245,6 +291,13 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
// itself. Each repeat doubles the wait before asking again, so such a bug costs one re-apply an
// hour rather than one a minute, and it is said loudly rather than quietly repeated.
const lost = new Map<string, { times: number; nextAt: number }>();
// Each consumer's unbroken run of failures — since when, how often, the last error, and when it was
// last announced (0: not yet) — and every consumer that has succeeded since this process started
// (novox/hq ADR 0224).
const trouble = new Map<string, {
node?: string; since: number; attempts: number; cls: string; text: string; saidAt: number;
}>();
const cleared = new Set<string>();
// login (`as`) -> what was last applied: the hash that says whether it changed, and the
// derivation to hand back on withdrawal, which by then is no longer in the file.
@@ -311,6 +364,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
`file and who this process runs as (novox/hq issue 225)`,
);
}
await failed(g.as, g.node, CLASS_SECRET, `secret not readable (${g.secret}): ${err}`);
continue;
}
waiting.delete(g.as);
@@ -328,11 +382,15 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
try {
if (await withTimeout(adapter.holds!(p), holdsTimeoutMs)) {
lost.delete(g.as);
await succeeded(g.as);
continue;
}
} catch (err) {
// Unable to ask is not evidence of loss. Kept as applied, asked again next time.
console.error(`[provisioner:${resource}] ${g.as}: could not check the backend, will ask again: ${scrub(err, password)}`);
// Unable to ask is not evidence of loss. Kept as applied, asked again next time — but a
// check that keeps failing is a consumer failed all the same (ADR 0224, issue 179).
const text = scrub(err, password);
console.error(`[provisioner:${resource}] ${g.as}: could not check the backend, will ask again: ${text}`);
await failed(g.as, g.node, classed(err, text), text);
// A backend that did not answer in time will not answer for the next consumer either, and
// each would cost a timeout, so the rest of this pass is not asked. Any other failure may
// be this consumer's alone, and the others are still asked.
@@ -356,6 +414,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
await adapter.reenable({ as: g.as, derived: was.derived ?? p.derived });
}
await adapter.create(p);
await succeeded(g.as);
applied.set(g.as, { hash: h, derived: p.derived, node: g.node });
if (retired.has(g.as)) {
// Asked for again: the ordinary create re-enabled it and cleared its mark (ADR 0230).
@@ -376,7 +435,9 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
}
}
} catch (err) {
console.error(`[provisioner:${resource}] ${g.as}: create failed, will retry: ${scrub(err, password)}`);
const text = scrub(err, password);
console.error(`[provisioner:${resource}] ${g.as}: create failed, will retry: ${text}`);
await failed(g.as, g.node, classed(err, text), text);
// A failed create is not a disagreement: asked again at the next check, the count unchanged.
if (reapplying !== undefined) lost.set(g.as, { times: reapplying - 1, nextAt: 0 });
}
@@ -386,6 +447,12 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
// `stablePasses` passes, and within the bound, or with a person's word (novox/hq ADR 0230).
// Only consumers this harness applied, or the backend says the mesh made, are touched.
const unasked = [...applied.keys()].filter((as) => !wantByAs.has(as)).sort();
// A consumer the mesh stopped asking for that this provider holds nothing for is no longer failed
// by anyone: said, so the standing the controller keeps for it is cleared rather than left naming a
// consumer that is gone. One still held is said recovered when it is retired (ADR 0224, 0230).
for (const as of [...trouble.keys()]) {
if (!wantByAs.has(as) && !applied.has(as)) await recovered(as, "no longer asked for");
}
const key = unasked.join(",");
if (pending && pending.consumers.join(",") !== key) {
say(`the retirement waiting for a person (${pending.consumers.join(", ")}) is no longer what the mesh says; settled, nothing retired`);
@@ -464,6 +531,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
}
applied.delete(as);
lost.delete(as);
await recovered(as, "retired");
retired.set(as, { node: was.node, at: at.getTime(), why, derived: was.derived, access: adapter.retire ? "disabled" : "kept" });
done.push(as);
} catch (err) {
@@ -485,16 +553,86 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
// Said on the bus, never failing the act it reports (the journal still has it).
async function tell(change: string, body: Record<string, unknown>): Promise<void> {
if (!opts.announce) return;
await said(RETIREMENT_EVENT, {
provider: resource, "provider-node": node, change, bound: bound(applied.size), at: iso(Date.now()), ...body,
}, change);
}
async function said(event: string, body: Record<string, unknown>, what = ""): Promise<void> {
try {
await opts.announce(RETIREMENT_EVENT, {
provider: resource, "provider-node": node, change, bound: bound(applied.size), at: iso(Date.now()), ...body,
});
await announce(event, body);
} catch (err) {
say(`could not announce ${RETIREMENT_EVENT} ${change}: ${err}`);
say(`could not announce ${event}${what ? ` ${what}` : ""}: ${err}`);
}
}
// An error's class: the adapter's word when it has one, else read from the text (ADR 0224).
function classed(err: unknown, text: string): string {
try {
const own = adapter.classify?.(err);
if (own) return own;
} catch {
// An adapter that cannot class its own error is read by the words.
}
// A check that did not answer in time is a backend out of reach, whatever its words.
if (err instanceof HoldsTimeout) return CLASS_UNREACHABLE;
return classOf(text);
}
// One more failure in a consumer's unbroken run, announced once the run has lasted failingAfterMs,
// then again every sayAgainEveryMs while it lasts (novox/hq ADR 0224).
async function failed(as: string, consumerNode: string | undefined, cls: string, text: string): Promise<void> {
const now = Date.now();
let s = trouble.get(as);
if (!s) {
s = { since: now, attempts: 0, cls, text, saidAt: 0 };
trouble.set(as, s);
}
s.node = consumerNode;
s.cls = cls;
s.text = text;
s.attempts++;
if (now - s.since < failingAfterMs) return;
if (s.saidAt !== 0 && now - s.saidAt < sayAgainEveryMs) return;
if (s.saidAt === 0) {
say(
`${as}: FAILING for ${Math.round((now - s.since) / 1000)}s (${s.attempts} attempts, ${cls}): ${text}. ` +
`Announced as ${FAILING_EVENT}; \`status\` names it until it succeeds (novox/hq ADR 0224)`,
);
}
s.saidAt = now;
await said(FAILING_EVENT, {
provider: resource, "provider-node": node, consumer: as, node: consumerNode ?? "",
class: cls, error: clip(text), since: iso(s.since), attempts: s.attempts,
});
}
// A success ends a consumer's run of failures; one that was announced is announced recovered. And
// the first success for a consumer since this process started is announced too, failing or not: a
// provider that announced a failure and was restarted has forgotten it, and without this the
// controller would name the consumer failing for ever after it recovered unheard.
async function succeeded(as: string): Promise<void> {
if (!trouble.has(as) && !cleared.has(as)) {
cleared.add(as);
await said(RECOVERED_EVENT, { provider: resource, "provider-node": node, consumer: as, why: "first-success" });
return;
}
cleared.add(as);
await recovered(as, "");
}
async function recovered(as: string, why: string): Promise<void> {
const s = trouble.get(as);
if (!s) return;
trouble.delete(as);
if (s.saidAt === 0) return; // never announced, so there is nothing to take back
if (!why) say(`${as}: recovered after ${Math.round((Date.now() - s.since) / 1000)}s and ${s.attempts} failed attempt(s)`);
await said(RECOVERED_EVENT, {
provider: resource, "provider-node": node, consumer: as, node: s.node ?? "",
since: iso(s.since), attempts: s.attempts, ...(why ? { why } : {}),
});
}
// What the backend holds that the mesh made: active ones become held, so one the mesh stopped asking
// for while this process was down is still retired, by the same rules; retired ones are known.
async function seed(): Promise<void> {
@@ -629,8 +767,8 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
const stop = () => {
stopped = true;
};
const handle: Provisioner = Object.assign(stop, { retirement, approve, reject, deleteRetired });
if (opts.module) registerModuleTools(opts.module, () => provisionerTools(handle));
const handle: ProviderLoop = Object.assign(stop, { retirement, approve, reject, deleteRetired });
if (opts.module) registerModuleTools(opts.module, () => providerLoopTools(handle));
return handle;
}
@@ -639,7 +777,7 @@ const text = (v: unknown): string => (typeof v === "string" ? v : "");
/** The four tools every provider serves for retirement (novox/hq ADR 0230), the controller's `retire`
* and `cleanup` verbs asking them per machine. */
export function provisionerTools(p: Provisioner): ToolDefinition[] {
export function providerLoopTools(p: ProviderLoop): ToolDefinition[] {
const act = { type: "string" };
return [
{
@@ -713,6 +851,27 @@ async function readContributions(
return (doc.given as Contribution[]).filter((g) => g.as && g.secret);
}
/** An error's class read from its text — the words the backends the mesh runs use (novox/hq ADR
* 0224): the same words, in the same order, as the Go copy of this loop reads. */
export function classOf(text: string): string {
const t = text.toLowerCase();
const credentials = ["invalid_grant", "invalid user credentials", "password authentication failed",
"authentication failed", "unauthorized", " 401"];
if (credentials.some((w) => t.includes(w))) return CLASS_CREDENTIALS;
const unreachable = ["connection refused", "no such host", "i/o timeout", "deadline exceeded",
"connection reset", "network is unreachable", "no route to host", "eof",
// Node's own words for the same faults.
"econnrefused", "enotfound", "etimedout", "econnreset", "ehostunreach", "enetunreach", "eai_again"];
if (unreachable.some((w) => t.includes(w))) return CLASS_UNREACHABLE;
return CLASS_REFUSED;
}
/** An announced error kept to what belongs in a status line. */
function clip(text: string): string {
const most = 300;
return text.length <= most ? text : `${text.slice(0, most)}…`;
}
/** A check that did not answer in time: the backend, not the consumer, is the likely cause. */
class HoldsTimeout extends Error {}
@@ -751,3 +910,12 @@ function envOrThrow(name: string): string {
if (!v) throw new Error(`${name} is not set — the provisioner cannot run without it`);
return v;
}
/** @deprecated The provider loop's name before design 57: {@link runProviderLoop}. */
export const runProvisioner = runProviderLoop;
/** @deprecated {@link providerLoopTools}. */
export const provisionerTools = providerLoopTools;
/** @deprecated {@link ProviderLoop}. */
export type Provisioner = ProviderLoop;
/** @deprecated {@link ProviderLoopOptions}. */
export type ProvisionerOptions = ProviderLoopOptions;
+90
View File
@@ -0,0 +1,90 @@
// The sealed box itself (novox/hq ADR 0183, ADR 0206), with its two sources of randomness as arguments so
// the tests can prove it byte for byte against the Go package's vectors. Modules import `./index.js`.
import {
createCipheriv,
createDecipheriv,
createPrivateKey,
createPublicKey,
diffieHellman,
hkdfSync,
type KeyObject,
} from "node:crypto";
/** A value sealed to one recipient, as it travels in JSON; every field but `v` is standard base64. */
export interface SealedBox {
readonly v: number;
/** The one-time public key, SPKI DER. */
readonly eph: string;
readonly iv: string;
readonly tag: string;
readonly ct: string;
}
/** The only box version `open` opens. */
export const VERSION = 1;
const INFO = "novox-mesh sealed box v1";
const IV_BYTES = 12;
const TAG_BYTES = 16;
function x25519(key: KeyObject, what: string): KeyObject {
if (key.asymmetricKeyType !== "x25519") throw new Error(`not an X25519 ${what} key`);
return key;
}
export function publicFromPem(pem: string): KeyObject {
return x25519(createPublicKey({ key: pem, format: "pem" }), "public");
}
export function privateFromPem(pem: string): KeyObject {
return x25519(createPrivateKey({ key: pem, format: "pem" }), "private");
}
/** The recipient's raw 32-byte public key: the last 32 bytes of its SPKI DER. */
function raw(pub: KeyObject): Buffer {
const der = pub.export({ format: "der", type: "spki" });
return der.subarray(der.length - 32);
}
function boxKey(secret: Buffer, ephDer: Buffer, recipientRaw: Buffer): Buffer {
return Buffer.from(hkdfSync("sha256", secret, Buffer.concat([ephDer, recipientRaw]), INFO, 32));
}
/** `seal` with the one-time private key and the iv given. */
export function sealWith(plaintext: string, recipientPem: string, eph: KeyObject, iv: Buffer): SealedBox {
const recipient = publicFromPem(recipientPem);
x25519(eph, "one-time");
if (iv.length !== IV_BYTES) throw new Error(`an iv of ${iv.length} bytes, not ${IV_BYTES}`);
const secret = diffieHellman({ privateKey: eph, publicKey: recipient });
const ephDer = createPublicKey(eph).export({ format: "der", type: "spki" });
const cipher = createCipheriv("aes-256-gcm", boxKey(secret, ephDer, raw(recipient)), iv);
const ct = Buffer.concat([cipher.update(plaintext, "utf8"), cipher.final()]);
return {
v: VERSION,
eph: ephDer.toString("base64"),
iv: iv.toString("base64"),
tag: cipher.getAuthTag().toString("base64"),
ct: ct.toString("base64"),
};
}
/** Open a box with the recipient's PEM private key; throws for a box to another key, changed, or of another version. */
export function open(box: SealedBox, privatePem: string): string {
if (box.v !== VERSION) throw new Error(`a sealed box of version ${box.v}, not ${VERSION}`);
const priv = privateFromPem(privatePem);
const ephDer = Buffer.from(box.eph, "base64");
const eph = x25519(createPublicKey({ key: ephDer, format: "der", type: "spki" }), "one-time");
const iv = Buffer.from(box.iv, "base64");
const tag = Buffer.from(box.tag, "base64");
// node accepts a shorter tag than 16 bytes; the box never has one, so a short one is refused here.
if (iv.length !== IV_BYTES || tag.length !== TAG_BYTES) throw new Error("the box does not open with this key");
const secret = diffieHellman({ privateKey: priv, publicKey: eph });
const decipher = createDecipheriv("aes-256-gcm", boxKey(secret, ephDer, raw(createPublicKey(priv))), iv);
decipher.setAuthTag(tag);
try {
return Buffer.concat([decipher.update(Buffer.from(box.ct, "base64")), decipher.final()]).toString("utf8");
} catch {
throw new Error("the box does not open with this key");
}
}
+29
View File
@@ -0,0 +1,29 @@
// Sealing to one recipient (novox/hq ADR 0183, ADR 0206): X25519 for the agreement, HKDF-SHA256 for the
// key, AES-256-GCM for the box. The TypeScript twin of the Go package `go/seal`, byte for byte: the same
// fixtures prove both. A box is `{ v: 1, eph, iv, tag, ct }`; keys travel as PEM (SPKI public, PKCS #8
// private).
import { generateKeyPairSync, randomBytes } from "node:crypto";
import { sealWith, open, VERSION, type SealedBox } from "./box.js";
export { open, VERSION, type SealedBox };
/** A recipient's keypair as the two PEM strings it is kept and sent as. */
export interface KeyPair {
readonly publicKey: string;
readonly privateKey: string;
}
/** A fresh X25519 keypair, PEM-encoded. */
export function generateKeyPair(): KeyPair {
const { publicKey, privateKey } = generateKeyPairSync("x25519");
return {
publicKey: publicKey.export({ format: "pem", type: "spki" }) as string,
privateKey: privateKey.export({ format: "pem", type: "pkcs8" }) as string,
};
}
/** Seal plaintext to the recipient's PEM public key, with a fresh one-time key and iv. */
export function seal(plaintext: string, recipientPem: string): SealedBox {
return sealWith(plaintext, recipientPem, generateKeyPairSync("x25519").privateKey, randomBytes(12));
}
+8 -7
View File
@@ -8,7 +8,7 @@ import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { runProvisioner, provisionerTools, RETIREMENT_EVENT, type Provisioner } from "../dist/provisioner/index.js";
import { runProviderLoop, providerLoopTools, RETIREMENT_EVENT, type ProviderLoop } from "../dist/provider-loop/index.js";
import { resetTools, collectTools } from "../dist/tools/index.js";
const EVERY = 10;
@@ -32,7 +32,7 @@ interface World {
reenabled: string[];
events: Record<string, unknown>[];
said: string[];
p: Provisioner;
p: ProviderLoop;
done(): void;
}
@@ -62,9 +62,10 @@ async function world(
...(opts.retire === false ? {} : { async retire(p: { as: string; why: string }) { w.retired.push({ as: p.as, why: p.why }); } }),
...(opts.held ? { async held() { return opts.held!; } } : {}),
};
w.p = runProvisioner("postgres-database", adapter, {
w.p = runProviderLoop("postgres-database", adapter, {
receives, everyMs: EVERY, node: "anchor", stableForMs: opts.stableForMs ?? 0,
announce: (event, body) => { assert.equal(event, RETIREMENT_EVENT); w.events.push(body); },
// Only retirement is these tests'; the standing (provisioner.failing, .recovered) is standing.test.ts's.
announce: (event, body) => { if (event === RETIREMENT_EVENT) w.events.push(body); },
});
// A pass is counted by the file being read: wait wall time for n passes and some slack.
w.done = () => { w.p(); console.error = was; };
@@ -167,7 +168,7 @@ test("five passes are not enough: the same set must also hold for stableForMs",
});
test("the default hold is ten minutes", async () => {
const { STABLE_FOR_MS } = await import("../dist/provisioner/index.js");
const { STABLE_FOR_MS } = await import("../dist/provider-loop/index.js");
assert.equal(STABLE_FOR_MS, 10 * 60_000);
});
@@ -326,7 +327,7 @@ test("the four tools are registered for the module and answer", async () => {
const dir = await mkdtemp(join(tmpdir(), "retire-tools-"));
const receives = join(dir, "r.json");
await writeFile(receives, JSON.stringify({ requirement: "r", given: [] }));
const p = runProvisioner("r", { async create() {}, async remove() {} }, { receives, everyMs: EVERY, module: "postgres" });
const p = runProviderLoop("r", { async create() {}, async remove() {} }, { receives, everyMs: EVERY, module: "postgres" });
try {
const mine = collectTools({}).find((c) => c.module === "postgres");
assert.deepEqual(mine?.tools.map((t) => t.name), [
@@ -335,7 +336,7 @@ test("the four tools are registered for the module and answer", async () => {
const answer = (await mine!.tools[0].run({})) as { resource: string; waiting: unknown };
assert.equal(answer.resource, "r");
assert.equal(answer.waiting, null);
assert.equal(provisionerTools(p).length, 4);
assert.equal(providerLoopTools(p).length, 4);
} finally {
p();
resetTools();
+26 -13
View File
@@ -6,7 +6,7 @@ import { join } from "node:path";
import { compareVersions } from "../dist/primitives/index.js";
import { registerModuleTools, collectTools, collectToolsEach, resetTools, serveTools, listTools, invokeTool } from "../dist/tools/index.js";
import { runProvisioner } from "../dist/provisioner/index.js";
import { runProviderLoop } from "../dist/provider-loop/index.js";
import { emit, on, type Event } from "../dist/events/index.js";
import { useBroker } from "../dist/messaging/index.js";
@@ -47,7 +47,7 @@ test("provisioner creates each consumer with the mesh's login and password; remo
doc([{ from: "webapp", node: "anchor", as: "webapp-anchor", secret: join(dir, "webapp.secret"), values: { name: "webapp" } }]),
);
const stop = runProvisioner(
const stop = runProviderLoop(
"analytics",
{
async create(p) {
@@ -101,7 +101,7 @@ test("provisioner hands the adapter what the mesh derived, and still has it at w
});
await writeFile(receives, doc([entry("mesh-anchor-ncloud")]));
const stop = runProvisioner(
const stop = runProviderLoop(
"s3-bucket",
{
async create(p) {
@@ -153,7 +153,7 @@ test("provisioner stops calling an unreadable secret a race, and says what to lo
console.error = (line: unknown) => { said.push(String(line)); };
let stop: (() => void) | undefined;
try {
stop = runProvisioner("db", { async create() {}, async remove() {} }, { receives, everyMs: 5 });
stop = runProviderLoop("db", { async create() {}, async remove() {} }, { receives, everyMs: 5 });
await waitFor(() => said.some((l) => l.includes("CANNOT READ")), 3000);
} finally {
stop?.();
@@ -185,7 +185,7 @@ test("provisioner removes nothing while its contributions file cannot be used (n
]);
await writeFile(receives, both);
const stop = runProvisioner(
const stop = runProviderLoop(
"postgres-database",
{
async create(p) {
@@ -239,7 +239,7 @@ test("provisioner applies again what the backend no longer holds, and trusts mem
const backend = new Set<string>();
let creates = 0;
let asked = 0;
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create(p) {
@@ -271,7 +271,7 @@ test("provisioner applies again what the backend no longer holds, and trusts mem
// An adapter without holds is trusted from memory, as before: a forgotten backend stays forgotten.
const forgetful = new Set<string>();
let plainCreates = 0;
const stopPlain = runProvisioner(
const stopPlain = runProviderLoop(
"cache",
{
async create(p) {
@@ -299,7 +299,7 @@ test("provisioner keeps a consumer applied when the backend cannot be asked", as
);
let creates = 0;
let asked = 0;
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create() {
@@ -330,7 +330,7 @@ test("provisioner backs off when create does not satisfy holds, and says so", as
const original = console.error;
console.error = (...a: unknown[]) => logged.push(a.join(" "));
try {
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create() {
@@ -371,7 +371,7 @@ test("provisioner treats a failed or hung holds as could-not-ask, and never logs
const original = console.error;
console.error = (...a: unknown[]) => logged.push(a.join(" "));
try {
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create() {
@@ -404,7 +404,7 @@ test("provisioner treats a failed or hung holds as could-not-ask, and never logs
const hungLogged: string[] = [];
console.error = (...a: unknown[]) => hungLogged.push(a.join(" "));
try {
const stopHung = runProvisioner(
const stopHung = runProviderLoop(
"cache",
{
async create() {
@@ -438,7 +438,7 @@ test("provisioner stops a verify pass at a timeout, and retries a failed re-appl
try {
// A timeout on "a" means "b" is not asked in that pass.
const askedFor: string[] = [];
const stopHung = runProvisioner(
const stopHung = runProviderLoop(
"cache",
{
async create() {},
@@ -460,7 +460,7 @@ test("provisioner stops a verify pass at a timeout, and retries a failed re-appl
let createCalls = 0;
const logged: string[] = [];
console.error = (...a: unknown[]) => logged.push(a.join(" "));
const stop = runProvisioner(
const stop = runProviderLoop(
"cache",
{
async create(p) {
@@ -649,3 +649,16 @@ test("each registration is given the environment asked for it, and sees no other
assert.deepEqual(seen.beta, { BETA_URL: "http://127.0.0.1:3000" });
resetTools();
});
test("the provider loop is named for what it is, and its old name stays an alias until its providers move (issue 678)", async () => {
const loop = await import("../dist/provider-loop/index.js");
const old = await import("../dist/index.js");
assert.equal(old.provisioner.runProvisioner, loop.runProviderLoop);
assert.equal(old.provisioner.provisionerTools, loop.providerLoopTools);
assert.equal(old.providerLoop.runProviderLoop, loop.runProviderLoop);
const pkg = JSON.parse(await (await import("node:fs/promises")).readFile(new URL("../package.json", import.meta.url), "utf8"));
assert.equal(pkg.exports["./provider-loop"], "./dist/provider-loop/index.js");
assert.equal(pkg.exports["./provisioner"], pkg.exports["./provider-loop"]);
// The events and tools are what the controller and the operator read on the bus: their names stay.
assert.equal(loop.FAILING_EVENT, "provisioner.failing");
});
+71
View File
@@ -0,0 +1,71 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { createPrivateKey } from "node:crypto";
import { seal, open, generateKeyPair, type SealedBox } from "../dist/seal/index.js";
import { sealWith } from "../dist/seal/box.js";
// The fixtures the Go package proves itself against: boxes the licence manager's own seal.go made, and
// the one the agent module's TypeScript made. One format, three implementations.
const fixtures = new URL("../go/seal/testdata/", import.meta.url);
const lm = JSON.parse(readFileSync(new URL("sealed-by-licence-manager.json", fixtures), "utf8")) as {
recipient: { publicKey: string; privateKey: string };
sealed: { plaintext: string; box: SealedBox }[];
vectors: { plaintext: string; ephPrivate: string; iv: string; box: SealedBox }[];
};
// PKCS #8 for an X25519 private key is this prefix and the 32 raw bytes.
const pkcs8 = (raw: Buffer) =>
createPrivateKey({ key: Buffer.concat([Buffer.from("302e020100300506032b656e04220420", "hex"), raw]), format: "der", type: "pkcs8" });
test("seal: every box the licence manager sealed opens to its plaintext", () => {
assert.ok(lm.sealed.length > 0);
for (const s of lm.sealed) assert.equal(open(s.box, lm.recipient.privateKey), s.plaintext);
});
test("seal: given the same one-time key and iv, the box is byte for byte the licence manager's", () => {
assert.ok(lm.vectors.length > 0);
for (const v of lm.vectors) {
const box = sealWith(v.plaintext, lm.recipient.publicKey, pkcs8(Buffer.from(v.ephPrivate, "base64")), Buffer.from(v.iv, "base64"));
assert.deepEqual(box, v.box);
assert.equal(open(box, lm.recipient.privateKey), v.plaintext);
}
});
test("seal: the agent module's TypeScript box opens", () => {
const f = JSON.parse(readFileSync(new URL("sealed-by-typescript.json", fixtures), "utf8"));
assert.equal(open(f.box, f.privateKey), f.plaintext);
});
test("seal: a box opens for its recipient only, and two boxes share no one-time key or iv", () => {
const a = generateKeyPair();
const b = generateKeyPair();
const box = seal("a token", a.publicKey);
assert.equal(open(box, a.privateKey), "a token");
assert.throws(() => open(box, b.privateKey));
const again = seal("a token", a.publicKey);
assert.notEqual(again.eph, box.eph);
assert.notEqual(again.iv, box.iv);
});
test("seal: a box changed in any field, or of another version, does not open", () => {
const good = lm.sealed[1].box;
const flip = (s: string) => {
const b = Buffer.from(s, "base64");
b[b.length - 1] ^= 1;
return b.toString("base64");
};
for (const box of [
{ ...good, ct: flip(good.ct) },
{ ...good, tag: flip(good.tag) },
{ ...good, iv: flip(good.iv) },
{ ...good, eph: flip(good.eph) },
{ ...good, v: 2 },
]) assert.throws(() => open(box as SealedBox, lm.recipient.privateKey));
});
test("seal: a key that is not X25519 PEM is refused", () => {
assert.throws(() => seal("x", "not a key"));
assert.throws(() => open(lm.sealed[0].box, lm.recipient.publicKey));
});
+272
View File
@@ -0,0 +1,272 @@
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224, issue 272): not on
// the first failure, which may be a restart; after failingAfterMs of failures with no success between;
// again every sayAgainEveryMs while it lasts; and recovered on the first success, or when the consumer
// goes. The same behaviour as the Go copy of the loop's standing_test.go, with the clock shortened.
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, writeFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
runProviderLoop, classOf, FAILING_EVENT, RECOVERED_EVENT, RETIREMENT_EVENT,
CLASS_CREDENTIALS, CLASS_UNREACHABLE, CLASS_SECRET, CLASS_REFUSED, type Adapter,
} from "../dist/provider-loop/index.js";
import { useBroker } from "../dist/messaging/index.js";
const EVERY = 10;
const FAILING_AFTER = 150;
const SAY_AGAIN = 400;
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
async function until(cond: () => boolean, ms = 4000): Promise<void> {
const start = Date.now();
while (!cond()) {
if (Date.now() - start > ms) throw new Error("timed out waiting");
await sleep(5);
}
}
interface Said { event: string; body: Record<string, unknown> }
interface World {
dir: string;
ask(list: { as: string; node?: string; values?: Record<string, unknown> }[]): Promise<void>;
failing: Error | null;
holdsErr: Error | null;
created: string[];
/** Every event but the first success since start, which has its own test. */
said: Said[];
all: Said[];
stop(): void;
}
async function world(opts: { classify?: boolean; announce?: false; stableForMs?: number } = {}): Promise<World> {
const dir = await mkdtemp(join(tmpdir(), "standing-"));
const receives = join(dir, "postgres-database.json");
const w = { dir, failing: null, holdsErr: null, created: [], said: [], all: [] } as unknown as World;
w.ask = async (list) => {
const given = [];
for (const g of list) {
await writeFile(join(dir, `${g.as}.secret`), `pw-${g.as}\n`);
given.push({ from: g.as, node: g.node ?? "anchor", as: g.as, secret: join(dir, `${g.as}.secret`), values: g.values });
}
await writeFile(receives, JSON.stringify({ contributions: 1, requirement: "postgres-database", given }));
};
const adapter: Adapter = {
async create(p) {
if (w.failing) throw w.failing;
w.created.push(p.as);
},
async remove() {},
async retire() {},
async holds() {
if (w.holdsErr) throw w.holdsErr;
return true;
},
...(opts.classify ? { classify: () => "its-own" } : {}),
};
const was = console.error;
console.error = () => {};
const stop = runProviderLoop("postgres-database", adapter, {
receives, everyMs: EVERY, verifyEveryMs: 0, node: "home-server",
failingAfterMs: FAILING_AFTER, sayAgainEveryMs: SAY_AGAIN, stableForMs: opts.stableForMs ?? 0,
...(opts.announce === false ? {} : {
announce: (event: string, body: Record<string, unknown>) => {
w.all.push({ event, body });
if (body.why !== "first-success") w.said.push({ event, body });
},
}),
});
w.stop = () => {
stop();
console.error = was;
void rm(dir, { recursive: true, force: true });
};
return w;
}
const events = (said: Said[]) => said.filter((s) => s.event !== RETIREMENT_EVENT).map((s) => s.event).join(",");
test("a consumer failed for minutes is announced, naming it, its machine and the class, again while it lasts, and recovered", async () => {
const w = await world();
try {
w.failing = new Error(`token request failed: 401 {"error":"invalid_grant"} for pw-a`);
await w.ask([{ as: "a", node: "anchor" }]);
await sleep(FAILING_AFTER / 2);
assert.equal(events(w.said), "", "announced before failingAfterMs");
await until(() => w.said.length === 1);
const b = w.said[0].body;
assert.equal(w.said[0].event, FAILING_EVENT);
assert.equal(b.consumer, "a");
assert.equal(b.node, "anchor");
assert.equal(b.class, CLASS_CREDENTIALS);
assert.equal(b.provider, "postgres-database");
assert.equal(b["provider-node"], "home-server");
assert.ok((b.attempts as number) > 3, `attempts ${b.attempts}`);
assert.match(String(b.since), /^\d{4}-\d{2}-\d{2}T/);
assert.doesNotMatch(String(b.error), /pw-a/, "the consumer's password travelled");
// Said again while it lasts, not every pass.
await sleep(SAY_AGAIN / 2);
assert.equal(events(w.said), FAILING_EVENT, "repeated too soon");
await until(() => w.said.length === 2);
assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`);
// The first success takes it back.
w.failing = null;
await until(() => w.said.length === 3);
assert.equal(w.said[2].event, RECOVERED_EVENT);
assert.equal(w.said[2].body.consumer, "a");
assert.equal(w.said[2].body.node, "anchor");
} finally {
w.stop();
}
});
test("one success between failures starts the run again", async () => {
const w = await world();
try {
w.failing = new Error("connection refused");
await w.ask([{ as: "a" }]);
await sleep(FAILING_AFTER * 0.6);
w.failing = null;
await until(() => w.created.length === 1);
w.failing = new Error("connection refused");
await w.ask([{ as: "a", values: { name: "changed" } }]);
await sleep(FAILING_AFTER * 0.6);
assert.equal(events(w.said), "", "two short runs are not one long one");
} finally {
w.stop();
}
});
// The check that failed for a day on 2026-10-05 (issue 179): consumers already made, every check
// refused. A check that cannot be asked is a failure too.
test("a check that keeps failing is a failure too", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }]);
await until(() => w.created.length === 1);
w.holdsErr = new Error("401 invalid_grant");
await until(() => w.said.length === 1);
assert.equal(w.said[0].event, FAILING_EVENT);
assert.equal(w.said[0].body.class, CLASS_CREDENTIALS);
w.holdsErr = null;
await until(() => w.said.length === 2);
assert.equal(events(w.said), `${FAILING_EVENT},${RECOVERED_EVENT}`);
} finally {
w.stop();
}
});
test("an unreadable secret is announced as such", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }]);
await rm(join(w.dir, "a.secret"));
await until(() => w.said.length === 1);
assert.equal(w.said[0].event, FAILING_EVENT);
assert.equal(w.said[0].body.class, CLASS_SECRET);
} finally {
w.stop();
}
});
test("a consumer no longer asked for is no longer failing", async () => {
const w = await world();
try {
w.failing = new Error("boom");
await w.ask([{ as: "a" }, { as: "b" }]);
await until(() => w.said.length === 2);
assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`);
await w.ask([{ as: "a" }]);
await until(() => w.said.some((s) => s.event === RECOVERED_EVENT));
const last = w.said.find((s) => s.event === RECOVERED_EVENT)!;
assert.equal(last.body.consumer, "b");
assert.equal(last.body.why, "no longer asked for");
} finally {
w.stop();
}
});
test("a failing consumer the mesh stopped asking for is said recovered when it is retired", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }, { as: "b" }, { as: "c" }]);
await until(() => w.created.length === 3);
w.holdsErr = new Error("boom");
await until(() => w.said.filter((s) => s.event === FAILING_EVENT).length >= 3);
// c is no longer asked for while it is still failing; the others then recover.
await w.ask([{ as: "a" }, { as: "b" }]);
w.holdsErr = null;
await until(() => w.said.some((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c"));
const c = w.said.find((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c")!;
assert.equal(c.body.why, "retired");
} finally {
w.stop();
}
});
test("an error is classed by its words", () => {
const table: Record<string, string> = {
'Keycloak token request failed: 401 {"error":"invalid_grant"}': CLASS_CREDENTIALS,
'FATAL: password authentication failed for user "postgres"': CLASS_CREDENTIALS,
"dial tcp 127.0.0.1:5432: connect: connection refused": CLASS_UNREACHABLE,
"context deadline exceeded": CLASS_UNREACHABLE,
"Error: connect ECONNREFUSED 127.0.0.1:9000": CLASS_UNREACHABLE,
"getaddrinfo ENOTFOUND minio": CLASS_UNREACHABLE,
'extension "nope" is not available': CLASS_REFUSED,
};
for (const [text, want] of Object.entries(table)) assert.equal(classOf(text), want, text);
});
test("an adapter that classes its own errors is believed", async () => {
const w = await world({ classify: true });
try {
w.failing = new Error("anything");
await w.ask([{ as: "a" }]);
await until(() => w.said.length === 1);
assert.equal(w.said[0].body.class, "its-own");
} finally {
w.stop();
}
});
// A provider restarted after announcing a failure has forgotten it; its first success for each
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
test("the first success since start is announced once", async () => {
const w = await world();
try {
await w.ask([{ as: "a" }]);
await until(() => w.all.length >= 1);
await sleep(EVERY * 10);
assert.equal(w.all.length, 1, JSON.stringify(w.all));
assert.equal(w.all[0].event, RECOVERED_EVENT);
assert.equal(w.all[0].body.why, "first-success");
assert.equal(w.all[0].body.consumer, "a");
} finally {
w.stop();
}
});
// No provider on main passed `announce`, so none said anything (issue 272): the loop says it through
// the SDK's own emit unless it is given another way.
test("without announce the standing is said through the SDK's emit", async () => {
const published: { key: string; body: Record<string, unknown> }[] = [];
useBroker(() => ({
async publish(env: { key: string; body: Record<string, unknown> }) { published.push({ key: env.key, body: env.body }); },
async subscribe() { return () => {}; },
}) as never);
const w = await world({ announce: false });
try {
w.failing = new Error("connection refused");
await w.ask([{ as: "a" }]);
await until(() => published.some((p) => p.key === FAILING_EVENT));
const p = published.find((x) => x.key === FAILING_EVENT)!;
assert.equal(p.body.consumer, "a");
assert.equal(p.body.class, CLASS_UNREACHABLE);
} finally {
w.stop();
}
});