Compare commits
31
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f185aedbf5 | ||
|
|
f70a99dbf4 | ||
|
|
a4e7f480b3 | ||
|
|
d7cac79585 | ||
|
|
377fd6d67d | ||
|
|
bb7c82cac6 | ||
|
|
0db956dd83 | ||
|
|
40f3658557 | ||
|
|
219bedbdd2 | ||
|
|
1bc5719b8b | ||
|
|
a2d2af00dc | ||
|
|
5933eb757d | ||
|
|
f4745447ba | ||
|
|
9588871d66 | ||
|
|
a0fc82fdcd | ||
|
|
d49d3a14d8 | ||
|
|
c1ae30b0a2 | ||
|
|
67bfa22132 | ||
|
|
4dc5f29d3e | ||
|
|
9747f3e1b1 | ||
|
|
d41d255f28 | ||
|
|
5794ccd569 | ||
|
|
eaed934629 | ||
|
|
4cf45b5ca0 | ||
|
|
d9e24e95a2 | ||
|
|
ee1e1a8303 | ||
|
|
24f5ee81bd | ||
|
|
b4900f52b8 | ||
|
|
19cfe0ef03 | ||
|
|
33917f91ac | ||
|
|
622b6b4f55 |
@@ -20,7 +20,13 @@ Not the manifest schema (the control plane owns and parses that, in Go). The sha
|
||||
contains — the contract both a provider and a consumer conform to), and the tool and event types.
|
||||
These change rarely and deliberately; when one does, a rebuild of everything is *correct*.
|
||||
|
||||
### `provisioner` — the reconcile harness every provider shares
|
||||
### `provider-loop` — the reconcile harness every provider on the grant files shares
|
||||
Called `provisioner` until design 57 (novox/hq issue 678); that word is mesh-provisioner's now. The old path and
|
||||
names (`/provisioner`, `runProvisioner`, `provisionerTools`) stay aliases until the providers importing them move to
|
||||
the Go helper that answers consumers over the bus (`go/provider`, design 57 B2 and B3) and this loop retires with the
|
||||
grant files. Its events and tools keep their names (`provisioner.failing`, `provisioner_retirement`, …): the
|
||||
controller and the operator read them.
|
||||
|
||||
The loop is identical for postgres, redis, minio and umami: watch the grants directory, for each
|
||||
requested grant call the provider's adapter, write the sealed credential, handle withdrawal, and
|
||||
keep converging (`watch`). That loop lives here. A **module provides only the adapter** — "create
|
||||
@@ -39,11 +45,21 @@ never called to retire**; it is the delete of an adapter without `delete`, reach
|
||||
person's `provisioner_delete`. A stable set of more than 3, or more than half of those held where more
|
||||
than one is held, is not retired without a person: the provider waits for
|
||||
`provisioner_retire_approve` or `provisioner_retire_reject`, and says `provisioner.retirement`
|
||||
(`waiting`) through the `announce` option. Asked for again, a retired consumer is created — re-enabled
|
||||
(`waiting`). Asked for again, a retired consumer is created — re-enabled
|
||||
— at once. Given `module`, the four tools (`provisioner_retirement`, `provisioner_retire_approve`,
|
||||
`provisioner_retire_reject`, `provisioner_delete`) are registered for it; `provisionerTools(handle)`
|
||||
builds them otherwise. Additions and changes are never delayed.
|
||||
|
||||
**A provider that keeps failing a consumer says so on the bus** (novox/hq ADR 0224). A consumer whose
|
||||
create, check (`holds`) or secret has failed for five minutes without one success between
|
||||
(`failingAfterMs`) is announced `provisioner.failing` — the consumer, its machine, the class of error
|
||||
(`credentials-rejected`, `unreachable`, `secret-unreadable`, `refused`; an adapter's `classify` is
|
||||
believed over the words), the error with the password removed, since when and how many attempts — and
|
||||
again every fifteen minutes while it lasts. The first success after it is `provisioner.recovered`, as is
|
||||
a consumer no longer asked for or retired, and the first success for each consumer since the process
|
||||
started. Every event goes through the `announce` option, which defaults to the SDK's `emit`: a provider
|
||||
says them without asking, and a failed publish is said in its journal, never failing the act.
|
||||
|
||||
### `tools` — the tool-serving harness
|
||||
`registerModuleTools`, the `ToolDefinition` type, and the registry/worker that loads a module's
|
||||
tools and exposes them through the mesh's command surface. *How* a tool is declared and served is
|
||||
@@ -136,12 +152,12 @@ import { UmamiClient } from "../client.js"; // the client lives in the module, n
|
||||
registerModuleTools("umami", (env) => getUmamiTools(new UmamiClient(env.UMAMI_URL)));
|
||||
```
|
||||
|
||||
A provider's provisioner:
|
||||
A provider's loop:
|
||||
|
||||
```ts
|
||||
import { runProvisioner, Grant, Credential } from "@novox/mesh-sdk/provisioner";
|
||||
import { runProviderLoop, Grant, Credential } from "@novox/mesh-sdk/provider-loop";
|
||||
|
||||
runProvisioner("analytics", {
|
||||
runProviderLoop("analytics", {
|
||||
async create(grant: Grant): Promise<Credential> { /* create a umami site, return the grant */ },
|
||||
async remove(grant: Grant): Promise<void> { /* delete the site */ },
|
||||
});
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Release 57: config and provisioning
|
||||
|
||||
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
|
||||
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
|
||||
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
|
||||
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
|
||||
+3
-3
@@ -2,8 +2,8 @@
|
||||
|
||||
`stdio.Serve("", tools)` speaks MCP over stdio to the node's tool runtime, as the module the runtime
|
||||
names (novox/hq ADR 0188, ADR 0193). A `Tool` is a name, a description, an argument schema and a
|
||||
function; calls run concurrently. A tool named `<seat>.<verb>` is the seat's implementation.
|
||||
`stdio.Emit(type, body)` publishes an event as the module, `stdio.Subscribe(pattern, handler)` receives the module's events (taken when the handler returns nil, offered again when it returns an error — the one rule is in the repository README, "Taking an event") and `stdio.Ask(key, body)` calls a tool, and `stdio.State(name)` gets, puts, deletes, lists and watches the module's declared state — all through the runtime (novox/hq ADR 0193, 0198, 0201). Standard library only.
|
||||
function; calls run concurrently. `stdio.CallerOf(args)` inside a tool is who called it: the bus user the runtime took from the subject the call arrived on, named in the call's `_meta`, never read from the arguments — or `stdio.Unproven` when the runtime named none (novox/hq issue 365). A tool named `<seat>.<verb>` is the seat's implementation.
|
||||
`stdio.Emit(type, body)` publishes an event as the module, `stdio.Subscribe(pattern, handler)` receives the module's events (taken when the handler returns nil, offered again when it returns an error — the one rule is in the repository README, "Taking an event") and `stdio.Ask(key, body)` calls a tool, and `stdio.State(name)` gets, puts, deletes, lists, watches and reads the kept history of the module's declared state, and `stdio.Log(name)` appends to, reads, and measures a log of operations the module declared — all through the runtime (novox/hq ADR 0193, 0198, 0201, 0297, 0307). `stdio.Config(key)` and `stdio.Bound(provision, key)` read, and `stdio.WatchConfig` and `stdio.WatchBound` watch, the module's own config values and the non-secret values of what it is bound to, for what its manifest declares: the runtime asks its machine's provisioner client as the module it launched, never a module the code names. `stdio.Secret(name)` is refused while the module's code runs in the tool runner; its secrets reach it through its files (novox/hq ADR 0349 §14, ADR 0370). `provider` is a providing module's half of a binding (novox/hq ADR 0349 §12, issue 678): it sees each consumer that binds the provider in the plans of the assignments mirror, writes one entry per consumer to `<module>_bound` (`<node>.<module>.<provision>[.<local>]`: served values composed with the consumer's identity, the login, and the name of the vault's credential, never a secret), hands the provider's adapter the vault's value (`provision/<provider>/<provision>` shared, `pair/<provider>/<provision>/<node>/<module>[/<local>]` per consumer), refuses a derived value to a consumer bound under a local name, naming both ends, and removes the entry of one that unbinds; the bus and the vault are interfaces the provider wires, the vault as `provider.ConsumersFile`, the file its machine's provisioner client fills from `${consumers:<provision>}` with each bound consumer's credential (novox/hq ADR 0373), since a provider in the tool runner never asks the vault itself. Standard library only.
|
||||
|
||||
## Installing
|
||||
|
||||
@@ -16,4 +16,4 @@ at the same version as the TypeScript package.
|
||||
|
||||
What it holds, under ADR 0039's test: the stdio loop and the tool type, and `cli`, the local protocol
|
||||
`mesh-cli` speaks to the node-engine on its machine (novox/hq ADR 0272): the socket's path and one request and one
|
||||
answer. Nothing else — no bus, no client, nothing of any module's. `go build ./example` and pipe a `tools/list` request into it.
|
||||
answer; and `seal`, the mesh's sealed box (novox/hq ADR 0206: X25519, HKDF-SHA256, AES-256-GCM, `{v, eph, iv, tag, ct}`), byte for byte the box the licence manager made before it lived here and the TypeScript `@novox/mesh-sdk/seal` makes. Nothing else — no bus, no client, nothing of any module's. `go build ./example` and pipe a `tools/list` request into it.
|
||||
|
||||
+69
-5
@@ -170,10 +170,18 @@ func (a Ask) Digest() string {
|
||||
canonical(&b, v)
|
||||
}
|
||||
}
|
||||
if a.Whole != "" || a.Details != "" {
|
||||
// Whole and Details are written whenever either or WholeDetails is given, and WholeDetails after them whenever
|
||||
// it is given (novox/hq ADR 0347): the fields after the options are then always one of three fixed shapes —
|
||||
// none, the two, or the three — and the count of values the sequence holds tells them apart, so no two asks
|
||||
// encode the same (the review of mesh-sdk #25). An ask with none of them digests as before they existed, and
|
||||
// one without WholeDetails as before it existed.
|
||||
if a.Whole != "" || a.Details != "" || a.WholeDetails != "" {
|
||||
canonical(&b, a.Whole)
|
||||
canonical(&b, a.Details)
|
||||
}
|
||||
if a.WholeDetails != "" {
|
||||
canonical(&b, a.WholeDetails)
|
||||
}
|
||||
sum := sha256.Sum256([]byte(b.String()))
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -202,9 +210,59 @@ type Ask struct {
|
||||
// The asker withholds a value shaped like a secret in it, and the router refuses the ask when one is left;
|
||||
// Explanation stays under the whole content rule everywhere else. Empty, Explanation is shown everywhere.
|
||||
Whole string `json:"whole,omitempty"`
|
||||
// Details is what the Details answer on the ask's message shows, line by line under the content rule: a
|
||||
// fingerprint, how to read the proposal whole at the terminal. Empty, an ask's own message offers no Details.
|
||||
// Details are the ask's secondary words, line by line under the content rule: who proposed it, a
|
||||
// fingerprint, how to read it whole. The router shows them in the note of the ask's own message
|
||||
// (Message.Note), after what the ask's options do and when it expires — never in its body (novox/hq
|
||||
// issue 639).
|
||||
Details string `json:"details,omitempty"`
|
||||
// WholeDetails are the Details with their exact values whole, shown in place of Details where Whole is shown
|
||||
// in place of the explanation (a kind that proves who answers, given the ask's answers): every value an
|
||||
// approval sets, whole, so the person reads exactly what they approve (novox/hq issue 383). Held, like
|
||||
// Whole, to the secret rule alone; the router refuses the ask when a secret's shape is left in it. Empty,
|
||||
// Details are shown everywhere.
|
||||
WholeDetails string `json:"whole-details,omitempty"`
|
||||
}
|
||||
|
||||
// What an ask's explanation is held to, so it reads at a glance on a phone (novox/hq issue 639): a summary
|
||||
// of two or three short sentences on its first line — what changes, why, what it costs or risks — then at
|
||||
// most three points, a line each, only where they add something. Whole is held to the same. Who asked, when,
|
||||
// the fingerprint and how to read it whole are Details, never the explanation.
|
||||
const (
|
||||
SummaryLength = 300
|
||||
MostPoints = 3
|
||||
PointLength = 160
|
||||
)
|
||||
|
||||
// Glance says why an explanation does not read at a glance, or nil: its first line, the summary, longer than
|
||||
// SummaryLength; more than MostPoints lines after it; a point longer than PointLength. Blank lines are not
|
||||
// counted. An asker holds its asks to it in its own tests; the router says, without refusing, an ask that
|
||||
// does not hold to it.
|
||||
func Glance(explanation string) error {
|
||||
var lines []string
|
||||
for _, l := range strings.Split(explanation, "\n") {
|
||||
if strings.TrimSpace(l) != "" {
|
||||
lines = append(lines, strings.TrimSpace(l))
|
||||
}
|
||||
}
|
||||
var problems []string
|
||||
switch {
|
||||
case len(lines) == 0:
|
||||
problems = append(problems, "it has no summary")
|
||||
case len([]rune(lines[0])) > SummaryLength:
|
||||
problems = append(problems, fmt.Sprintf("its summary is %d characters, more than %d", len([]rune(lines[0])), SummaryLength))
|
||||
}
|
||||
if len(lines) > 1+MostPoints {
|
||||
problems = append(problems, fmt.Sprintf("it has %d points after its summary, more than %d", len(lines)-1, MostPoints))
|
||||
}
|
||||
for i, l := range lines {
|
||||
if i > 0 && len([]rune(l)) > PointLength {
|
||||
problems = append(problems, fmt.Sprintf("its point %d is %d characters, more than %d", i, len([]rune(l)), PointLength))
|
||||
}
|
||||
}
|
||||
if len(problems) > 0 {
|
||||
return errors.New("the explanation does not read at a glance: " + strings.Join(problems, "; "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// The bounds of an ask (novox/hq ADR 0234 §8, ADR 0259 §4).
|
||||
@@ -435,8 +493,14 @@ type Message struct {
|
||||
Title string `json:"title"`
|
||||
Body string `json:"body"`
|
||||
Buttons []Button `json:"buttons,omitempty"`
|
||||
Urgent bool `json:"urgent,omitempty"`
|
||||
Silent bool `json:"silent,omitempty"`
|
||||
// Note is the message's secondary words, after its body and its buttons: who asked and when, what each
|
||||
// answer does, when it expires, and the ask's Details. A channel shows it smaller, collapsed or set apart
|
||||
// where it can, and after the body where it cannot (novox/hq issue 639). A channel that bounds a message's
|
||||
// length cuts the body first and the note only when the title and the note alone do not fit, saying each
|
||||
// cut (novox/hq ADR 0347 §5).
|
||||
Note string `json:"note,omitempty"`
|
||||
Urgent bool `json:"urgent,omitempty"`
|
||||
Silent bool `json:"silent,omitempty"`
|
||||
// Reply is the Choice or LinkAsked this answers, by its ID: the channel shows it where that was made.
|
||||
Reply string `json:"reply,omitempty"`
|
||||
// To is the account linked as the operator on this kind, for a channel that verifies its sender: where
|
||||
|
||||
+88
-2
@@ -253,8 +253,8 @@ func TestDigestsAreCanonicalOverNamedFields(t *testing.T) {
|
||||
t.Error("changing Whole beside Details left the digest the same")
|
||||
}
|
||||
// A field added to Ask or Option without being added to Digest fails here.
|
||||
if n := reflect.TypeOf(Ask{}).NumField(); n != 11 {
|
||||
t.Errorf("Ask has %d fields, and Digest covers 11: add the new one to Digest and to this test", n)
|
||||
if n := reflect.TypeOf(Ask{}).NumField(); n != 12 {
|
||||
t.Errorf("Ask has %d fields, and Digest covers 12: add the new one to Digest and to this test", n)
|
||||
}
|
||||
if n := reflect.TypeOf(Option{}).NumField(); n != 5 {
|
||||
t.Errorf("Option has %d fields, and Digest covers 5: add the new one to Digest and to this test", n)
|
||||
@@ -279,3 +279,89 @@ func TestAWarrantForAnAskWithNoExpiryIsRefused(t *testing.T) {
|
||||
t.Error("a warrant for an ask with no expiry was taken")
|
||||
}
|
||||
}
|
||||
|
||||
// An explanation that reads at a glance passes; a wall of text says each way it is one (novox/hq issue 639).
|
||||
func TestAnExplanationReadsAtAGlance(t *testing.T) {
|
||||
if err := Glance("Your agents' memory follows you to every machine. A file changed on two keeps both.\n\n• Only the memory folders are carried\n• A conflict reaches you as a warning"); err != nil {
|
||||
t.Fatalf("a summary and two points: %v", err)
|
||||
}
|
||||
var wall strings.Builder
|
||||
wall.WriteString("ADR 0343 was proposed by fix-367 and approved by the knowledge base's own review.")
|
||||
for i := 1; i <= 13; i++ {
|
||||
wall.WriteString("\n" + strings.Repeat("A clause lead. ", 3))
|
||||
}
|
||||
err := Glance(wall.String())
|
||||
if err == nil || !strings.Contains(err.Error(), "13 points") {
|
||||
t.Fatalf("thirteen clause lines pass as a glance: %v", err)
|
||||
}
|
||||
if err := Glance(strings.Repeat("word ", 70)); err == nil || !strings.Contains(err.Error(), "summary is 349") {
|
||||
t.Fatalf("a long summary passes: %v", err)
|
||||
}
|
||||
if err := Glance("short.\n" + strings.Repeat("p", PointLength+1)); err == nil || !strings.Contains(err.Error(), "point 1") {
|
||||
t.Fatalf("a long point passes: %v", err)
|
||||
}
|
||||
if err := Glance(" \n"); err == nil {
|
||||
t.Fatal("an empty explanation passes")
|
||||
}
|
||||
}
|
||||
|
||||
// A message's note travels beside its body, and is left out of the encoding when empty, so a channel older
|
||||
// than it reads every message it read before.
|
||||
func TestAMessageCarriesItsNoteApart(t *testing.T) {
|
||||
raw, _ := json.Marshal(Message{Handle: "h", Title: "t", Body: "b", Note: "Asked by x."})
|
||||
if !strings.Contains(string(raw), `"note":"Asked by x."`) {
|
||||
t.Fatalf("the note is not encoded: %s", raw)
|
||||
}
|
||||
raw, _ = json.Marshal(Message{Handle: "h", Title: "t"})
|
||||
if strings.Contains(string(raw), "note") {
|
||||
t.Fatalf("an empty note is encoded: %s", raw)
|
||||
}
|
||||
}
|
||||
|
||||
// WholeDetails is digested when given, and an ask without it digests as before it existed: a router and an asker
|
||||
// of different builds still agree on every such ask.
|
||||
func TestWholeDetailsAreDigestedOnlyWhenGiven(t *testing.T) {
|
||||
a := Ask{ID: "a", Headline: "h", Explanation: "e", Details: "d"}
|
||||
before := a.Digest()
|
||||
a.WholeDetails = "+ shares: media=/storage/media"
|
||||
if a.Digest() == before {
|
||||
t.Fatal("the whole details are not in the digest")
|
||||
}
|
||||
b := a
|
||||
b.WholeDetails = "+ shares: media=/storage/other"
|
||||
if a.Digest() == b.Digest() {
|
||||
t.Fatal("two whole details digest the same")
|
||||
}
|
||||
a.WholeDetails = ""
|
||||
if a.Digest() != before {
|
||||
t.Fatal("an ask without whole details digests otherwise than before")
|
||||
}
|
||||
}
|
||||
|
||||
// No two asks encode the same (the review of mesh-sdk #25): whole details given beside nothing else are not read as
|
||||
// a Whole and Details that happen to spell the same values, nor as any other split of the three fields.
|
||||
func TestWholeDetailsCollideWithNoOtherAsk(t *testing.T) {
|
||||
base := Ask{ID: "a", Headline: "h", Explanation: "e"}
|
||||
shapes := []Ask{}
|
||||
for _, f := range []struct{ whole, details, wholeDetails string }{
|
||||
{"", "", "+ x: 1"},
|
||||
{"whole-details", "+ x: 1", ""},
|
||||
{"", "+ x: 1", ""},
|
||||
{"+ x: 1", "", ""},
|
||||
{"", "", "whole-details"},
|
||||
{"", "+ x: 1", "+ x: 1"},
|
||||
{"+ x: 1", "+ x: 1", ""},
|
||||
{"", "", ""},
|
||||
} {
|
||||
a := base
|
||||
a.Whole, a.Details, a.WholeDetails = f.whole, f.details, f.wholeDetails
|
||||
shapes = append(shapes, a)
|
||||
}
|
||||
seen := map[string]int{}
|
||||
for i, a := range shapes {
|
||||
if j, ok := seen[a.Digest()]; ok {
|
||||
t.Errorf("asks %d and %d digest the same: %+v / %+v", j, i, shapes[j], a)
|
||||
}
|
||||
seen[a.Digest()] = i
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
package stdio
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"regexp"
|
||||
"sync"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
// Unproven is the caller of a call the runtime named no caller for: one that arrived on the subject that names
|
||||
// none, or from a runtime that names none (novox/hq issue 365). Said as a word, never left empty, so a record
|
||||
// shows it was not proven.
|
||||
const Unproven = "unproven"
|
||||
|
||||
// MetaCaller is where in a `tools/call`'s `_meta` the runtime names the caller: the bus user that published the
|
||||
// call, which the runtime takes from the subject the call arrived on — that subject's last token, which the
|
||||
// bus lets each user publish with its own name alone (novox/hq issue 365, by ADR 0259 §3's precedent). The
|
||||
// runtime writes `_meta`; a caller writes only `arguments`, so no caller can name another.
|
||||
const MetaCaller = "mesh/caller"
|
||||
|
||||
// busUser is the shape of a bus user's name: `<node>.<module>`, `person.<name>`, `controller` — names whose
|
||||
// parts are safe in a subject. Anything else the runtime named is not a caller and reads as Unproven.
|
||||
var busUser = regexp.MustCompile(`^[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*$`)
|
||||
|
||||
// calls holds the caller of every call while its tool runs, by the arguments map that call was handed.
|
||||
var calls sync.Map // unsafe.Pointer → string
|
||||
|
||||
func argsKey(args map[string]any) unsafe.Pointer {
|
||||
return reflect.ValueOf(args).UnsafePointer()
|
||||
}
|
||||
|
||||
// CallerOf is who called the tool that was handed args, as the runtime proved it: the bus user that
|
||||
// published the call, or Unproven. Read from the runtime, never from the arguments' contents — a `caller`
|
||||
// argument names nobody. Valid while the tool runs; arguments no call was handed have no caller.
|
||||
func CallerOf(args map[string]any) string {
|
||||
if args == nil {
|
||||
return Unproven
|
||||
}
|
||||
if c, ok := calls.Load(argsKey(args)); ok {
|
||||
return c.(string)
|
||||
}
|
||||
return Unproven
|
||||
}
|
||||
|
||||
// callerIn is the caller a `tools/call`'s `_meta` names, or Unproven.
|
||||
func callerIn(meta map[string]any) string {
|
||||
if s, ok := meta[MetaCaller].(string); ok && busUser.MatchString(s) {
|
||||
return s
|
||||
}
|
||||
return Unproven
|
||||
}
|
||||
|
||||
// runAs runs a tool with the caller held for its arguments for as long as it runs.
|
||||
func runAs(t Tool, caller string, args map[string]any) (any, error) {
|
||||
key := argsKey(args)
|
||||
calls.Store(key, caller)
|
||||
defer calls.Delete(key)
|
||||
return t.Run(args)
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package stdio
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// callWith serves one tool that answers the caller CallerOf reads from its arguments, sends it one tools/call
|
||||
// with the params given, and returns the caller it answered.
|
||||
func callWith(t *testing.T, params map[string]any) string {
|
||||
t.Helper()
|
||||
inR, inW := io.Pipe()
|
||||
outR, outW := io.Pipe()
|
||||
go func() {
|
||||
_ = ServeOn("tracker", []Tool{{Name: "who", Run: func(args map[string]any) (any, error) {
|
||||
return map[string]any{"caller": CallerOf(args)}, nil
|
||||
}}}, inR, outW)
|
||||
}()
|
||||
defer inW.Close()
|
||||
b, _ := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": params})
|
||||
_, _ = inW.Write(append(b, '\n'))
|
||||
lines := bufio.NewScanner(outR)
|
||||
done := make(chan string, 1)
|
||||
go func() {
|
||||
if lines.Scan() {
|
||||
var m struct {
|
||||
Result struct {
|
||||
Content []struct {
|
||||
Text string `json:"text"`
|
||||
} `json:"content"`
|
||||
} `json:"result"`
|
||||
}
|
||||
_ = json.Unmarshal(lines.Bytes(), &m)
|
||||
var answer struct {
|
||||
Caller string `json:"caller"`
|
||||
}
|
||||
if len(m.Result.Content) > 0 {
|
||||
_ = json.Unmarshal([]byte(m.Result.Content[0].Text), &answer)
|
||||
}
|
||||
done <- answer.Caller
|
||||
}
|
||||
}()
|
||||
select {
|
||||
case got := <-done:
|
||||
return got
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("the bundle did not answer the call")
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// **A tool reads its caller from the runtime, never from the call's arguments** (novox/hq issue 365): the
|
||||
// runtime names the caller it took from the subject the call arrived on in the call's `_meta`, and CallerOf
|
||||
// hands it to the tool by the arguments the tool was given.
|
||||
func TestAToolReadsTheCallerTheRuntimeNamed(t *testing.T) {
|
||||
got := callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"title": "x"},
|
||||
"_meta": map[string]any{MetaCaller: "shanks.node-tools"}})
|
||||
if got != "shanks.node-tools" {
|
||||
t.Fatalf("the tool read caller %q, want the one the runtime named", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A caller named in the arguments is not the caller: an argument is the caller's to write.
|
||||
func TestACallerInTheArgumentsIsNeverTheCaller(t *testing.T) {
|
||||
got := callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"caller": "novox.node-tools",
|
||||
"_meta": map[string]any{MetaCaller: "novox.node-tools"}, MetaCaller: "novox.node-tools"}})
|
||||
if got != Unproven {
|
||||
t.Fatalf("a caller the arguments claimed was read as %q, want %q", got, Unproven)
|
||||
}
|
||||
got = callWith(t, map[string]any{"name": "who", "arguments": map[string]any{"caller": "novox.node-tools"},
|
||||
"_meta": map[string]any{MetaCaller: "shanks.node-tools"}})
|
||||
if got != "shanks.node-tools" {
|
||||
t.Fatalf("with a caller in the arguments the tool read %q, want the runtime's", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A runtime that names no caller — one from before issue 365, or a call on the subject that names none — is
|
||||
// said as Unproven, explicitly; a caller that is not a bus user's name is Unproven too.
|
||||
func TestACallTheRuntimeNamedNoCallerForIsUnproven(t *testing.T) {
|
||||
for _, params := range []map[string]any{
|
||||
{"name": "who", "arguments": map[string]any{}},
|
||||
{"name": "who"},
|
||||
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{}},
|
||||
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: ""}},
|
||||
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: "a b"}},
|
||||
{"name": "who", "arguments": map[string]any{}, "_meta": map[string]any{MetaCaller: 7}},
|
||||
} {
|
||||
if got := callWith(t, params); got != Unproven {
|
||||
t.Errorf("%v: the tool read caller %q, want %q", params, got, Unproven)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Arguments the runtime never handed a tool have no caller.
|
||||
func TestArgumentsOfNoCallHaveNoCaller(t *testing.T) {
|
||||
if got := CallerOf(map[string]any{"x": 1}); got != Unproven {
|
||||
t.Fatalf("CallerOf of a map no call was given answered %q", got)
|
||||
}
|
||||
if got := CallerOf(nil); got != Unproven {
|
||||
t.Fatalf("CallerOf(nil) answered %q", got)
|
||||
}
|
||||
}
|
||||
+15
-1
@@ -140,7 +140,21 @@ func AskWith(ctx context.Context, socket string, req Request) (Answer, error) {
|
||||
_ = conn.SetDeadline(deadline)
|
||||
}
|
||||
if err := WriteRequest(conn, req); err != nil {
|
||||
return Answer{}, fmt.Errorf("asking the node-engine: %w", err)
|
||||
// The engine may have answered before it read the request — a refusal (busy, an unknown account) written
|
||||
// and the connection closed — and the write then fails on the close. That answer is already waiting, and
|
||||
// it says why where the write error says only that the pipe broke (novox/hq issue 444, found under issue
|
||||
// 434). So read it, within the deadline the connection already has, and only when none came say the write.
|
||||
// Only a refusal is taken: the request was never wholly written, so nothing can have run, and any other
|
||||
// answer here is not one a correct engine gives — passing it on as success would hide the failure.
|
||||
a, rerr := ReadAnswer(conn)
|
||||
if rerr != nil {
|
||||
return Answer{}, fmt.Errorf("asking the node-engine: %w; and no answer came (%v)", err, rerr)
|
||||
}
|
||||
if a.Refused == "" {
|
||||
return Answer{}, fmt.Errorf("asking the node-engine: %w; and the answer that came is not a refusal, "+
|
||||
"though the request was never wholly written", err)
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
a, err := ReadAnswer(conn)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// anEngineThatClosesFirst listens like the node-engine before mesh-host#80: it writes its answer, if any, and
|
||||
// closes without reading the request (novox/hq issue 434). It answers one connection.
|
||||
func anEngineThatClosesFirst(t *testing.T, answer *Answer) string {
|
||||
t.Helper()
|
||||
socket := filepath.Join(t.TempDir(), "cli.sock")
|
||||
l, err := net.Listen("unix", socket)
|
||||
if err != nil {
|
||||
t.Skipf("no unix socket here: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { l.Close() })
|
||||
go func() {
|
||||
conn, err := l.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if answer != nil {
|
||||
_ = WriteAnswer(conn, *answer)
|
||||
}
|
||||
conn.Close()
|
||||
}()
|
||||
return socket
|
||||
}
|
||||
|
||||
// aLineTheSocketCannotHold is a request larger than a unix socket buffers, so its write cannot finish before
|
||||
// the engine reads it — and an engine that closes first makes it fail every time, not only when it loses a race.
|
||||
func aLineTheSocketCannotHold() []string { return []string{"status", strings.Repeat("x", 4<<20)} }
|
||||
|
||||
// An engine that refuses and closes before reading the request is heard: its refusal reaches the caller, not
|
||||
// the broken pipe its close left the client's write (novox/hq issue 444).
|
||||
func TestARefusalWrittenBeforeTheRequestIsReadReachesTheCaller(t *testing.T) {
|
||||
busy := Refusal("the node-engine is busy: too many lines at once")
|
||||
socket := anEngineThatClosesFirst(t, &busy)
|
||||
a, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
|
||||
if err != nil {
|
||||
t.Fatalf("the engine said %q and the caller heard: %v", busy.Refused, err)
|
||||
}
|
||||
if a.Refused != busy.Refused || a.Exit != 1 {
|
||||
t.Fatalf("the engine said %q and the caller heard %+v", busy.Refused, a)
|
||||
}
|
||||
}
|
||||
|
||||
// When the write fails and no answer is waiting, the error says both: what failed, and that no answer came.
|
||||
func TestAFailedWriteWithNoAnswerSaysBoth(t *testing.T) {
|
||||
socket := anEngineThatClosesFirst(t, nil)
|
||||
start := time.Now()
|
||||
_, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
|
||||
if err == nil {
|
||||
t.Fatal("a request the engine never read, and never answered, came back without an error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "asking the node-engine") || !strings.Contains(err.Error(), "no answer came") {
|
||||
t.Fatalf("the error does not say what failed and that no answer came: %v", err)
|
||||
}
|
||||
if waited := time.Since(start); waited > 30*time.Second {
|
||||
t.Fatalf("waited %s on a connection the engine had closed", waited)
|
||||
}
|
||||
}
|
||||
|
||||
// After a failed write only a refusal is taken: nothing can have run, so an answer saying it ran and exited 0
|
||||
// is an error, never a success (novox/hq issue 444).
|
||||
func TestAnAnswerOtherThanARefusalAfterAFailedWriteIsAnError(t *testing.T) {
|
||||
ran := Answer{Exit: 0, Stdout: []byte("ok")}
|
||||
socket := anEngineThatClosesFirst(t, &ran)
|
||||
a, err := Ask(context.Background(), socket, aLineTheSocketCannotHold())
|
||||
if err == nil {
|
||||
t.Fatalf("a request never wholly written came back as success: %+v", a)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "not a refusal") {
|
||||
t.Fatalf("the error does not say the answer was not a refusal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Reading after a failed write never waits longer than the ask already allowed: a peer that never reads and
|
||||
// never answers is given up on at the context's deadline, not later (novox/hq issue 444).
|
||||
func TestAFailedWriteWaitsNoLongerThanTheDeadline(t *testing.T) {
|
||||
socket := filepath.Join(t.TempDir(), "cli.sock")
|
||||
l, err := net.Listen("unix", socket)
|
||||
if err != nil {
|
||||
t.Skipf("no unix socket here: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { l.Close() })
|
||||
held := make(chan net.Conn, 1)
|
||||
go func() {
|
||||
conn, err := l.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
held <- conn // neither read nor answered, nor closed, until the test ends
|
||||
}()
|
||||
t.Cleanup(func() {
|
||||
select {
|
||||
case c := <-held:
|
||||
c.Close()
|
||||
default:
|
||||
}
|
||||
})
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
_, err = Ask(ctx, socket, aLineTheSocketCannotHold())
|
||||
waited := time.Since(start)
|
||||
if err == nil {
|
||||
t.Fatal("a peer that never read nor answered gave no error")
|
||||
}
|
||||
if !errors.Is(err, os.ErrDeadlineExceeded) {
|
||||
t.Fatalf("the error is not the deadline: %v", err)
|
||||
}
|
||||
if waited > 3*time.Second {
|
||||
t.Fatalf("waited %s with a deadline of 1s", waited)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
// Package health is the answer a module's health tool gives, and how the node-engine reads it (novox/hq ADR 0240
|
||||
// rule 2, ADR 0283).
|
||||
//
|
||||
// A health tool answers one of three:
|
||||
//
|
||||
// - healthy: everything the module runs works;
|
||||
// - unhealthy: something is wrong, said in Why;
|
||||
// - waits: nothing is wrong but a part that cannot work until the operator gives something — one of the module's
|
||||
// own secrets, or one of its settings. Each Wait names exactly one of them, the part, and what the operator
|
||||
// gives, in words with no path, address or value.
|
||||
//
|
||||
// The answer keeps `healthy` beside `state`, so a node-engine older than ADR 0283 reads a wait as not healthy, as
|
||||
// it did before. An answer without `state` is read from `healthy`.
|
||||
//
|
||||
// The module writes its answer with Healthy, Unhealthy or Waiting, and the node-engine reads it with Read. Read
|
||||
// never trusts a shape it cannot check: a wait it cannot read makes the answer unhealthy, saying why.
|
||||
package health
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The three states an answer says.
|
||||
const (
|
||||
StateHealthy = "healthy"
|
||||
StateUnhealthy = "unhealthy"
|
||||
StateWaits = "waits"
|
||||
)
|
||||
|
||||
// Answer is a health tool's answer.
|
||||
type Answer struct {
|
||||
// State is healthy, unhealthy or waits.
|
||||
State string `json:"state"`
|
||||
// Healthy is State == healthy, kept for a reader older than State.
|
||||
Healthy bool `json:"healthy"`
|
||||
// Why is the answer in one or more plain sentences.
|
||||
Why string `json:"why"`
|
||||
// Waits is what the module waits for, when State is waits; and, beside a fault, what it also waits for.
|
||||
Waits []Wait `json:"waits,omitempty"`
|
||||
}
|
||||
|
||||
// Wait is one thing a module waits for the operator to give: exactly one of Secret and Setting.
|
||||
type Wait struct {
|
||||
// Part is what waits, in words: "the source games".
|
||||
Part string `json:"part"`
|
||||
// Secret is one of the module's own secrets, by name (a member of a secret family by its full name).
|
||||
Secret string `json:"secret,omitempty"`
|
||||
// Setting is one of the module's settings, by name.
|
||||
Setting string `json:"setting,omitempty"`
|
||||
// What is what the operator gives, in words: "the password of the source games".
|
||||
What string `json:"what"`
|
||||
}
|
||||
|
||||
// Healthy is a healthy answer.
|
||||
func Healthy(why string) Answer {
|
||||
return Answer{State: StateHealthy, Healthy: true, Why: why}
|
||||
}
|
||||
|
||||
// Unhealthy is an unhealthy answer. Waits may be given beside the fault: they are said, and excuse nothing.
|
||||
func Unhealthy(why string, waits ...Wait) Answer {
|
||||
return Answer{State: StateUnhealthy, Why: why, Waits: waits}
|
||||
}
|
||||
|
||||
// Waiting is an answer that says nothing is wrong but what the operator has yet to give. With no wait it is
|
||||
// unhealthy: a module that waits names what for.
|
||||
func Waiting(why string, waits ...Wait) Answer {
|
||||
if len(waits) == 0 {
|
||||
return Unhealthy(orSaid(why, "it says it waits") + " (and names nothing it waits for)")
|
||||
}
|
||||
for _, w := range waits {
|
||||
if err := w.Check(); err != nil {
|
||||
return Unhealthy(orSaid(why, "it says it waits") + " (" + err.Error() + ")")
|
||||
}
|
||||
}
|
||||
return Answer{State: StateWaits, Why: why, Waits: waits}
|
||||
}
|
||||
|
||||
// name is a secret's or a setting's name: what a manifest may declare.
|
||||
var name = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
|
||||
|
||||
// looksLikeAddress finds what the operator's words never carry: a URL, an IPv4 or IPv6 address, a host and port, a
|
||||
// path (absolute, from a home or relative, on its own or after a bracket or a quote), a UNC path or a drive path.
|
||||
// It cannot find a value typed in plain words ("the password hunter2"): a module never puts one there, and the
|
||||
// words are its own, not the operator's.
|
||||
var looksLikeAddress = regexp.MustCompile(strings.Join([]string{
|
||||
`://`,
|
||||
`\b[0-9]{1,3}(\.[0-9]{1,3}){3}\b`,
|
||||
`[0-9A-Fa-f]{0,4}::[0-9A-Fa-f]{0,4}`,
|
||||
`\b[0-9A-Fa-f]{1,4}(:[0-9A-Fa-f]{1,4}){3,}\b`,
|
||||
`\b[A-Za-z0-9.-]+:[0-9]{1,5}\b`,
|
||||
`(^|[\s(\[{"'\x60])[~.]{0,2}/`,
|
||||
`\\\\`,
|
||||
`\b[A-Za-z]:\\`,
|
||||
}, "|"))
|
||||
|
||||
// Check says why a wait cannot be read, or nil.
|
||||
func (w Wait) Check() error {
|
||||
switch {
|
||||
case w.Secret != "" && w.Setting != "":
|
||||
return fmt.Errorf("a wait names both the secret %q and the setting %q; it names one", w.Secret, w.Setting)
|
||||
case w.Secret == "" && w.Setting == "":
|
||||
return fmt.Errorf("a wait names neither a secret nor a setting")
|
||||
case w.Secret != "" && !name.MatchString(w.Secret):
|
||||
return fmt.Errorf("a wait names the secret %q, which is not a name", w.Secret)
|
||||
case w.Setting != "" && !name.MatchString(w.Setting):
|
||||
return fmt.Errorf("a wait names the setting %q, which is not a name", w.Setting)
|
||||
case strings.TrimSpace(w.What) == "":
|
||||
return fmt.Errorf("a wait for %s does not say what the operator gives", w.Names())
|
||||
case strings.TrimSpace(w.Part) == "":
|
||||
return fmt.Errorf("a wait for %s does not say which part waits", w.Names())
|
||||
}
|
||||
for _, words := range []string{w.What, w.Part} {
|
||||
if strings.ContainsAny(words, "\n\r") {
|
||||
return fmt.Errorf("a wait for %s says more than one line", w.Names())
|
||||
}
|
||||
if looksLikeAddress.MatchString(words) {
|
||||
return fmt.Errorf("a wait for %s says a path or an address, which the operator's words never carry", w.Names())
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Names is what the wait names, as "the secret x" or "the setting y".
|
||||
func (w Wait) Names() string {
|
||||
if w.Secret != "" {
|
||||
return "the secret " + w.Secret
|
||||
}
|
||||
if w.Setting != "" {
|
||||
return "the setting " + w.Setting
|
||||
}
|
||||
return "nothing"
|
||||
}
|
||||
|
||||
// Read reads a health tool's result. An answer it cannot read at all is an error; an answer whose wait it cannot
|
||||
// read is unhealthy, saying why, so a malformed wait never excuses anything.
|
||||
func Read(result []byte) (Answer, error) {
|
||||
var a Answer
|
||||
if err := json.Unmarshal(result, &a); err != nil {
|
||||
return Answer{}, fmt.Errorf("it answered something other than healthy or not: %w", err)
|
||||
}
|
||||
switch a.State {
|
||||
case "":
|
||||
if a.Healthy {
|
||||
a.State = StateHealthy
|
||||
} else {
|
||||
a.State = StateUnhealthy
|
||||
}
|
||||
case StateHealthy:
|
||||
if !a.Healthy {
|
||||
return Unhealthy(fmt.Sprintf("it answered the state healthy and healthy false, which contradict each other (%s)",
|
||||
orSaid(a.Why, "and said nothing more"))), nil
|
||||
}
|
||||
// A healthy module waits for nothing: a wait beside healthy is a contradiction, and is dropped.
|
||||
a.Waits = nil
|
||||
case StateUnhealthy:
|
||||
a.Healthy = false
|
||||
case StateWaits:
|
||||
a.Healthy = false
|
||||
if len(a.Waits) == 0 {
|
||||
return Unhealthy(orSaid(a.Why, "it says it waits") + " (and names nothing it waits for)"), nil
|
||||
}
|
||||
for _, w := range a.Waits {
|
||||
if err := w.Check(); err != nil {
|
||||
return Unhealthy(orSaid(a.Why, "it says it waits") + " (" + err.Error() + ")"), nil
|
||||
}
|
||||
}
|
||||
default:
|
||||
return Unhealthy(fmt.Sprintf("it answered the state %q, which is none of healthy, unhealthy or waits", a.State)), nil
|
||||
}
|
||||
if a.State != StateWaits {
|
||||
// Waits beside a fault are said and excuse nothing; one that cannot be read is dropped, not believed.
|
||||
var kept []Wait
|
||||
for _, w := range a.Waits {
|
||||
if w.Check() == nil {
|
||||
kept = append(kept, w)
|
||||
}
|
||||
}
|
||||
a.Waits = kept
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
func orSaid(s, otherwise string) string {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return otherwise
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package health
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func roundTrip(t *testing.T, a Answer) Answer {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(a)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := Read(b)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
var password = Wait{Part: "the source games", Secret: "smb-password-games", What: "the password of the source games"}
|
||||
|
||||
func TestEachConstructorSaysItsStateAndKeepsHealthyForAnOlderReader(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
a Answer
|
||||
state string
|
||||
healthy bool
|
||||
}{
|
||||
{Healthy("all armed"), StateHealthy, true},
|
||||
{Unhealthy("the share is unreachable"), StateUnhealthy, false},
|
||||
{Waiting("the source games waits for its password", password), StateWaits, false},
|
||||
} {
|
||||
b, _ := json.Marshal(c.a)
|
||||
var old struct {
|
||||
Healthy bool `json:"healthy"`
|
||||
Why string `json:"why"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &old); err != nil || old.Healthy != c.healthy || old.Why == "" {
|
||||
t.Fatalf("an older reader of %s reads %+v (%v)", b, old, err)
|
||||
}
|
||||
if got := roundTrip(t, c.a); got.State != c.state || got.Healthy != c.healthy {
|
||||
t.Fatalf("%s read as %+v", b, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWaitIsReadWithWhatItNames(t *testing.T) {
|
||||
got := roundTrip(t, Waiting("waits", password, Wait{Part: "the source games", Setting: "smb-users", What: "the username of the source games"}))
|
||||
if got.State != StateWaits || len(got.Waits) != 2 || got.Waits[0].Secret != "smb-password-games" || got.Waits[1].Setting != "smb-users" {
|
||||
t.Fatalf("read as %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAnswerWithoutStateIsReadFromHealthy(t *testing.T) {
|
||||
for raw, want := range map[string]string{`{"healthy":true,"why":"ok"}`: StateHealthy, `{"healthy":false,"why":"down"}`: StateUnhealthy} {
|
||||
got, err := Read([]byte(raw))
|
||||
if err != nil || got.State != want {
|
||||
t.Fatalf("%s read as %+v, %v", raw, got, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWaitThatCannotBeReadIsUnhealthyAndSaysWhy(t *testing.T) {
|
||||
for name, c := range map[string]struct {
|
||||
w []Wait
|
||||
says string
|
||||
}{
|
||||
"no wait": {nil, "names nothing"},
|
||||
"neither": {[]Wait{{Part: "p", What: "w"}}, "neither a secret nor a setting"},
|
||||
"both": {[]Wait{{Part: "p", Secret: "a", Setting: "b", What: "w"}}, "both"},
|
||||
"not a name": {[]Wait{{Part: "p", Secret: "Smb Password", What: "w"}}, "not a name"},
|
||||
"no what": {[]Wait{{Part: "p", Secret: "a"}}, "does not say what"},
|
||||
"no part": {[]Wait{{Secret: "a", What: "w"}}, "which part"},
|
||||
"two lines": {[]Wait{{Part: "p", Secret: "a", What: "the password\nhunter2"}}, "more than one line"},
|
||||
"an address": {[]Wait{{Part: "p", Secret: "a", What: "the password of 192.0.2.76"}}, "path or an address"},
|
||||
"a url": {[]Wait{{Part: "p", Secret: "a", What: "the password of smb://host/share"}}, "path or an address"},
|
||||
"a path": {[]Wait{{Part: "the source at /mnt/games", Secret: "a", What: "the password"}}, "path or an address"},
|
||||
} {
|
||||
b, _ := json.Marshal(Answer{State: StateWaits, Why: "it waits", Waits: c.w})
|
||||
got, err := Read(b)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if got.State != StateUnhealthy || got.Healthy || !strings.Contains(got.Why, c.says) {
|
||||
t.Fatalf("%s: read as %+v, want unhealthy saying %q", name, got, c.says)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnknownStateIsUnhealthy(t *testing.T) {
|
||||
got, err := Read([]byte(`{"state":"fine","healthy":true}`))
|
||||
if err != nil || got.State != StateUnhealthy || got.Healthy {
|
||||
t.Fatalf("read as %+v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWaitsBesideAFaultAreSaidAndExcuseNothing(t *testing.T) {
|
||||
got := roundTrip(t, Unhealthy("the share is unreachable", password, Wait{Part: "p", What: "w"}))
|
||||
if got.State != StateUnhealthy || len(got.Waits) != 1 {
|
||||
t.Fatalf("read as %+v: the readable wait is kept, the other dropped, the state stays unhealthy", got)
|
||||
}
|
||||
if h := roundTrip(t, Answer{State: StateHealthy, Healthy: true, Why: "ok", Waits: []Wait{password}}); len(h.Waits) != 0 {
|
||||
t.Fatalf("a healthy answer kept its waits: %+v", h)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWaitingWithNothingIsUnhealthy(t *testing.T) {
|
||||
if a := Waiting("it waits"); a.State != StateUnhealthy {
|
||||
t.Fatalf("Waiting with no wait is %+v", a)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotAnAnswerIsAnError(t *testing.T) {
|
||||
if _, err := Read([]byte(`"yes"`)); err == nil {
|
||||
t.Fatal("a string read as an answer")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryFormOfPathOrAddressIsRefused(t *testing.T) {
|
||||
for _, words := range []string{
|
||||
"the source (/mnt/games)", `the source "/mnt/games"`, "the folder ~/games", "the folder ./games",
|
||||
"2001:db8::1", "fe80::1%eth0", "2001:0db8:0000:0000:0000:0000:0000:0001", "nas.lan:445", "192.0.2.7:445",
|
||||
`\\nas\games`, `C:\Users\x`, "smb://nas/games", "198.51.100.4",
|
||||
} {
|
||||
w := Wait{Part: "the source games", Secret: "smb-password-games", What: "the password of " + words}
|
||||
if err := w.Check(); err == nil {
|
||||
t.Errorf("%q passed as words for the operator", words)
|
||||
}
|
||||
}
|
||||
for _, words := range []string{"the password of the source games", "the username of the source games-2", "the API key, issued by the vendor"} {
|
||||
w := Wait{Part: "the source games", Secret: "smb-password-games", What: words}
|
||||
if err := w.Check(); err != nil {
|
||||
t.Errorf("%q refused: %v", words, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHealthyStateWithHealthyFalseIsUnhealthy(t *testing.T) {
|
||||
got, err := Read([]byte(`{"state":"healthy","healthy":false,"why":"x"}`))
|
||||
if err != nil || got.State != StateUnhealthy || !strings.Contains(got.Why, "contradict") {
|
||||
t.Fatalf("read as %+v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWaitingChecksItsWaits(t *testing.T) {
|
||||
if a := Waiting("it waits", Wait{Part: "p", What: "w"}); a.State != StateUnhealthy || !strings.Contains(a.Why, "neither") {
|
||||
t.Fatalf("Waiting with a malformed wait is %+v", a)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package stdio
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Log — a module's record of operations on the bus, kept as long as the log is (novox/hq ADR 0297). Not the
|
||||
// current value: that is state, and a log's state is its fold. Each entry is one operation on one key, its
|
||||
// sequence in the log is its identity, and the order of the log is the order of the operations. A module
|
||||
// declares each log it owns in its manifest (`logs`) and names it here as it named it there; a log is its
|
||||
// owner's alone, and the runtime refuses, with the reason, one it did not issue.
|
||||
|
||||
// LogEntry is one operation on one key, as the log holds it.
|
||||
type LogEntry struct {
|
||||
Sequence uint64 `json:"sequence"`
|
||||
Key string `json:"key"`
|
||||
Value json.RawMessage `json:"value"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// LogPage is one page of a read: its entries, in order, the sequence to read on from (0 when nothing
|
||||
// follows) and the log's last sequence when it was read.
|
||||
type LogPage struct {
|
||||
Entries []LogEntry `json:"entries"`
|
||||
Next uint64 `json:"next"`
|
||||
Last uint64 `json:"last"`
|
||||
}
|
||||
|
||||
// LogInfo is how much a log holds: its entries and bytes, its cap in bytes, and its first and last sequence.
|
||||
type LogInfo struct {
|
||||
Entries uint64 `json:"entries"`
|
||||
Bytes uint64 `json:"bytes"`
|
||||
MaxBytes uint64 `json:"max_bytes"`
|
||||
First uint64 `json:"first"`
|
||||
Last uint64 `json:"last"`
|
||||
}
|
||||
|
||||
// KeptLog is one log this module owns.
|
||||
type KeptLog struct{ name string }
|
||||
|
||||
// Log is a log this module owns, by its local name.
|
||||
func Log(name string) KeptLog { return KeptLog{name: name} }
|
||||
|
||||
const logTimeout = 30 * time.Second
|
||||
|
||||
func (l KeptLog) ask(verb string, params map[string]any) (json.RawMessage, error) {
|
||||
c, err := live()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
params["log"] = l.name
|
||||
return c.ask("mesh/log."+verb, params, logTimeout)
|
||||
}
|
||||
|
||||
func (l KeptLog) append(params map[string]any) (uint64, error) {
|
||||
raw, err := l.ask("append", params)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var appended struct {
|
||||
Sequence uint64 `json:"sequence"`
|
||||
}
|
||||
return appended.Sequence, json.Unmarshal(raw, &appended)
|
||||
}
|
||||
|
||||
// Append writes an entry for key, whatever came before it, and returns its sequence. A key is one or more
|
||||
// names of letters, digits, `-` and `_`, joined by `.`; a value is JSON of at most 256 KiB. A full log
|
||||
// refuses the entry: it never drops an older one to make room.
|
||||
func (l KeptLog) Append(key string, value any) (uint64, error) {
|
||||
return l.append(map[string]any{"key": key, "value": value})
|
||||
}
|
||||
|
||||
// AppendAt writes an entry for key only if the key's last entry is still at expect — the sequence the caller
|
||||
// read, 0 for a key with no entry yet — and returns its sequence. When another entry for the key came in
|
||||
// between, nothing is written and the error is ErrChanged, as for a lost State update: read again and
|
||||
// decide again.
|
||||
func (l KeptLog) AppendAt(key string, value any, expect uint64) (uint64, error) {
|
||||
return l.append(map[string]any{"key": key, "value": value, "expect": expect})
|
||||
}
|
||||
|
||||
// Read is the entries from the sequence from on, oldest first, at most limit of them (0 is the runtime's
|
||||
// page, 100; it gives at most 1000). keys narrows it: a key, with `*` for one name and `>` for the rest;
|
||||
// empty is every key. Read on from the page's Next until it is 0.
|
||||
func (l KeptLog) Read(from uint64, keys string, limit int) (*LogPage, error) {
|
||||
if limit < 0 {
|
||||
return nil, fmt.Errorf("a read of %s asks for at least no entries: limit %d", l.name, limit)
|
||||
}
|
||||
params := map[string]any{"from": from}
|
||||
if keys != "" {
|
||||
params["keys"] = keys
|
||||
}
|
||||
if limit > 0 {
|
||||
params["limit"] = limit
|
||||
}
|
||||
raw, err := l.ask("read", params)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var p LogPage
|
||||
return &p, json.Unmarshal(raw, &p)
|
||||
}
|
||||
|
||||
// Last is a key's last entry, or nil when the key has none.
|
||||
func (l KeptLog) Last(key string) (*LogEntry, error) {
|
||||
raw, err := l.ask("last", map[string]any{"key": key})
|
||||
if err != nil || len(raw) == 0 || string(raw) == "null" {
|
||||
return nil, err
|
||||
}
|
||||
var e LogEntry
|
||||
return &e, json.Unmarshal(raw, &e)
|
||||
}
|
||||
|
||||
// Info is how much the log holds and how much it may.
|
||||
func (l KeptLog) Info() (*LogInfo, error) {
|
||||
raw, err := l.ask("info", map[string]any{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var i LogInfo
|
||||
return &i, json.Unmarshal(raw, &i)
|
||||
}
|
||||
+354
@@ -0,0 +1,354 @@
|
||||
package stdio
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// logRuntime is the runtime's side of mesh/log.append, read, last and info as the bus does them (novox/hq
|
||||
// ADR 0297): one sequence per entry across the log, an append with expect refused unless the key's last
|
||||
// entry is still at it (0 for a key with none), that refusal answered with CodeChanged as a lost state
|
||||
// update is, and a read paged by its limit with the sequence to read on from. asked keeps every ask's
|
||||
// method and params, as the bundle sent them.
|
||||
type logRuntime struct {
|
||||
mu sync.Mutex
|
||||
entries []LogEntry
|
||||
asked []message
|
||||
}
|
||||
|
||||
func (r *logRuntime) last(key string) uint64 {
|
||||
for i := len(r.entries) - 1; i >= 0; i-- {
|
||||
if r.entries[i].Key == key {
|
||||
return r.entries[i].Sequence
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// matches is a key pattern as the runtime reads it: `*` one name, `>` the rest.
|
||||
func matches(pattern, key string) bool {
|
||||
if pattern == "" {
|
||||
return true
|
||||
}
|
||||
p, k := strings.Split(pattern, "."), strings.Split(key, ".")
|
||||
for i, t := range p {
|
||||
if t == ">" {
|
||||
return len(k) > i
|
||||
}
|
||||
if i >= len(k) || (t != "*" && t != k[i]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return len(p) == len(k)
|
||||
}
|
||||
|
||||
func (r *logRuntime) answer(m message) map[string]any {
|
||||
var p struct {
|
||||
Log string `json:"log"`
|
||||
Key string `json:"key"`
|
||||
Value json.RawMessage `json:"value"`
|
||||
Expect *uint64 `json:"expect"`
|
||||
From uint64 `json:"from"`
|
||||
Keys string `json:"keys"`
|
||||
Limit int `json:"limit"`
|
||||
}
|
||||
_ = json.Unmarshal(m.Params, &p)
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.asked = append(r.asked, m)
|
||||
if p.Log != "changes" {
|
||||
return map[string]any{"error": map[string]any{"code": -32000, "message": "issues was issued no log " + p.Log}}
|
||||
}
|
||||
switch strings.TrimPrefix(m.Method, "mesh/log.") {
|
||||
case "append":
|
||||
if p.Expect != nil && r.last(p.Key) != *p.Expect {
|
||||
return map[string]any{"error": map[string]any{"code": CodeChanged, "message": p.Key + ": the key was written by another writer first"}}
|
||||
}
|
||||
seq := uint64(len(r.entries)) + 1
|
||||
r.entries = append(r.entries, LogEntry{Sequence: seq, Key: p.Key, Value: p.Value,
|
||||
At: time.Date(2026, 10, 11, 18, 0, int(seq), 0, time.UTC)})
|
||||
return map[string]any{"result": map[string]any{"sequence": seq}}
|
||||
case "read":
|
||||
limit := p.Limit
|
||||
if limit == 0 {
|
||||
limit = 100
|
||||
}
|
||||
page := map[string]any{"entries": []LogEntry{}, "next": 0, "last": len(r.entries)}
|
||||
var got []LogEntry
|
||||
for _, e := range r.entries {
|
||||
if e.Sequence < p.From || !matches(p.Keys, e.Key) {
|
||||
continue
|
||||
}
|
||||
if len(got) == limit {
|
||||
page["next"] = e.Sequence
|
||||
break
|
||||
}
|
||||
got = append(got, e)
|
||||
}
|
||||
if got != nil {
|
||||
page["entries"] = got
|
||||
}
|
||||
return map[string]any{"result": page}
|
||||
case "last":
|
||||
seq := r.last(p.Key)
|
||||
if seq == 0 {
|
||||
return map[string]any{"result": nil}
|
||||
}
|
||||
return map[string]any{"result": r.entries[seq-1]}
|
||||
case "info":
|
||||
var bytes int
|
||||
for _, e := range r.entries {
|
||||
bytes += len(e.Value)
|
||||
}
|
||||
info := map[string]any{"entries": len(r.entries), "bytes": bytes, "max_bytes": 1 << 30, "first": 0, "last": len(r.entries)}
|
||||
if len(r.entries) > 0 {
|
||||
info["first"] = 1
|
||||
}
|
||||
return map[string]any{"result": info}
|
||||
}
|
||||
return map[string]any{"error": map[string]any{"code": -32601, "message": "no " + m.Method}}
|
||||
}
|
||||
|
||||
// logging serves a bundle whose one tool, "log", runs run against r, answering each mesh/log ask beside
|
||||
// the loop as the runtime does. It returns what the tool answered.
|
||||
func logging(t *testing.T, r *logRuntime, run func() (any, error)) map[string]any {
|
||||
t.Helper()
|
||||
t.Setenv(ServedAs, "issues")
|
||||
inR, inW := io.Pipe()
|
||||
outR, outW := io.Pipe()
|
||||
go func() {
|
||||
_ = ServeOn("", []Tool{{Name: "log", Run: func(map[string]any) (any, error) { return run() }}}, inR, outW)
|
||||
}()
|
||||
var wmu sync.Mutex
|
||||
send := func(m map[string]any) {
|
||||
m["jsonrpc"] = "2.0"
|
||||
b, _ := json.Marshal(m)
|
||||
wmu.Lock()
|
||||
defer wmu.Unlock()
|
||||
_, _ = inW.Write(append(b, '\n'))
|
||||
}
|
||||
done := make(chan map[string]any, 1)
|
||||
go func() {
|
||||
lines := bufio.NewScanner(outR)
|
||||
lines.Buffer(make([]byte, 1<<20), 1<<24)
|
||||
for lines.Scan() {
|
||||
var m message
|
||||
if json.Unmarshal(lines.Bytes(), &m) != nil {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(m.Method, "mesh/log.") {
|
||||
go func(m message) {
|
||||
a := r.answer(m)
|
||||
a["id"] = m.ID
|
||||
send(a)
|
||||
}(m)
|
||||
continue
|
||||
}
|
||||
var answer map[string]any
|
||||
_ = json.Unmarshal(lines.Bytes(), &answer)
|
||||
if answer["id"] == float64(1) {
|
||||
done <- answer
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
send(map[string]any{"id": 1, "method": "tools/call", "params": map[string]any{"name": "log"}})
|
||||
t.Cleanup(func() { _ = inW.Close() })
|
||||
select {
|
||||
case a := <-done:
|
||||
return a
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("the bundle did not answer")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// params is what the bundle sent with its n-th ask, and the ask's method.
|
||||
func (r *logRuntime) params(t *testing.T, n int) (string, map[string]any) {
|
||||
t.Helper()
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if n >= len(r.asked) {
|
||||
t.Fatalf("the bundle asked %d times, not %d", len(r.asked), n+1)
|
||||
}
|
||||
var p map[string]any
|
||||
_ = json.Unmarshal(r.asked[n].Params, &p)
|
||||
return r.asked[n].Method, p
|
||||
}
|
||||
|
||||
// A bundle appends to, reads, and measures its log through the runtime (novox/hq ADR 0297): each verb is
|
||||
// asked as mesh/log.<verb> naming the log, its answer is handed back whole, a key with no entry is nil,
|
||||
// and a read pages on from Next until it is 0.
|
||||
func TestABundleKeepsALogThroughTheRuntime(t *testing.T) {
|
||||
r := &logRuntime{}
|
||||
var (
|
||||
first, second, third uint64
|
||||
pages []*LogPage
|
||||
last, none *LogEntry
|
||||
info *LogInfo
|
||||
narrowed *LogPage
|
||||
)
|
||||
got := logging(t, r, func() (any, error) {
|
||||
changes := Log("changes")
|
||||
var err error
|
||||
if first, err = changes.Append("401", map[string]any{"op": "opened", "by": "fix@g14"}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if second, err = changes.Append("402", map[string]any{"op": "opened"}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if third, err = changes.AppendAt("401", map[string]any{"op": "moved", "to": "fixing"}, first); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, k := range []string{"403", "404", "405"} {
|
||||
if _, err = changes.Append(k, map[string]any{"op": "opened"}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
for from := uint64(1); ; {
|
||||
p, err := changes.Read(from, "", 2)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pages = append(pages, p)
|
||||
if p.Next == 0 {
|
||||
break
|
||||
}
|
||||
from = p.Next
|
||||
}
|
||||
if narrowed, err = changes.Read(0, "401", 0); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if last, err = changes.Last("401"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if none, err = changes.Last("999"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
info, err = changes.Info()
|
||||
return "done", err
|
||||
})
|
||||
if out, isError := text(t, got); isError || out != `"done"` {
|
||||
t.Fatalf("the tool answered %s", out)
|
||||
}
|
||||
if first != 1 || second != 2 || third != 3 {
|
||||
t.Fatalf("appended at %d, %d, %d", first, second, third)
|
||||
}
|
||||
|
||||
// Asked as the contract names them: the verb, the log, and only what was given.
|
||||
method, p := r.params(t, 0)
|
||||
if method != "mesh/log.append" || p["log"] != "changes" || p["key"] != "401" || p["value"].(map[string]any)["by"] != "fix@g14" {
|
||||
t.Fatalf("an append asked %s %v", method, p)
|
||||
}
|
||||
if _, has := p["expect"]; has {
|
||||
t.Fatalf("an Append without expect named one: %v", p)
|
||||
}
|
||||
if method, p = r.params(t, 2); method != "mesh/log.append" || p["expect"] != float64(1) {
|
||||
t.Fatalf("an AppendAt asked %s %v", method, p)
|
||||
}
|
||||
if method, p = r.params(t, 6); method != "mesh/log.read" || p["from"] != float64(1) || p["limit"] != float64(2) || p["keys"] != nil {
|
||||
t.Fatalf("a read asked %s %v", method, p)
|
||||
}
|
||||
|
||||
// Paged two at a time through six entries: three pages, every entry once, in order.
|
||||
if len(pages) != 3 {
|
||||
t.Fatalf("read %d pages, not 3", len(pages))
|
||||
}
|
||||
var seqs []uint64
|
||||
for _, pg := range pages {
|
||||
if pg.Last != 6 {
|
||||
t.Fatalf("a page said the last sequence is %d", pg.Last)
|
||||
}
|
||||
for _, e := range pg.Entries {
|
||||
seqs = append(seqs, e.Sequence)
|
||||
}
|
||||
}
|
||||
if fmt.Sprint(seqs) != "[1 2 3 4 5 6]" || pages[0].Next != 3 || pages[2].Next != 0 {
|
||||
t.Fatalf("paged %v, next %d then %d", seqs, pages[0].Next, pages[2].Next)
|
||||
}
|
||||
if e := pages[1].Entries[0]; e.Key != "401" || string(e.Value) != `{"op":"moved","to":"fixing"}` ||
|
||||
!e.At.Equal(time.Date(2026, 10, 11, 18, 0, 3, 0, time.UTC)) {
|
||||
t.Fatalf("an entry read as %+v", e)
|
||||
}
|
||||
|
||||
// Narrowed to a key, with the runtime's page: no limit sent.
|
||||
method, p = r.params(t, 9)
|
||||
if method != "mesh/log.read" || p["keys"] != "401" || p["limit"] != nil {
|
||||
t.Fatalf("a narrowed read asked %s %v", method, p)
|
||||
}
|
||||
if len(narrowed.Entries) != 2 || narrowed.Entries[0].Sequence != 1 || narrowed.Entries[1].Sequence != 3 || narrowed.Next != 0 {
|
||||
t.Fatalf("narrowed to 401: %+v", narrowed)
|
||||
}
|
||||
|
||||
if last == nil || last.Sequence != 3 || last.Key != "401" {
|
||||
t.Fatalf("the last of 401 is %+v", last)
|
||||
}
|
||||
if method, p = r.params(t, 10); method != "mesh/log.last" || p["key"] != "401" {
|
||||
t.Fatalf("a last asked %s %v", method, p)
|
||||
}
|
||||
if none != nil {
|
||||
t.Fatalf("a key with no entry has a last: %+v", none)
|
||||
}
|
||||
if info == nil || info.Entries != 6 || info.MaxBytes != 1<<30 || info.First != 1 || info.Last != 6 || info.Bytes == 0 {
|
||||
t.Fatalf("the log's info is %+v", info)
|
||||
}
|
||||
if method, p = r.params(t, 12); method != "mesh/log.info" || p["log"] != "changes" || len(p) != 1 {
|
||||
t.Fatalf("an info asked %s %v", method, p)
|
||||
}
|
||||
}
|
||||
|
||||
// An AppendAt on a stale expect is ErrChanged and writes nothing: another entry for the key came in
|
||||
// between. Expect 0 is a key with no entry yet, and is ErrChanged once the key has one. A log not issued
|
||||
// is refused with the runtime's reason, and is not ErrChanged.
|
||||
func TestAnAppendAtAStaleSequenceIsErrChangedAndWritesNothing(t *testing.T) {
|
||||
r := &logRuntime{}
|
||||
var errs []error
|
||||
got := logging(t, r, func() (any, error) {
|
||||
changes := Log("changes")
|
||||
read, err := changes.AppendAt("401", map[string]any{"op": "opened"}, 0)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Another writer is first.
|
||||
if _, err := changes.Append("401", map[string]any{"op": "noted"}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
_, stale := changes.AppendAt("401", map[string]any{"op": "moved"}, read)
|
||||
_, again := changes.AppendAt("401", map[string]any{"op": "opened"}, 0)
|
||||
_, other := Log("elsewhere").Append("401", map[string]any{})
|
||||
errs = []error{stale, again, other}
|
||||
return "done", nil
|
||||
})
|
||||
if out, isError := text(t, got); isError || out != `"done"` {
|
||||
t.Fatalf("the tool answered %s", out)
|
||||
}
|
||||
if !errors.Is(errs[0], ErrChanged) || !errors.Is(errs[1], ErrChanged) {
|
||||
t.Fatalf("a stale expect answered %v and %v, not ErrChanged", errs[0], errs[1])
|
||||
}
|
||||
if errs[2] == nil || errors.Is(errs[2], ErrChanged) || !strings.Contains(errs[2].Error(), "no log elsewhere") {
|
||||
t.Fatalf("a log not issued answered %v", errs[2])
|
||||
}
|
||||
if len(r.entries) != 2 {
|
||||
t.Fatalf("the log holds %d entries, not 2: a refused append wrote", len(r.entries))
|
||||
}
|
||||
}
|
||||
|
||||
// A read asking for fewer than no entries is refused before the runtime is asked.
|
||||
func TestAReadOfANegativeLimitIsRefused(t *testing.T) {
|
||||
r := &logRuntime{}
|
||||
var err error
|
||||
logging(t, r, func() (any, error) {
|
||||
_, err = Log("changes").Read(0, "", -1)
|
||||
return "done", nil
|
||||
})
|
||||
if err == nil || len(r.asked) != 0 {
|
||||
t.Fatalf("a negative limit answered %v after %d asks", err, len(r.asked))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
// Package outward is the content rule (novox/hq ADR 0310 §4, to-be 45 §5, research 028 Q8, ADR 0227): what
|
||||
// may leave the mesh, and what an answer may carry at all. One rule, here, so every module that answers outward
|
||||
// applies the same one rather than a copy: the operator channel's router and every node-reads answer.
|
||||
//
|
||||
// Two strengths:
|
||||
//
|
||||
// - Check is the operator channel's: roles and words only. A message carrying an address, a path or
|
||||
// anything shaped like a secret is refused, because a channel such as Telegram is not end-to-end
|
||||
// encrypted for bots and this rule is the only thing between a condition's words and someone else's
|
||||
// server. Deliberately wider than it must be: a commit id or a long random name is refused too.
|
||||
// - CheckSecret is the secret class alone: what leaves the mesh nowhere and is answered nowhere. A path
|
||||
// or an address is what a file read or an operator's approval is about; a secret's shape never is.
|
||||
//
|
||||
// A refusal says its class and a few words, never the text itself.
|
||||
package outward
|
||||
|
||||
import (
|
||||
"math"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// Refusal says why a text may not leave: the class of what it carried, never the text itself.
|
||||
type Refusal struct {
|
||||
Class string // address, path or secret
|
||||
What string // a few words: "an IPv4 address", "a URL", …
|
||||
}
|
||||
|
||||
func (r Refusal) String() string { return r.Class + " (" + r.What + ")" }
|
||||
|
||||
var (
|
||||
reURL = regexp.MustCompile(`(?i)\b[a-z][a-z0-9+.-]*://`)
|
||||
reEmail = regexp.MustCompile(`[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,}`)
|
||||
reIPv4 = regexp.MustCompile(`\b\d{1,3}(\.\d{1,3}){3}\b`)
|
||||
reIPv6 = regexp.MustCompile(`(?i)(^|[^0-9a-z:])(([0-9a-f]{1,4}:){4,7}[0-9a-f]{1,4}|([0-9a-f]{1,4}:)*[0-9a-f]{0,4}::([0-9a-f]{1,4}:)*[0-9a-f]{0,4})([^0-9a-z:]|$)`)
|
||||
reMAC = regexp.MustCompile(`(?i)\b([0-9a-f]{2}[:-]){5}[0-9a-f]{2}\b`)
|
||||
rePEM = regexp.MustCompile(`-----BEGIN [A-Z ]+-----`)
|
||||
reJWT = regexp.MustCompile(`\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}`)
|
||||
reBotToken = regexp.MustCompile(`\b\d{6,}:[A-Za-z0-9_-]{30,}`)
|
||||
reKnown = regexp.MustCompile(`\b(gh[pousr]_[A-Za-z0-9]{20,}|glpat-[A-Za-z0-9_-]{16,}|sk-[A-Za-z0-9_-]{16,}|xox[abprs]-[A-Za-z0-9-]{10,}|AKIA[0-9A-Z]{16})`)
|
||||
reAssigned = regexp.MustCompile(`(?i)\b(password|passwd|passphrase|secret|token|api[_-]?key|apikey|credential|private[_-]?key)\s*[=:]\s*\S`)
|
||||
// reNamedSecret is a value given to a name that ends in a secret's word after `_` or `-`, as an environment
|
||||
// or configuration name is written: MESH_TOKEN=, DB_PASS=, aws_secret_access_key = …, SUDO_PASSWORD value.
|
||||
// The word ends the name: what follows it is `=`, `:` or a space and then a value.
|
||||
reNamedSecret = regexp.MustCompile(`(?i)(^|[^a-z0-9_-])[a-z0-9]+[_-]([a-z0-9]+[_-])*(password|passwd|passphrase|pass|pwd|secret|token|api[_-]?key|apikey|access[_-]?key|secret[_-]?key|private[_-]?key|credentials?|psk)(\s*[=:]\s*|\s+)[^\s=:]`)
|
||||
// reURLCredential is a password inside an address: scheme://user:password@host.
|
||||
reURLCredential = regexp.MustCompile(`(?i)\b[a-z][a-z0-9+.-]*://[^/\s:@]+:[^/\s@]+@`)
|
||||
// reSecretRun is a run CheckSecret judges: reRun with `-`, so a base64url value is judged whole.
|
||||
reSecretRun = regexp.MustCompile(`[A-Za-z0-9+/=_-]{20,}`)
|
||||
reUUID = regexp.MustCompile(`^(?i)[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
|
||||
reHex = regexp.MustCompile(`(?i)\b[0-9a-f]{32,}\b`)
|
||||
reRun = regexp.MustCompile(`[A-Za-z0-9+/=_]{20,}`)
|
||||
reWinPath = regexp.MustCompile(`(?i)\b[a-z]:\\`)
|
||||
)
|
||||
|
||||
// topLevel are names that end a host name — the generic and country ones a mesh's names use, and the
|
||||
// private ones (.internal, .lan, .home, .local) a mesh is likelier to.
|
||||
var topLevel = map[string]bool{}
|
||||
|
||||
func init() {
|
||||
for _, t := range strings.Fields(`com net org edu gov mil int io dev app cloud ai co me info biz xyz
|
||||
site online tech page link
|
||||
be nl de fr uk lu eu ch at it es pt se no dk fi pl cz us ca au nz jp cn ru in br ie
|
||||
internal lan home local localdomain corp intranet private arpa test example invalid localhost`) {
|
||||
topLevel[t] = true
|
||||
}
|
||||
}
|
||||
|
||||
// Check says whether a text may leave the mesh, and when not, why.
|
||||
func Check(text string) (Refusal, bool) {
|
||||
switch {
|
||||
case reURL.MatchString(text):
|
||||
return Refusal{"address", "a URL"}, false
|
||||
case reEmail.MatchString(text):
|
||||
return Refusal{"address", "a mail address"}, false
|
||||
case reIPv4.MatchString(text):
|
||||
return Refusal{"address", "an IPv4 address"}, false
|
||||
case reMAC.MatchString(text):
|
||||
return Refusal{"address", "a hardware address"}, false
|
||||
case reIPv6.MatchString(text):
|
||||
return Refusal{"address", "an IPv6 address"}, false
|
||||
}
|
||||
if refusal, ok := secretShape(text); !ok {
|
||||
return refusal, false
|
||||
}
|
||||
if reWinPath.MatchString(text) {
|
||||
return Refusal{"path", "a drive path"}, false
|
||||
}
|
||||
if refusal, ok := randomRun(text); !ok {
|
||||
return refusal, false
|
||||
}
|
||||
for _, word := range strings.FieldsFunc(text, func(r rune) bool {
|
||||
return unicode.IsSpace(r) || strings.ContainsRune("\"'`()[]{}<>,;|", r)
|
||||
}) {
|
||||
w := strings.TrimRight(word, ".:!?")
|
||||
if isPath(w) {
|
||||
return Refusal{"path", "a file path"}, false
|
||||
}
|
||||
if isHostName(w) {
|
||||
return Refusal{"address", "a host name"}, false
|
||||
}
|
||||
}
|
||||
return Refusal{}, true
|
||||
}
|
||||
|
||||
// CheckSecret says whether a text carries a secret's shape — the one class that leaves the mesh nowhere. It is
|
||||
// what an ask's whole words (asks.Ask.Whole, novox/hq issue 383) are held to: on a kind that proves who answers,
|
||||
// a path or an address is what the operator approves and may be shown, a secret never. A path is read as one:
|
||||
// a run with a slash in it is judged piece by piece between the slashes, so "/mnt/Photos_2024/Jochen" is a
|
||||
// path and not a random string (the review of 2026-10-10); the named shapes — a key block, a signed token, a
|
||||
// bot token, a known token shape, a value given to a secret's name, a long hexadecimal string — hold whatever
|
||||
// the run holds.
|
||||
func CheckSecret(text string) (Refusal, bool) {
|
||||
if refusal, ok := secretShape(text); !ok {
|
||||
return refusal, false
|
||||
}
|
||||
return randomRunOutsidePaths(text)
|
||||
}
|
||||
|
||||
// randomRunOutsidePaths judges every long run of a value's characters, a path's pieces between its slashes
|
||||
// alone (the review of mesh-sdk #23): a run is read as a path only where it is one — it starts with a slash,
|
||||
// follows a dot or a tilde (a host's or a home's path), or a name's `=` gives it one — so a base64 value, whose
|
||||
// slashes fall anywhere, is judged whole. A UUID is a name, never a secret.
|
||||
func randomRunOutsidePaths(text string) (Refusal, bool) {
|
||||
for _, loc := range reSecretRun.FindAllStringIndex(text, -1) {
|
||||
var before byte
|
||||
if loc[0] > 0 {
|
||||
before = text[loc[0]-1]
|
||||
}
|
||||
run := text[loc[0]:loc[1]]
|
||||
// Standard base64 ends in its padding: such a run is a value, even when it starts with a slash.
|
||||
padded := strings.HasSuffix(run, "=") && !strings.Contains(strings.TrimRight(run, "="), "=")
|
||||
for i, seg := range strings.Split(run, "=") {
|
||||
path := !padded && (strings.HasPrefix(seg, "/") || (i == 0 && (before == '.' || before == '~')))
|
||||
pieces := []string{seg}
|
||||
if path {
|
||||
pieces = strings.Split(seg, "/")
|
||||
}
|
||||
for _, piece := range pieces {
|
||||
if len(piece) >= 20 && !reUUID.MatchString(piece) && looksRandomValue(piece) {
|
||||
return Refusal{"secret", "a long random-looking string"}, false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return Refusal{}, true
|
||||
}
|
||||
|
||||
// secretShape is the secret shapes a pattern names.
|
||||
func secretShape(text string) (Refusal, bool) {
|
||||
switch {
|
||||
case rePEM.MatchString(text):
|
||||
return Refusal{"secret", "a key block"}, false
|
||||
case reJWT.MatchString(text):
|
||||
return Refusal{"secret", "a signed token"}, false
|
||||
case reBotToken.MatchString(text):
|
||||
return Refusal{"secret", "a bot token"}, false
|
||||
case reKnown.MatchString(text):
|
||||
return Refusal{"secret", "a known token shape"}, false
|
||||
case reURLCredential.MatchString(text):
|
||||
return Refusal{"secret", "a password in an address"}, false
|
||||
case reAssigned.MatchString(text), reNamedSecret.MatchString(text):
|
||||
return Refusal{"secret", "a value given to a secret's name"}, false
|
||||
case reHex.MatchString(text):
|
||||
return Refusal{"secret", "a long hexadecimal string"}, false
|
||||
}
|
||||
return Refusal{}, true
|
||||
}
|
||||
|
||||
// randomRun is a long unbroken run of characters spread as a random string's are.
|
||||
func randomRun(text string) (Refusal, bool) {
|
||||
for _, run := range reRun.FindAllString(text, -1) {
|
||||
if looksRandom(run) {
|
||||
return Refusal{"secret", "a long random-looking string"}, false
|
||||
}
|
||||
}
|
||||
return Refusal{}, true
|
||||
}
|
||||
|
||||
// isPath: absolute, home-relative or dot-relative, or two separators deep. A mesh address names one
|
||||
// machine and one tool (`ace/postgres.query`) and has one; a ratio ("3/4") has digits only.
|
||||
func isPath(w string) bool {
|
||||
if w == "" {
|
||||
return false
|
||||
}
|
||||
if strings.HasPrefix(w, "/") && len(w) > 1 {
|
||||
return true
|
||||
}
|
||||
for _, p := range []string{"~/", "./", "../", "$HOME", "${"} {
|
||||
if strings.HasPrefix(w, p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return strings.Count(w, "/") >= 2 || strings.Contains(w, "\\")
|
||||
}
|
||||
|
||||
// isHostName: two names or more, the last a top-level one. A condition key's last name is its kind
|
||||
// (`machine.ace.silent`), which none of these is.
|
||||
func isHostName(w string) bool {
|
||||
w = strings.ToLower(w)
|
||||
if w == "localhost" {
|
||||
return true
|
||||
}
|
||||
parts := strings.Split(w, ".")
|
||||
if len(parts) < 2 {
|
||||
return false
|
||||
}
|
||||
for _, p := range parts {
|
||||
if p == "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return topLevel[parts[len(parts)-1]]
|
||||
}
|
||||
|
||||
// looksRandom: letters and digits mixed, and the characters spread as a random string's are. A
|
||||
// sentence's words are separated, so only an unbroken run reaches here.
|
||||
func looksRandom(s string) bool {
|
||||
var letters, digits int
|
||||
counts := map[rune]int{}
|
||||
for _, r := range s {
|
||||
counts[r]++
|
||||
switch {
|
||||
case unicode.IsLetter(r):
|
||||
letters++
|
||||
case unicode.IsDigit(r):
|
||||
digits++
|
||||
}
|
||||
}
|
||||
if letters == 0 || digits == 0 {
|
||||
// One class only: a word, or a number. A long number of digits alone is a count or a time.
|
||||
return letters > 0 && hasUpperAndLower(s) && entropy(counts, len(s)) >= 4.0
|
||||
}
|
||||
return entropy(counts, len(s)) >= 3.3
|
||||
}
|
||||
|
||||
// looksRandomValue is looksRandom for a value in a file: letters and digits mixed and spread as a random string's
|
||||
// are; letters alone when the case changes as a random string's does — about as many capitals as small letters,
|
||||
// where a CamelCase name (sshd's PasswordAuthentication) has one a word.
|
||||
func looksRandomValue(s string) bool {
|
||||
var upper, lower, digits int
|
||||
counts := map[rune]int{}
|
||||
for _, r := range s {
|
||||
counts[r]++
|
||||
switch {
|
||||
case unicode.IsUpper(r):
|
||||
upper++
|
||||
case unicode.IsLower(r):
|
||||
lower++
|
||||
case unicode.IsDigit(r):
|
||||
digits++
|
||||
}
|
||||
}
|
||||
if upper+lower == 0 {
|
||||
return false
|
||||
}
|
||||
if digits > 0 {
|
||||
return entropy(counts, len(s)) >= 3.3
|
||||
}
|
||||
share := float64(upper) / float64(upper+lower)
|
||||
return share >= 0.3 && share <= 0.7 && entropy(counts, len(s)) >= 3.5
|
||||
}
|
||||
|
||||
func hasUpperAndLower(s string) bool {
|
||||
return strings.IndexFunc(s, unicode.IsUpper) >= 0 && strings.IndexFunc(s, unicode.IsLower) >= 0
|
||||
}
|
||||
|
||||
func entropy(counts map[rune]int, n int) float64 {
|
||||
e := 0.0
|
||||
for _, c := range counts {
|
||||
p := float64(c) / float64(n)
|
||||
e -= p * math.Log2(p)
|
||||
}
|
||||
return e
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
package outward
|
||||
|
||||
// The operator channel's cases, carried here with the rule (novox/hq ADR 0310 §4): the rule moved, its meaning did not.
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRolesAndWordsMayLeave(t *testing.T) {
|
||||
for _, s := range []string{
|
||||
"URGENT: the home server has not been heard for 15 min",
|
||||
"key: provider.keycloak.ace.gitea.failing",
|
||||
"more: conditions show machine.ace.silent",
|
||||
"since: 2026-10-06 12:30 UTC",
|
||||
"call call-1759752000123456789-12 has run past its bound of 10 min",
|
||||
"ace/postgres.query answers no more",
|
||||
"3/4 machines answered; plan 41 waits on build 7f3a9c1e",
|
||||
"the bus advisory slow-consumer for ace_records",
|
||||
"core.controller.novox.rolled-back",
|
||||
"seat node-notifier has no live holder",
|
||||
"HELD BACK: 5 message(s) over the cap of 20 an hour",
|
||||
"STILL OPEN after 1.5 h: the controller's event loop takes no message",
|
||||
"node-engine and node tools builds differ from the plan's",
|
||||
} {
|
||||
if r, ok := Check(s); !ok {
|
||||
t.Errorf("refused %q: %s", s, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAddressAPathOrASecretMayNot(t *testing.T) {
|
||||
for s, class := range map[string]string{
|
||||
"cannot reach 192.168.1.20": "address",
|
||||
"listening on 10.0.0.7:5432": "address",
|
||||
"route fd00:1234:5678::1 is gone": "address",
|
||||
"fe80::1 answered": "address",
|
||||
"2001:db8:0:0:0:0:2:1 answered": "address",
|
||||
"see https://git.example.org/x": "address",
|
||||
"git.novox.internal does not answer": "address",
|
||||
"the mail for admin@example.org bounced": "address",
|
||||
"zurag.be is down": "address",
|
||||
"localhost refused": "address",
|
||||
"the card aa:bb:cc:dd:ee:ff went away": "address",
|
||||
"/var/lib/mesh-controller is full": "path",
|
||||
"~/.config/hal/env changed": "path",
|
||||
"read ./grants.json": "path",
|
||||
"services/postgres/data/pg_hba.conf": "path",
|
||||
"C:\\Users\\x": "path",
|
||||
"token 123456789:AAHdqTcvCH1vGWJxfSeofSAs0K5PALDsaw": "secret",
|
||||
"ghp_abcdefghijklmnopqrstuvwxyz0123456789": "secret",
|
||||
"password=hunter2": "secret",
|
||||
"api_key: x": "secret",
|
||||
"-----BEGIN OPENSSH PRIVATE KEY-----": "secret",
|
||||
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0": "secret",
|
||||
"commit 3f2a9c1e5b7d4f6a8c0e2b4d6f8a0c2e4b6d8f0a": "secret",
|
||||
"value Zx8Qp2Lm9Rt4Vb7Nc1Kd5Hs3Jf6": "secret",
|
||||
} {
|
||||
r, ok := Check(s)
|
||||
if ok {
|
||||
t.Errorf("let %q leave", s)
|
||||
continue
|
||||
}
|
||||
if r.Class != class {
|
||||
t.Errorf("%q refused as %s, want %s", s, r, class)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The secret class alone (novox/hq ADR 0310 §4, issue 383): a path and an address pass, a path's pieces are
|
||||
// judged between its slashes, and every named secret shape and a random run outside a path are refused.
|
||||
func TestCheckSecretRefusesASecretsShapeAndNothingElse(t *testing.T) {
|
||||
for text, may := range map[string]bool{
|
||||
"+ shares: photos=/mnt/Photos_2024/Jochen": true,
|
||||
"+ sources: games=smb://nas.lan/Recalbox_Games2024": true,
|
||||
"+ library: /srv/media/Series_Archive/Season01": true,
|
||||
"root:x:0:0::/root:/usr/bin/bash": true,
|
||||
"nameserver 192.168.1.1": true,
|
||||
"drwxr-x--- jochen users /home/jochen": true,
|
||||
"+ key: xK9mQ2vL8pR4tW7yB3nF6hJ1dG5sA0zC": false,
|
||||
"-----BEGIN OPENSSH PRIVATE KEY-----": false,
|
||||
"password=hunter2": false,
|
||||
"Authorization: token ghp_abcdefghijklmnopqrstuvwxyz0123456789": false,
|
||||
"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0": false,
|
||||
"sha 3f2a9c1e5b7d4f6a8c0e2b4d6f8a0c2e4b6d8f0a": false,
|
||||
} {
|
||||
r, ok := CheckSecret(text)
|
||||
if ok != may {
|
||||
t.Errorf("CheckSecret(%q) = %v (%s), want %v", text, ok, r, may)
|
||||
}
|
||||
if !ok && r.Class != "secret" {
|
||||
t.Errorf("CheckSecret(%q) refused as %s: only the secret class is refused there", text, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The review of mesh-sdk #23: the shapes node-reads meets in env files, unit files, configurations and
|
||||
// history files, each refused by its shape whatever the value looks like.
|
||||
func TestCheckSecretRefusesTheShapesOfConfigurationFiles(t *testing.T) {
|
||||
for _, text := range []string{
|
||||
"url = nats://user:pa55word@host:4222",
|
||||
"DATABASE_URL=postgres://app:hunter2@db/app",
|
||||
"MESH_TOKEN=s3cr3t-v4lue",
|
||||
"DB_PASS=hunter2",
|
||||
"export GITEA_SECRET=abc",
|
||||
"Environment=APP_API_KEY=xyz",
|
||||
"SUDO_PASSWORD hunter2",
|
||||
"aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
"password = hunter2",
|
||||
"wifi_psk: correcthorse",
|
||||
"value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
} {
|
||||
if r, ok := CheckSecret(text); ok || r.Class != "secret" {
|
||||
t.Errorf("CheckSecret(%q) let it through (%v)", text, r)
|
||||
}
|
||||
}
|
||||
// What a configuration file says that is not a secret.
|
||||
for _, text := range []string{
|
||||
"ExecStart=/usr/lib/systemd/systemd-resolved",
|
||||
"UUID=3f2a9c1e-5b7d-4f6a-8c0e-2b4d6f8a0c2e /home btrfs defaults 0 0",
|
||||
"password required pam_unix.so",
|
||||
"bypass = true",
|
||||
"PasswordAuthentication no",
|
||||
"url = https://git.example.org/novox/mesh-sdk.git",
|
||||
"HOME=/var/lib/mesh-reads",
|
||||
"TOKEN_FILE=/var/lib/app/token",
|
||||
} {
|
||||
if r, ok := CheckSecret(text); !ok {
|
||||
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Random values as base64 and base64url, wherever their slashes fall: every one is refused.
|
||||
func TestCheckSecretRefusesRandomValuesAsBase64(t *testing.T) {
|
||||
rng := rand.New(rand.NewSource(418))
|
||||
passed := map[string]int{}
|
||||
for i := 0; i < 1000; i++ {
|
||||
for _, n := range []int{16, 32} {
|
||||
b := make([]byte, n)
|
||||
rng.Read(b)
|
||||
for name, enc := range map[string]*base64.Encoding{"std": base64.StdEncoding, "url": base64.RawURLEncoding} {
|
||||
v := enc.EncodeToString(b)
|
||||
if _, ok := CheckSecret("key " + v); ok {
|
||||
passed[fmt.Sprintf("%s/%d", name, n)]++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(passed) > 0 {
|
||||
t.Errorf("random values passed as clean, of 1000 each: %v", passed)
|
||||
}
|
||||
}
|
||||
|
||||
// A CamelCase name is a word, not a value: the case changes once a word.
|
||||
func TestCheckSecretLetsACamelCaseNameThrough(t *testing.T) {
|
||||
for _, text := range []string{"PasswordAuthenticationIsOff", "ChallengeResponseAuthentication no", "KbdInteractiveAuthentication"} {
|
||||
if r, ok := CheckSecret(text); !ok {
|
||||
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package provider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
)
|
||||
|
||||
// The provider's consumers' credentials, from the file its machine's provisioner client fills (novox/hq ADR 0373,
|
||||
// issue 679).
|
||||
//
|
||||
// **The provider never asks the vault.** A provider in the tool runner has no account of its own, and the vault
|
||||
// hands a secret only to a node's provisioner client or to a module under an account of its own (ADR 0349 clause
|
||||
// 11, ADR 0370 clause 2). So the provider's template names `${consumers:<provision>}`, and its machine's client
|
||||
// fills that file with one entry per consumer bound to it for that provision, each credential asked of the vault in
|
||||
// the client's own name; the file is the provider's alone (mode 0600, owned). ConsumersFile is that file as the
|
||||
// helper's Vault: it reads the file on each ask, so a consumer that binds or a credential the vault remakes reaches
|
||||
// the provider at its next pass.
|
||||
|
||||
// ConsumerCredential is one entry of the file: the consumer, the vault's name of its credential, and its value.
|
||||
type ConsumerCredential struct {
|
||||
// Consumer and Module are the consumer's machine and module, Local its local name for the provision, if any.
|
||||
Consumer string `json:"consumer"`
|
||||
Module string `json:"module"`
|
||||
Local string `json:"local,omitempty"`
|
||||
// Credential is the vault's name of the consumer's credential, PairName's.
|
||||
Credential string `json:"credential"`
|
||||
// Value is the credential itself.
|
||||
Value string `json:"value"`
|
||||
}
|
||||
|
||||
// ConsumersFile answers a credential from the files the provider machine's client fills: one per provision whose
|
||||
// template names `${consumers:<provision>}`, each a JSON list of ConsumerCredential.
|
||||
type ConsumersFile struct {
|
||||
Paths []string
|
||||
}
|
||||
|
||||
// Value is the credential named name, from the first file carrying it. A file not there yet is passed over; one not
|
||||
// in the client's form is an error, never read as no consumers.
|
||||
func (f ConsumersFile) Value(_ context.Context, name string) ([]byte, error) {
|
||||
var missing []string
|
||||
for _, at := range f.Paths {
|
||||
raw, err := os.ReadFile(at)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
missing = append(missing, at)
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("the consumers' file %s: %w", at, err)
|
||||
}
|
||||
var entries []ConsumerCredential
|
||||
if err := json.Unmarshal(raw, &entries); err != nil {
|
||||
return nil, fmt.Errorf("the consumers' file %s is not the provisioner client's list of consumers", at)
|
||||
}
|
||||
for _, e := range entries {
|
||||
if e.Credential == name {
|
||||
if e.Value == "" {
|
||||
return nil, fmt.Errorf("the consumers' file %s carries %s with no value", at, name)
|
||||
}
|
||||
return []byte(e.Value), nil
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(missing) == len(f.Paths) {
|
||||
return nil, fmt.Errorf("%s: no consumers' file is filled yet (%v)", name, f.Paths)
|
||||
}
|
||||
return nil, fmt.Errorf("%s is in no consumers' file: the vault holds none for it yet, or its machine's client "+
|
||||
"has not filled it", name)
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package provider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The provider reads its consumers' credentials only from the file its machine's provisioner client fills (novox/hq
|
||||
// ADR 0373 clause 3, issue 679): one entry per consumer, naming the consumer and the vault's name for its credential,
|
||||
// and carrying the value. The helper is wired to it as its Vault and asks nothing of the vault itself.
|
||||
func TestTheConsumersFileAnswersEachConsumersCredential(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
at := filepath.Join(dir, "consumers.json")
|
||||
file := `[{"consumer":"ace","module":"letta","credential":"pair/novox/postgres-database/ace/letta","value":"s3cret"},
|
||||
{"consumer":"g14","module":"gitea","local":"cache","credential":"pair/novox/postgres-database/g14/gitea/cache","value":"other"}]`
|
||||
if err := os.WriteFile(at, []byte(file), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
v := ConsumersFile{Paths: []string{filepath.Join(dir, "absent.json"), at}}
|
||||
got, err := v.Value(context.Background(), PairName("novox", "postgres-database", "ace", "letta", ""))
|
||||
if err != nil || string(got) != "s3cret" {
|
||||
t.Fatalf("ace's letta: %q, %v; want s3cret", got, err)
|
||||
}
|
||||
got, err = v.Value(context.Background(), PairName("novox", "postgres-database", "g14", "gitea", "cache"))
|
||||
if err != nil || string(got) != "other" {
|
||||
t.Fatalf("g14's gitea under cache: %q, %v; want other", got, err)
|
||||
}
|
||||
// A consumer the file does not carry is waited for, and never told of a login its provider has not made.
|
||||
_, err = v.Value(context.Background(), PairName("novox", "postgres-database", "shanks", "photos", ""))
|
||||
if err == nil || !strings.Contains(err.Error(), "pair/novox/postgres-database/shanks/photos") {
|
||||
t.Fatalf("a consumer the file does not carry: %v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "s3cret") || strings.Contains(err.Error(), "other") {
|
||||
t.Fatalf("an error names a credential: %v", err)
|
||||
}
|
||||
|
||||
// Wired as the helper's Vault, the file is what makes each consumer's login.
|
||||
bus := newStandIn()
|
||||
bus.Put(context.Background(), AssignmentsBucket, "ace.letta", []byte(`{"node":"ace","module":"letta",
|
||||
"provisioning":{"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres"}]}}`))
|
||||
var made []string
|
||||
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v,
|
||||
Offers: map[string]Offer{"postgres-database": {Serves: map[string]any{"port": 5432}}},
|
||||
Make: func(_ context.Context, c Consumer) error {
|
||||
made = append(made, c.Entry.User+"="+string(c.Credential))
|
||||
return nil
|
||||
}, Logf: func(string, ...any) {}})
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(made) != 1 || made[0] != "mesh_ace_letta=s3cret" {
|
||||
t.Fatalf("made %v, want ace's letta under the file's credential", made)
|
||||
}
|
||||
}
|
||||
|
||||
// A file that is not the client's form is said, never read as no consumers.
|
||||
func TestAConsumersFileNotInItsFormIsRefused(t *testing.T) {
|
||||
at := filepath.Join(t.TempDir(), "consumers.json")
|
||||
if err := os.WriteFile(at, []byte(`{"not": "a list"}`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err := ConsumersFile{Paths: []string{at}}.Value(context.Background(), "pair/novox/p/ace/m")
|
||||
if err == nil || !strings.Contains(err.Error(), at) {
|
||||
t.Fatalf("a file not in the client's form: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package provider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Issue 1098: an assignment with `ended` set, or with a `state`, is not a standing assignment: the helper reads it
|
||||
// as a removed key, so the entry of its consumer is removed and the adapter hears it is gone. The fields the
|
||||
// controller adds change nothing.
|
||||
func TestAnEndedAssignmentIsReadAsARemovedKey(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
p := postgres(bus, v, m)
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.postgres-database"}) {
|
||||
t.Fatalf("entries %v", got)
|
||||
}
|
||||
for _, mark := range []map[string]any{{"ended": "2026-10-18T01:00:00Z"}, {"state": "pending"}} {
|
||||
raw := bus.get(AssignmentsBucket, "ace.letta")
|
||||
var a map[string]any
|
||||
if err := json.Unmarshal(raw, &a); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for k, v := range mark {
|
||||
a[k] = v
|
||||
}
|
||||
raw, _ = json.Marshal(a)
|
||||
_ = bus.Put(context.Background(), AssignmentsBucket, "ace.letta", raw)
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := bus.keys(BucketOf("postgres")); len(got) != 0 || len(out) != 1 || out[0].Status != Removed {
|
||||
t.Fatalf("%v: entries %v, outcomes %+v", mark, got, out)
|
||||
}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestStandingIsFalseOnlyForAnEndedOrPendingValue(t *testing.T) {
|
||||
for raw, want := range map[string]bool{
|
||||
`{"node":"a","module":"b"}`: true,
|
||||
`{"node":"a","module":"b","ended":"","state":"","x":{"y":1}}`: true,
|
||||
`{"node":"a","module":"b","ended":null}`: true,
|
||||
`{"node":"a","module":"b","build":"b1","commit":"c","rollout":"r","since":"t","artifact":{"name":"n"}}`: true,
|
||||
`{"node":"a","module":"b","ended":"2026-10-18T01:00:00Z"}`: false,
|
||||
`{"node":"a","module":"b","state":"pending"}`: false,
|
||||
} {
|
||||
if got := Standing([]byte(raw)); got != want {
|
||||
t.Errorf("%s: %v, want %v", raw, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
package harness
|
||||
|
||||
import "context"
|
||||
|
||||
var ctx = context.Background()
|
||||
@@ -0,0 +1,62 @@
|
||||
package harness
|
||||
|
||||
// The moved path started before its runtime answers asks again until it does (issue 943).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/provider"
|
||||
)
|
||||
|
||||
// earlyBus refuses its first watches as the runtime does before Serve runs, then answers as the bucket bus does.
|
||||
type earlyBus struct {
|
||||
*bucketBus
|
||||
refuse int
|
||||
}
|
||||
|
||||
func (b *earlyBus) Watch(ctx context.Context, bucket string, handle func(string, []byte)) error {
|
||||
if b.refuse > 0 {
|
||||
b.refuse--
|
||||
return errors.New("before Serve: a bundle reaches the bus through the runtime that launched it")
|
||||
}
|
||||
return b.bucketBus.Watch(ctx, bucket, handle)
|
||||
}
|
||||
|
||||
func TestAMovedPathStartedTooEarlyAsksAgainUntilTheRuntimeAnswers(t *testing.T) {
|
||||
w := newMovedWorld(t, map[[2]string]string{})
|
||||
w.run.FirstRetry = time.Millisecond
|
||||
w.run.loaded = false
|
||||
bus := &earlyBus{bucketBus: w.bus, refuse: 3}
|
||||
p := provider.New(provider.Options{Node: "novox", Module: "postgres", Bus: w.bus,
|
||||
Vault: provider.ConsumersFile{Paths: []string{w.file}}, Make: w.run.Make, Gone: w.run.Gone,
|
||||
Every: time.Hour, Logf: func(string, ...any) {}})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- w.run.Run(ctx, p, bus) }()
|
||||
deadline := time.After(time.Second)
|
||||
for {
|
||||
w.run.mu.Lock()
|
||||
loaded := w.run.loaded
|
||||
w.run.mu.Unlock()
|
||||
if loaded {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case err := <-done:
|
||||
cancel()
|
||||
t.Fatalf("Run stopped before the runtime answered: %v", err)
|
||||
case <-deadline:
|
||||
cancel()
|
||||
t.Fatal("the plans were never watched")
|
||||
case <-time.After(time.Millisecond):
|
||||
}
|
||||
}
|
||||
cancel()
|
||||
<-done
|
||||
if bus.refuse != 0 {
|
||||
t.Fatalf("%d refusals left", bus.refuse)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package harness
|
||||
|
||||
// postgres handed to the provisioner still serves the consumers the controller's grant file gives (issue 937): a
|
||||
// consumer not moved has no plan naming postgres, only its grant, until R1 (issue 682).
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// granting gives the moved world a grant file, as the controller writes it for the consumers not moved.
|
||||
func (w *movedWorld) granting(given ...map[string]any) {
|
||||
if w.h.Receives == "" {
|
||||
w.h.Receives = filepath.Join(w.t.TempDir(), "mesh.json")
|
||||
}
|
||||
dir := filepath.Dir(w.h.Receives)
|
||||
for _, g := range given {
|
||||
secret := filepath.Join(dir, g["as"].(string)+".secret")
|
||||
if err := os.WriteFile(secret, []byte("pw-"+g["as"].(string)+"\n"), 0o600); err != nil {
|
||||
w.t.Fatal(err)
|
||||
}
|
||||
g["secret"] = secret
|
||||
}
|
||||
if given == nil {
|
||||
given = []map[string]any{}
|
||||
}
|
||||
raw, _ := json.Marshal(map[string]any{"requirement": ProvisionName, "given": given})
|
||||
if err := os.WriteFile(w.h.Receives, raw, 0o600); err != nil {
|
||||
w.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// settled is enough passes and time for a retirement to be stable.
|
||||
func (w *movedWorld) settled() {
|
||||
for i := 0; i < 8; i++ {
|
||||
w.pass()
|
||||
}
|
||||
w.now = w.now.Add(15 * time.Minute)
|
||||
w.pass()
|
||||
w.pass()
|
||||
}
|
||||
|
||||
// The logins the grant file gives are asked for: made as the grant path makes them, never offered for retirement,
|
||||
// beside a moved consumer the helper makes from its plan.
|
||||
func TestAConsumerOnlyTheGrantFileGivesIsAskedForAndMade(t *testing.T) {
|
||||
bound, logins := bindings(1)
|
||||
w := newMovedWorld(t, bound)
|
||||
w.a.inv.Active = []string{"mesh_ace_baserow", "mesh_ace_letta", logins[0]}
|
||||
w.bus.bind("ace", "app1", nil)
|
||||
w.granting(
|
||||
map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}},
|
||||
map[string]any{"as": "mesh_ace_letta", "node": "ace",
|
||||
"values": map[string]any{"name": "letta", "extensions": []any{"vector"}}},
|
||||
)
|
||||
w.settled()
|
||||
if len(w.a.removed) != 0 || w.h.r.waiting != nil {
|
||||
t.Fatalf("retired %v, waiting %+v: the grant file's logins were counted unasked", w.a.removed, w.h.r.waiting)
|
||||
}
|
||||
made := map[string]Provision{}
|
||||
for _, c := range w.a.created {
|
||||
made[c.As] = c
|
||||
}
|
||||
letta := made["mesh_ace_letta"]
|
||||
if letta.Password != "pw-mesh_ace_letta" || letta.Consumer != "ace" ||
|
||||
!reflect.DeepEqual(letta.Values, map[string]any{"name": "letta", "extensions": []any{"vector"}}) {
|
||||
t.Fatalf("made %+v for letta, as its grant gives it", letta)
|
||||
}
|
||||
if _, ok := made["mesh_ace_baserow"]; !ok {
|
||||
t.Fatalf("baserow not made: %+v", w.a.created)
|
||||
}
|
||||
if _, ok := made[logins[0]]; !ok {
|
||||
t.Fatalf("the moved consumer not made: %+v", w.a.created)
|
||||
}
|
||||
}
|
||||
|
||||
// A login both a plan and the grant file name is the plan's: made once, with the plan's credential.
|
||||
func TestALoginAPlanNamesIsNotMadeFromTheGrantFile(t *testing.T) {
|
||||
bound, logins := bindings(1)
|
||||
w := newMovedWorld(t, bound)
|
||||
w.bus.bind("ace", "app1", nil)
|
||||
w.granting(map[string]any{"as": logins[0], "node": "ace", "values": map[string]any{"name": "app1"}})
|
||||
w.pass()
|
||||
w.pass()
|
||||
for _, c := range w.a.created {
|
||||
if c.As == logins[0] && c.Password != "pw-app1" {
|
||||
t.Fatalf("made %s with the grant's password %q, not the plan's", c.As, c.Password)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An unreadable grant file is not nobody asking (issue 241): nothing is retired while it cannot be read.
|
||||
func TestAnUnreadableGrantFileRetiresNothing(t *testing.T) {
|
||||
bound, logins := bindings(1)
|
||||
w := newMovedWorld(t, bound)
|
||||
w.a.inv.Active = []string{"mesh_ace_baserow", logins[0]}
|
||||
w.bus.bind("ace", "app1", nil)
|
||||
w.h.Receives = filepath.Join(t.TempDir(), "absent.json")
|
||||
w.settled()
|
||||
if len(w.a.removed) != 0 || w.h.r.waiting != nil {
|
||||
t.Fatalf("retired %v, waiting %+v with the grant file unreadable", w.a.removed, w.h.r.waiting)
|
||||
}
|
||||
}
|
||||
|
||||
// A login the grant file stops giving is retired by the usual rules: stable, then retired, never dropped.
|
||||
func TestALoginTheGrantFileStopsGivingIsRetired(t *testing.T) {
|
||||
bound, _ := bindings(1)
|
||||
w := newMovedWorld(t, bound)
|
||||
w.bus.bind("ace", "app1", nil)
|
||||
w.granting(
|
||||
map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}},
|
||||
map[string]any{"as": "mesh_ace_letta", "node": "ace", "values": map[string]any{"name": "letta"}},
|
||||
)
|
||||
w.pass()
|
||||
w.granting(map[string]any{"as": "mesh_ace_baserow", "node": "ace", "values": map[string]any{"name": "baserow"}})
|
||||
w.settled()
|
||||
if !reflect.DeepEqual(w.a.removed, []string{"mesh_ace_letta"}) || len(w.a.deleted) != 0 {
|
||||
t.Fatalf("retired %v, deleted %v", w.a.removed, w.a.deleted)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,572 @@
|
||||
// Package harness is the reconcile loop and the retirement of a provider that keeps a resource per consumer; the
|
||||
// module supplies an Adapter.
|
||||
package harness
|
||||
|
||||
// The reconcile loop every provider shares, as the TypeScript SDK's runProvisioner runs it
|
||||
// (@novox/mesh-sdk/provisioner, 0.1.12), line for line in behaviour. It moved here from the postgres and keycloak
|
||||
// providers, which each carried an identical copy (novox/hq ADR 0039: the loop is the SDK's, the adapter is the
|
||||
// module's; issue 849).
|
||||
//
|
||||
// Read the contributions the mesh delivered; bring each consumer's resource into being through the
|
||||
// adapter, under the login and password the mesh minted; retire what the mesh no longer asks for.
|
||||
// **A provider creates the credential the mesh minted, and seals nothing (novox/hq ADR 0048).**
|
||||
//
|
||||
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
|
||||
// whose create, check or secret has failed without one success in between for FailingAfter is
|
||||
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
|
||||
// again every SayAgainEvery while it lasts; the first success after that is `provisioner.recovered`.
|
||||
// The controller keeps the newest per provider and consumer and `status` names it. On 2026-10-05 the
|
||||
// identity provider failed every consumer 31,000 times in a day and said so only in its journal
|
||||
// (novox/hq issue 179).
|
||||
//
|
||||
// **A consumer the mesh stops asking for is retired, not withdrawn, and deleted only by a person
|
||||
// (novox/hq ADR 0230, the operator's model of 2026-10-06).** Retiring disables its access — reversibly —
|
||||
// and marks its login and data "to delete" with when and why; nothing is deleted. Asked for again, the
|
||||
// ordinary create re-enables it as it was. It is retired only once the same set has gone unasked in
|
||||
// StablePasses consecutive passes and for StableFor, and a set larger than the bound waits for a
|
||||
// person. retirement.go.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Provision is one consumer's resource to bring into being — everything the mesh derived and delivered.
|
||||
type Provision struct {
|
||||
// As is the login the mesh derived and gave the consumer to present.
|
||||
As string
|
||||
// Password is the one the mesh minted, read from the file the host unsealed.
|
||||
Password string
|
||||
// Values are what the consumer contributed (e.g. {"name": "letta", "extensions": ["vector"]}).
|
||||
Values map[string]any
|
||||
// Derived is what this provider's own definition derives for the consumer (novox/hq ADR 0201).
|
||||
Derived map[string]any
|
||||
// At is where the consumer is; Consumer is its node.
|
||||
At string
|
||||
Consumer string
|
||||
}
|
||||
|
||||
// Adapter is the per-service half.
|
||||
type Adapter interface {
|
||||
// Create brings the consumer into being under the mesh's login and password — and, for one that
|
||||
// was retired, re-enables it as it was and clears its mark to delete.
|
||||
Create(ctx context.Context, p Provision) error
|
||||
// Retire disables the consumer's access, reversibly, and marks its login and data to delete with
|
||||
// when and why. It deletes nothing (novox/hq ADR 0230). derived is what the mesh last derived,
|
||||
// remembered here; nil for a consumer this process did not make.
|
||||
Retire(ctx context.Context, as string, derived map[string]any, why string, at time.Time) error
|
||||
// Holds says whether the backend still holds the consumer exactly as p says. Read-only.
|
||||
Holds(ctx context.Context, p Provision) (bool, error)
|
||||
// Inventory is what the backend holds that the mesh made: consumers active and retired, read from
|
||||
// the backend itself so a restart forgets nothing. Read-only.
|
||||
Inventory(ctx context.Context) (Inventory, error)
|
||||
// Delete removes one retired consumer — its login and its data — for good. Only ever asked by a
|
||||
// person, through `cleanup delete`; the harness has checked it is retired and not asked for.
|
||||
Delete(ctx context.Context, r Retired) (freedBytes int64, err error)
|
||||
}
|
||||
|
||||
// Harness is the loop's settings and memory.
|
||||
type Harness struct {
|
||||
Resource string
|
||||
Receives string
|
||||
Adapter Adapter
|
||||
Every time.Duration // 5s
|
||||
VerifyEvery time.Duration // 60s
|
||||
HoldsTimeout time.Duration // 30s
|
||||
Log func(format string, args ...any)
|
||||
Now func() time.Time
|
||||
// Announce publishes one of the provider's standing events; nil announces nothing. Node is the
|
||||
// machine this provider runs on, said in each.
|
||||
Announce func(event string, body map[string]any)
|
||||
Node string
|
||||
// FailingAfter is how long a consumer fails without a success before it is announced (5m);
|
||||
// SayAgainEvery is how often it is announced again while it lasts (15m), so a controller that
|
||||
// missed the first hears the next, and a standing nobody repeats can be told from one that holds.
|
||||
FailingAfter time.Duration
|
||||
SayAgainEvery time.Duration
|
||||
// StablePasses (5) is how many consecutive passes must see the same set no longer asked for, and
|
||||
// StableFor (10m) how long it must have held since the first of them, before it is retired. RetireAtOnce (3) and RetireFraction (0.5) bound what is retired without a person:
|
||||
// more consumers than RetireAtOnce, or — where more than one is held — a larger share of those held
|
||||
// than RetireFraction, waits for `retire approve` (novox/hq ADR 0230).
|
||||
StablePasses int
|
||||
StableFor time.Duration
|
||||
RetireAtOnce int
|
||||
RetireFraction float64
|
||||
|
||||
// mu is held by a pass and by every tool a person asks, so the two never interleave.
|
||||
mu sync.Mutex
|
||||
verifiedAt time.Time
|
||||
r retirement
|
||||
applied map[string]appliedEntry
|
||||
lost map[string]brake
|
||||
waiting map[string]int
|
||||
failing map[string]failure
|
||||
trouble map[string]*standing
|
||||
cleared map[string]bool
|
||||
lastWarning string
|
||||
}
|
||||
|
||||
// standing is one consumer's unbroken run of failures: since when, how often, and the last error.
|
||||
type standing struct {
|
||||
node string
|
||||
since time.Time
|
||||
attempts int
|
||||
class string
|
||||
text string
|
||||
saidAt time.Time
|
||||
}
|
||||
|
||||
// The events a provider's standing is announced as (novox/hq ADR 0224). The controller derives the
|
||||
// permission to emit them for every module that receives contributions; no manifest lists them.
|
||||
const (
|
||||
EventFailing = "provisioner.failing"
|
||||
EventRecovered = "provisioner.recovered"
|
||||
)
|
||||
|
||||
// The classes of error a standing is announced with: what a person reading `status` needs to know
|
||||
// before reading the journal. An adapter may say better (Classifier).
|
||||
const (
|
||||
ClassCredentials = "credentials-rejected"
|
||||
ClassUnreachable = "unreachable"
|
||||
ClassSecret = "secret-unreadable"
|
||||
ClassRefused = "refused"
|
||||
)
|
||||
|
||||
// Classifier is an adapter that can say what class an error of its own is.
|
||||
type Classifier interface {
|
||||
Class(err error) string
|
||||
}
|
||||
|
||||
type appliedEntry struct {
|
||||
hash string
|
||||
derived map[string]any
|
||||
node string
|
||||
}
|
||||
|
||||
type brake struct {
|
||||
times int
|
||||
nextAt time.Time
|
||||
}
|
||||
|
||||
type failure struct {
|
||||
text string
|
||||
times int
|
||||
}
|
||||
|
||||
// The longest a consumer whose create keeps failing to satisfy holds waits between checks.
|
||||
const maxBackoff = time.Hour
|
||||
|
||||
// How many passes a secret may be unreadable before it stops being called a race (issue 225), and
|
||||
// once said loudly, how often it is repeated. The same cadence quiets a create that keeps failing
|
||||
// the same way.
|
||||
const (
|
||||
patiently = 12
|
||||
loudlyEvery = 240
|
||||
)
|
||||
|
||||
type contribution struct {
|
||||
As string `json:"as"`
|
||||
Secret string `json:"secret"`
|
||||
Node string `json:"node"`
|
||||
At string `json:"at"`
|
||||
Values map[string]any `json:"values"`
|
||||
Derived map[string]any `json:"derived"`
|
||||
}
|
||||
|
||||
func (h *Harness) init() {
|
||||
if h.Every == 0 {
|
||||
h.Every = 5 * time.Second
|
||||
}
|
||||
if h.VerifyEvery == 0 {
|
||||
h.VerifyEvery = time.Minute
|
||||
}
|
||||
if h.HoldsTimeout == 0 {
|
||||
h.HoldsTimeout = 30 * time.Second
|
||||
}
|
||||
if h.Now == nil {
|
||||
h.Now = time.Now
|
||||
}
|
||||
if h.FailingAfter == 0 {
|
||||
h.FailingAfter = 5 * time.Minute
|
||||
}
|
||||
if h.SayAgainEvery == 0 {
|
||||
h.SayAgainEvery = 15 * time.Minute
|
||||
}
|
||||
if h.StablePasses == 0 {
|
||||
h.StablePasses = 5
|
||||
}
|
||||
if h.StableFor == 0 {
|
||||
h.StableFor = StableFor
|
||||
}
|
||||
if h.RetireAtOnce == 0 {
|
||||
h.RetireAtOnce = 3
|
||||
}
|
||||
if h.RetireFraction == 0 {
|
||||
h.RetireFraction = 0.5
|
||||
}
|
||||
if h.Log == nil {
|
||||
h.Log = func(format string, args ...any) { fmt.Fprintf(os.Stderr, format+"\n", args...) }
|
||||
}
|
||||
if h.applied == nil {
|
||||
h.applied = map[string]appliedEntry{}
|
||||
h.lost = map[string]brake{}
|
||||
h.waiting = map[string]int{}
|
||||
h.failing = map[string]failure{}
|
||||
h.trouble = map[string]*standing{}
|
||||
h.cleared = map[string]bool{}
|
||||
h.r.retired = map[string]retiredEntry{}
|
||||
}
|
||||
}
|
||||
|
||||
// Run reconciles until ctx ends. One consumer's failure never stops the others'.
|
||||
func (h *Harness) Run(ctx context.Context) {
|
||||
h.init()
|
||||
for {
|
||||
h.Reconcile(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(h.Every):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (h *Harness) say(format string, args ...any) {
|
||||
h.Log("[provisioner:"+h.Resource+"] "+format, args...)
|
||||
}
|
||||
|
||||
func (h *Harness) warn(why string) {
|
||||
if why == h.lastWarning {
|
||||
return
|
||||
}
|
||||
if why != "" {
|
||||
h.say("%s; nothing applied or removed until it can be read", why)
|
||||
} else {
|
||||
h.say("contributions file readable again")
|
||||
}
|
||||
h.lastWarning = why
|
||||
}
|
||||
|
||||
// readContributions answers the consumers asked for, or nil when the file says nothing usable.
|
||||
// **Nothing read is not nobody asking** (novox/hq issue 241): only a file that was read can retire anything.
|
||||
func (h *Harness) readContributions() []contribution {
|
||||
raw, err := os.ReadFile(h.Receives)
|
||||
if err != nil {
|
||||
h.warn(fmt.Sprintf("contributions file unreadable (%s): %v", h.Receives, err))
|
||||
return nil
|
||||
}
|
||||
var doc struct {
|
||||
Requirement string `json:"requirement"`
|
||||
Given json.RawMessage `json:"given"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
h.warn(fmt.Sprintf("contributions file is not JSON (%s): %v", h.Receives, err))
|
||||
return nil
|
||||
}
|
||||
if doc.Requirement != "" && doc.Requirement != h.Resource {
|
||||
h.warn(fmt.Sprintf("%s is for %s, not %s", h.Receives, doc.Requirement, h.Resource))
|
||||
return nil
|
||||
}
|
||||
var given []contribution
|
||||
if len(doc.Given) == 0 || string(doc.Given) == "null" || json.Unmarshal(doc.Given, &given) != nil {
|
||||
h.warn(fmt.Sprintf("%s has no given list", h.Receives))
|
||||
return nil
|
||||
}
|
||||
h.warn("")
|
||||
out := []contribution{}
|
||||
for _, g := range given {
|
||||
// No `as` is not a credential grant: nothing to create for it.
|
||||
if g.As != "" && g.Secret != "" {
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func hashOf(as, password string, values, derived map[string]any) string {
|
||||
// Derived is in the hash: a provider that renames what it derives gave a different resource.
|
||||
b, _ := json.Marshal([]any{as, password, orEmpty(values), orEmpty(derived)})
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func orEmpty(m map[string]any) map[string]any {
|
||||
if m == nil {
|
||||
return map[string]any{}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Reconcile is one pass.
|
||||
func (h *Harness) Reconcile(ctx context.Context) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.init()
|
||||
given := h.readContributions()
|
||||
if given == nil {
|
||||
// Not a result: the passes that must agree before anything is retired start again.
|
||||
h.r.key, h.r.count = "", 0
|
||||
return
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, g := range given {
|
||||
want[g.As] = true
|
||||
}
|
||||
h.seed(ctx, want)
|
||||
h.apply(ctx, given)
|
||||
|
||||
// Retire what the mesh has stably stopped asking for — within the bound, or with a person.
|
||||
h.retireUnasked(ctx, want)
|
||||
h.r.lastWant = want
|
||||
for as := range h.failing {
|
||||
if !want[as] {
|
||||
delete(h.failing, as)
|
||||
}
|
||||
}
|
||||
// A consumer the mesh stopped asking for that this provider holds nothing for is no longer failed by
|
||||
// anyone: said, so a standing the controller keeps for it is cleared rather than left naming a
|
||||
// consumer that is gone. One still held is said recovered when it is retired.
|
||||
for as := range h.trouble {
|
||||
if _, held := h.applied[as]; !want[as] && !held {
|
||||
h.recovered(as, "no longer asked for")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// apply makes each consumer the grant file gives, and checks every VerifyEvery that the backend still holds it. The
|
||||
// caller holds h.mu. One consumer's failure never stops the others'.
|
||||
func (h *Harness) apply(ctx context.Context, given []contribution) {
|
||||
verifying := h.Now().Sub(h.verifiedAt) >= h.VerifyEvery
|
||||
if verifying {
|
||||
h.verifiedAt = h.Now()
|
||||
}
|
||||
|
||||
for _, g := range given {
|
||||
raw, err := os.ReadFile(g.Secret)
|
||||
if err != nil {
|
||||
// A secret the host has not written yet is a race on the first pass; past a minute it is
|
||||
// a person's to look at, and said so (novox/hq issue 225).
|
||||
n := h.waiting[g.As] + 1
|
||||
h.waiting[g.As] = n
|
||||
if n <= patiently {
|
||||
h.say("%s: secret not readable yet (%s): %v", g.As, g.Secret, err)
|
||||
} else if n == patiently+1 || n%loudlyEvery == 0 {
|
||||
h.say("%s: CANNOT READ the secret after %d attempts (%s): %v. This is not a race any more — "+
|
||||
"nothing has been provisioned for this consumer and nothing will be until somebody looks. "+
|
||||
"Check who owns the file and who this process runs as (novox/hq issue 225)", g.As, n, g.Secret, err)
|
||||
}
|
||||
h.failed(g.As, g.Node, ClassSecret, fmt.Sprintf("secret not readable (%s): %v", g.Secret, err))
|
||||
continue
|
||||
}
|
||||
delete(h.waiting, g.As)
|
||||
password := strings.TrimSuffix(string(raw), "\n")
|
||||
p := Provision{As: g.As, Password: password, Values: orEmpty(g.Values), Derived: orEmpty(g.Derived), At: g.At, Consumer: g.Node}
|
||||
hash := hashOf(g.As, password, g.Values, g.Derived)
|
||||
|
||||
reapplying := 0
|
||||
if was, ok := h.applied[g.As]; ok && was.hash == hash {
|
||||
if !verifying {
|
||||
continue
|
||||
}
|
||||
b, braked := h.lost[g.As]
|
||||
if braked && h.Now().Before(b.nextAt) {
|
||||
continue
|
||||
}
|
||||
hctx, cancel := context.WithTimeout(ctx, h.HoldsTimeout)
|
||||
held, err := h.Adapter.Holds(hctx, p)
|
||||
timedOut := errors.Is(hctx.Err(), context.DeadlineExceeded)
|
||||
cancel()
|
||||
if err != nil {
|
||||
// Unable to ask is not evidence of loss. A backend that timed out will time out for
|
||||
// the next consumer too, so the rest of this pass is not asked.
|
||||
text := scrub(err, password)
|
||||
h.say("%s: could not check the backend, will ask again: %s", g.As, text)
|
||||
h.failed(g.As, g.Node, h.classOf(err, text), text)
|
||||
if timedOut {
|
||||
verifying = false
|
||||
}
|
||||
continue
|
||||
}
|
||||
if held {
|
||||
delete(h.lost, g.As)
|
||||
h.succeeded(g.As)
|
||||
continue
|
||||
}
|
||||
reapplying = b.times + 1
|
||||
if reapplying == 1 {
|
||||
h.say("%s: the backend no longer holds it; applying again", g.As)
|
||||
} else {
|
||||
h.say("%s: still not held after being applied again (%d times in a row) — create does not "+
|
||||
"produce what holds checks; applying again", g.As, reapplying)
|
||||
}
|
||||
}
|
||||
if err := h.Adapter.Create(ctx, p); err != nil {
|
||||
text := scrub(err, password)
|
||||
f := h.failing[g.As]
|
||||
if f.text != text {
|
||||
f = failure{text: text}
|
||||
}
|
||||
f.times++
|
||||
h.failing[g.As] = f
|
||||
// Said each time it changes, and while it stays the same, as rarely as a lost secret.
|
||||
if f.times == 1 || f.times%loudlyEvery == 0 {
|
||||
h.say("%s: create failed, will retry: %s", g.As, text)
|
||||
}
|
||||
h.failed(g.As, g.Node, h.classOf(err, text), text)
|
||||
if reapplying > 0 {
|
||||
h.lost[g.As] = brake{times: reapplying - 1}
|
||||
}
|
||||
continue
|
||||
}
|
||||
if f, was := h.failing[g.As]; was {
|
||||
h.say("%s: created, after %d failed attempt(s)", g.As, f.times)
|
||||
delete(h.failing, g.As)
|
||||
}
|
||||
h.succeeded(g.As)
|
||||
h.applied[g.As] = appliedEntry{hash: hash, derived: p.Derived, node: g.Node}
|
||||
h.reenabled(g.As, g.Node)
|
||||
if reapplying == 0 {
|
||||
delete(h.lost, g.As)
|
||||
} else {
|
||||
wait := h.VerifyEvery << (reapplying - 1)
|
||||
if wait > maxBackoff || wait <= 0 {
|
||||
wait = maxBackoff
|
||||
}
|
||||
h.lost[g.As] = brake{times: reapplying, nextAt: h.Now().Add(wait)}
|
||||
if reapplying > 1 {
|
||||
h.say("%s: next check in %s", g.As, wait.Round(time.Second))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// failed counts one more failure in a consumer's unbroken run, and announces the run once it has
|
||||
// lasted FailingAfter — then again every SayAgainEvery while it lasts.
|
||||
func (h *Harness) failed(as, node, class, text string) {
|
||||
now := h.Now()
|
||||
s := h.trouble[as]
|
||||
if s == nil {
|
||||
s = &standing{since: now}
|
||||
h.trouble[as] = s
|
||||
}
|
||||
s.node, s.class, s.text = node, class, text
|
||||
s.attempts++
|
||||
if now.Sub(s.since) < h.FailingAfter {
|
||||
return
|
||||
}
|
||||
if !s.saidAt.IsZero() && now.Sub(s.saidAt) < h.SayAgainEvery {
|
||||
return
|
||||
}
|
||||
first := s.saidAt.IsZero()
|
||||
s.saidAt = now
|
||||
if first {
|
||||
h.say("%s: FAILING for %s (%d attempts, %s): %s. Announced as %s; `status` names it until it "+
|
||||
"succeeds (novox/hq ADR 0224)", as, now.Sub(s.since).Round(time.Second), s.attempts, class, text, EventFailing)
|
||||
}
|
||||
h.announce(EventFailing, map[string]any{
|
||||
"provider": h.Resource, "provider-node": h.Node,
|
||||
"consumer": as, "node": node,
|
||||
"class": class, "error": clip(text),
|
||||
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
|
||||
})
|
||||
}
|
||||
|
||||
// succeeded ends a consumer's run of failures; one that was announced is announced recovered.
|
||||
//
|
||||
// **And the first success for a consumer since this process started is announced too**, failing or
|
||||
// not: a provider that announced a failure and was restarted has forgotten it, and without this the
|
||||
// controller would name the consumer failing for ever after it recovered unheard.
|
||||
func (h *Harness) succeeded(as string) {
|
||||
if h.trouble[as] == nil && !h.cleared[as] {
|
||||
h.cleared[as] = true
|
||||
h.announce(EventRecovered, map[string]any{
|
||||
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "why": "first-success",
|
||||
})
|
||||
return
|
||||
}
|
||||
h.cleared[as] = true
|
||||
h.recovered(as, "")
|
||||
}
|
||||
|
||||
func (h *Harness) recovered(as, why string) {
|
||||
s := h.trouble[as]
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
delete(h.trouble, as)
|
||||
if s.saidAt.IsZero() {
|
||||
return // never announced, so there is nothing to take back
|
||||
}
|
||||
if why == "" {
|
||||
h.say("%s: recovered after %s and %d failed attempt(s)", as, h.Now().Sub(s.since).Round(time.Second), s.attempts)
|
||||
}
|
||||
body := map[string]any{
|
||||
"provider": h.Resource, "provider-node": h.Node, "consumer": as, "node": s.node,
|
||||
"since": s.since.UTC().Format(time.RFC3339), "attempts": s.attempts,
|
||||
}
|
||||
if why != "" {
|
||||
body["why"] = why
|
||||
}
|
||||
h.announce(EventRecovered, body)
|
||||
}
|
||||
|
||||
func (h *Harness) announce(event string, body map[string]any) {
|
||||
if h.Announce != nil {
|
||||
h.Announce(event, body)
|
||||
}
|
||||
}
|
||||
|
||||
// classOf is an error's class: the adapter's word when it has one, else read from the text.
|
||||
func (h *Harness) classOf(err error, text string) string {
|
||||
if c, ok := h.Adapter.(Classifier); ok {
|
||||
if class := c.Class(err); class != "" {
|
||||
return class
|
||||
}
|
||||
}
|
||||
return ClassOf(text)
|
||||
}
|
||||
|
||||
// ClassOf reads an error's class from its text — the words the backends the mesh runs use.
|
||||
func ClassOf(text string) string {
|
||||
t := strings.ToLower(text)
|
||||
for _, w := range []string{"invalid_grant", "invalid user credentials", "password authentication failed",
|
||||
"authentication failed", "unauthorized", " 401"} {
|
||||
if strings.Contains(t, w) {
|
||||
return ClassCredentials
|
||||
}
|
||||
}
|
||||
for _, w := range []string{"connection refused", "no such host", "i/o timeout", "deadline exceeded",
|
||||
"connection reset", "network is unreachable", "no route to host", "eof"} {
|
||||
if strings.Contains(t, w) {
|
||||
return ClassUnreachable
|
||||
}
|
||||
}
|
||||
return ClassRefused
|
||||
}
|
||||
|
||||
// clip keeps an announced error to what belongs in a status line.
|
||||
func clip(text string) string {
|
||||
const most = 300
|
||||
if len(text) <= most {
|
||||
return text
|
||||
}
|
||||
return text[:most] + "…"
|
||||
}
|
||||
|
||||
// scrub is an error's text with the consumer's password removed, raw and URL-encoded.
|
||||
func scrub(err error, password string) string {
|
||||
text := err.Error()
|
||||
if password == "" {
|
||||
return text
|
||||
}
|
||||
for _, form := range []string{password, url.QueryEscape(password), url.PathEscape(password)} {
|
||||
text = strings.ReplaceAll(text, form, "***")
|
||||
}
|
||||
return text
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
package harness
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type world struct {
|
||||
t *testing.T
|
||||
dir string
|
||||
receives string
|
||||
now time.Time
|
||||
h *Harness
|
||||
a *recorder
|
||||
said []string
|
||||
}
|
||||
|
||||
func newWorld(t *testing.T) *world {
|
||||
w := &world{t: t, dir: t.TempDir(), now: time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC), a: &recorder{held: true}}
|
||||
w.receives = filepath.Join(w.dir, "mesh.json")
|
||||
w.h = &Harness{Resource: "postgres-database", Receives: w.receives, Adapter: w.a,
|
||||
Now: func() time.Time { return w.now },
|
||||
Log: func(f string, a ...any) { w.said = append(w.said, f) }}
|
||||
return w
|
||||
}
|
||||
|
||||
func (w *world) give(given ...map[string]any) {
|
||||
for _, g := range given {
|
||||
secret := filepath.Join(w.dir, g["as"].(string)+".secret")
|
||||
if err := os.WriteFile(secret, []byte("pw-"+g["as"].(string)+"\n"), 0o600); err != nil {
|
||||
w.t.Fatal(err)
|
||||
}
|
||||
g["secret"] = secret
|
||||
}
|
||||
if given == nil {
|
||||
given = []map[string]any{}
|
||||
}
|
||||
raw, _ := json.Marshal(map[string]any{"requirement": "postgres-database", "given": given})
|
||||
if err := os.WriteFile(w.receives, raw, 0o600); err != nil {
|
||||
w.t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAConsumerIsCreatedOnceUnderTheMeshsLoginAndPassword(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
w.give(map[string]any{"as": "mesh_ace_letta", "node": "ace", "values": map[string]any{"name": "letta"}})
|
||||
w.h.Reconcile(ctx)
|
||||
w.h.Reconcile(ctx)
|
||||
if len(w.a.created) != 1 {
|
||||
t.Fatalf("created %d times", len(w.a.created))
|
||||
}
|
||||
p := w.a.created[0]
|
||||
if p.As != "mesh_ace_letta" || p.Password != "pw-mesh_ace_letta" || p.Consumer != "ace" {
|
||||
t.Fatalf("%+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAConsumerNoLongerAskedForIsRetiredOnceStable(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
|
||||
w.h.Reconcile(ctx)
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.settle() // five passes and ten minutes
|
||||
if strings.Join(w.a.removed, ",") != "b" {
|
||||
t.Fatal(w.a.removed)
|
||||
}
|
||||
// Only a file that says nobody asks retires the last one.
|
||||
w.give()
|
||||
w.settle()
|
||||
if strings.Join(w.a.removed, ",") != "b,a" {
|
||||
t.Fatal(w.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNothingReadIsNotNobodyAsking(t *testing.T) {
|
||||
for name, content := range map[string]string{
|
||||
"unreadable": "",
|
||||
"not JSON": "{",
|
||||
"no given": `{"requirement": "postgres-database"}`,
|
||||
"another": `{"requirement": "mssql-database", "given": []}`,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.h.Reconcile(ctx)
|
||||
if content == "" {
|
||||
os.Remove(w.receives)
|
||||
} else {
|
||||
os.WriteFile(w.receives, []byte(content), 0o600)
|
||||
}
|
||||
w.h.Reconcile(ctx)
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("retired %v on a file it could not use", w.a.removed)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestALostConsumerIsAppliedAgainAndBraked(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.h.Reconcile(ctx)
|
||||
w.a.held = false
|
||||
w.now = w.now.Add(2 * time.Minute)
|
||||
w.h.Reconcile(ctx)
|
||||
if len(w.a.created) != 2 {
|
||||
t.Fatalf("created %d times", len(w.a.created))
|
||||
}
|
||||
// Still not held a minute later: applied again, then braked for two minutes.
|
||||
w.now = w.now.Add(61 * time.Second)
|
||||
w.h.Reconcile(ctx)
|
||||
w.now = w.now.Add(61 * time.Second)
|
||||
w.h.Reconcile(ctx)
|
||||
if len(w.a.created) != 3 {
|
||||
t.Fatalf("not braked: created %d times", len(w.a.created))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFailingCreateIsRetriedAndSaidOnce(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
w.a.failing = &pgErr{"extension \"nope\" refused, password pw-a"}
|
||||
w.give(map[string]any{"as": "a"})
|
||||
for i := 0; i < 5; i++ {
|
||||
w.h.Reconcile(ctx)
|
||||
}
|
||||
n := 0
|
||||
for _, s := range w.said {
|
||||
if strings.Contains(s, "create failed") {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("said %d times", n)
|
||||
}
|
||||
w.a.failing = nil
|
||||
w.h.Reconcile(ctx)
|
||||
if len(w.a.created) != 1 {
|
||||
t.Fatal("not retried")
|
||||
}
|
||||
}
|
||||
|
||||
type pgErr struct{ s string }
|
||||
|
||||
func (e *pgErr) Error() string { return e.s }
|
||||
|
||||
func TestScrubRemovesThePassword(t *testing.T) {
|
||||
if got := scrub(&pgErr{"bad pw a/b c and a%2Fb+c"}, "a/b c"); strings.Contains(got, "a/b c") || strings.Contains(got, "a%2Fb+c") {
|
||||
t.Fatal(got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,339 @@
|
||||
package harness
|
||||
|
||||
// A provider handed to the provisioner (novox/hq ADR 0373, ADR 0349 clause 12, issues 780 and 805): where its
|
||||
// consumers file is set, it runs on the provider helper (package provider) instead of the harness reading the
|
||||
// controller's grant files. Moved is that path, shared by every provider that keeps a resource per consumer; a module
|
||||
// says only what differs, in a MovedSpec.
|
||||
//
|
||||
// The helper learns who binds the provider from the plans, composes each consumer's entry and reads its credential
|
||||
// through ConsumersFile, the file this machine's provisioner client fills. It leaves the rest to the two hooks
|
||||
// here: Make brings the consumer's resource into being exactly as the harness's adapter does, and Gone leaves
|
||||
// retirement to the harness's guards (retirement.go): the resource is disabled and marked to delete, never removed.
|
||||
//
|
||||
// **A consumer not moved is still the grant file's** (issue 937, ADR 0349 clause 17), for a provider whose spec says
|
||||
// WithGrants: only a moved consumer's plan names its provider, so the controller still gives every other consumer
|
||||
// through the grant file (MESH_RECEIVES) until R1 (issue 682). Each login it gives is asked for, and one no plan
|
||||
// names is made and checked by the harness's grant pass exactly as on the grant path; a login a plan names is the
|
||||
// helper's alone. A grant file that cannot be read is not nobody asking (issue 241): nothing is retired until it is
|
||||
// read again.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/provider"
|
||||
)
|
||||
|
||||
// MovedSpec is what differs per provider on the moved path.
|
||||
type MovedSpec struct {
|
||||
// Module and Provision are the provider module's name and what it provides, as its manifest says.
|
||||
Module, Provision string
|
||||
// Offer is what it serves: its credential form and served values.
|
||||
Offer provider.Offer
|
||||
// Login is the identity a consumer is made under at the provider's end (postgres: the entry's User; keycloak: its
|
||||
// As); an entry with none is not made.
|
||||
Login func(e provider.Entry) string
|
||||
// Values are the values the adapter is handed for a consumer; the entry's contribution when not set.
|
||||
Values func(e provider.Entry) map[string]any
|
||||
// Noun names what Login is in words ("login", "client id"), Made what is made ("login", "client"), and Backend
|
||||
// the service in "no longer holds it" ("the backend", "Keycloak").
|
||||
Noun, Made, Backend string
|
||||
// WithGrants makes the logins the grant file gives, that no plan names, asked for and made by the harness's grant
|
||||
// pass (issue 937).
|
||||
WithGrants bool
|
||||
}
|
||||
|
||||
// Moved is a provider on the provider helper, with the harness's own memory and retirement behind it: what is held,
|
||||
// what is counted unasked, what waits for a person (retirement.go). The harness is never Run here; the helper's Make
|
||||
// and Gone and the loop below drive the same state, so the guards are the harness's, not a copy.
|
||||
type Moved struct {
|
||||
h *Harness
|
||||
spec MovedSpec
|
||||
|
||||
// FirstRetry is the first wait before a watch asked too early is asked again (issue 943); a second when zero.
|
||||
FirstRetry time.Duration
|
||||
|
||||
mu sync.Mutex // guards assignments and loaded
|
||||
assignments map[string][]byte
|
||||
loaded bool
|
||||
verified map[string]time.Time
|
||||
}
|
||||
|
||||
func newMoved(h *Harness, spec MovedSpec) *Moved {
|
||||
h.mu.Lock()
|
||||
h.init()
|
||||
h.mu.Unlock()
|
||||
return &Moved{h: h, spec: spec, assignments: map[string][]byte{}, verified: map[string]time.Time{}}
|
||||
}
|
||||
|
||||
// Make brings one consumer's resource into being under its login and the vault's credential, as the harness does:
|
||||
// at once when it is new or changed, and otherwise only when the backend no longer holds it, which is asked every
|
||||
// VerifyEvery.
|
||||
func (m *Moved) Make(ctx context.Context, c provider.Consumer) error {
|
||||
h, e := m.h, c.Entry
|
||||
as := m.spec.Login(e)
|
||||
if as == "" {
|
||||
return fmt.Errorf("%s has no %s to make", e.Key(), m.spec.Noun)
|
||||
}
|
||||
if len(c.Credential) == 0 {
|
||||
return fmt.Errorf("%s has no credential to make its %s with", e.Key(), m.spec.Made)
|
||||
}
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
password := string(c.Credential)
|
||||
values := orEmpty(e.Contributes)
|
||||
if m.spec.Values != nil {
|
||||
values = m.spec.Values(e)
|
||||
}
|
||||
p := Provision{As: as, Password: password, Values: values, Consumer: e.Consumer}
|
||||
hash := hashOf(as, password, p.Values, nil)
|
||||
if was, ok := h.applied[as]; ok && was.hash == hash {
|
||||
if h.Now().Sub(m.verified[as]) < h.VerifyEvery {
|
||||
return nil
|
||||
}
|
||||
m.verified[as] = h.Now()
|
||||
hctx, cancel := context.WithTimeout(ctx, h.HoldsTimeout)
|
||||
held, err := h.Adapter.Holds(hctx, p)
|
||||
cancel()
|
||||
if err != nil {
|
||||
text := scrub(err, password)
|
||||
h.failed(as, e.Consumer, h.classOf(err, text), text)
|
||||
return errors.New(text)
|
||||
}
|
||||
if held {
|
||||
h.succeeded(as)
|
||||
return nil
|
||||
}
|
||||
h.say("%s: %s no longer holds it; applying again", as, m.spec.Backend)
|
||||
}
|
||||
if err := h.Adapter.Create(ctx, p); err != nil {
|
||||
text := scrub(err, password)
|
||||
h.failed(as, e.Consumer, h.classOf(err, text), text)
|
||||
return errors.New(text)
|
||||
}
|
||||
h.succeeded(as)
|
||||
h.applied[as] = appliedEntry{hash: hash, node: e.Consumer}
|
||||
m.verified[as] = h.Now()
|
||||
h.reenabled(as, e.Consumer)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Gone is the helper asking whether it may forget a consumer that no longer binds. **It never retires**: retiring is
|
||||
// the harness's rule, the same unasked set over StablePasses passes and StableFor, at most RetireAtOnce at a time
|
||||
// (retirement.go, ADR 0230), run by retire below. So a consumer missing from one read of the plans loses nothing:
|
||||
// the helper is told no, and asks again at its next pass, until the login is retired, or asked for again.
|
||||
func (m *Moved) Gone(_ context.Context, e provider.Entry) error {
|
||||
h := m.h
|
||||
as := m.spec.Login(e)
|
||||
if as == "" {
|
||||
return nil // refused: nothing was made for it
|
||||
}
|
||||
// The plans and the grant file as they are now, not as the last retirement pass read them: with either not
|
||||
// read, nothing is let go.
|
||||
want, read := m.asked(h.Node)
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
asked := want[as]
|
||||
if m.spec.WithGrants {
|
||||
_, granted, ok := m.granted(want)
|
||||
read = read && ok
|
||||
asked = asked || granted[as]
|
||||
}
|
||||
_, held := h.applied[as]
|
||||
_, retired := h.r.retired[as]
|
||||
switch {
|
||||
case !read:
|
||||
case asked:
|
||||
return nil // another holder of the same login still binds: nothing to retire
|
||||
case retired, !held && h.r.seeded:
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("%s is kept until its retirement is stable (%d passes and %s) and within the bound, or a "+
|
||||
"person approves it", as, h.StablePasses, h.StableFor)
|
||||
}
|
||||
|
||||
// plan is as much of an assignment as is read to learn who binds the provider.
|
||||
type plan struct {
|
||||
Node string `json:"node"`
|
||||
Module string `json:"module"`
|
||||
Provisioning *struct {
|
||||
Provisions []struct {
|
||||
Provision string `json:"provision"`
|
||||
Provider string `json:"provider"`
|
||||
ProviderModule string `json:"provider-module"`
|
||||
// As is who the consumer is at its provider's end, its slug first (issue 788).
|
||||
As string `json:"as"`
|
||||
} `json:"provisions"`
|
||||
} `json:"provisioning"`
|
||||
}
|
||||
|
||||
// asked is the logins the plans bind the provider for, and whether they could all be read: a plan that is not the
|
||||
// controller's, or no read yet, is not an answer, so nothing is counted unasked from it (novox/hq issue 241).
|
||||
func (m *Moved) asked(node string) (map[string]bool, bool) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if !m.loaded || len(m.assignments) == 0 {
|
||||
return nil, false
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for key, raw := range m.assignments {
|
||||
var a plan
|
||||
if err := json.Unmarshal(raw, &a); err != nil || a.Node+"."+a.Module != key {
|
||||
return nil, false
|
||||
}
|
||||
if a.Provisioning == nil {
|
||||
continue
|
||||
}
|
||||
for _, b := range a.Provisioning.Provisions {
|
||||
if b.Provider == node && b.ProviderModule == m.spec.Module && b.Provision == m.spec.Provision {
|
||||
as := b.As
|
||||
if as == "" {
|
||||
as = provider.ConsumerIdentity(a.Node, a.Module) // a plan that names none
|
||||
}
|
||||
want[as] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return want, true
|
||||
}
|
||||
|
||||
// granted is what the grant file gives that no plan names (issue 937), and every login it gives; and whether it
|
||||
// could be read. The caller holds h.mu. With no grant file named there is nothing given, and that is a read.
|
||||
func (m *Moved) granted(planned map[string]bool) ([]contribution, map[string]bool, bool) {
|
||||
h := m.h
|
||||
if h.Receives == "" {
|
||||
return nil, map[string]bool{}, true
|
||||
}
|
||||
given := h.readContributions()
|
||||
if given == nil {
|
||||
return nil, nil, false
|
||||
}
|
||||
var own []contribution
|
||||
logins := map[string]bool{}
|
||||
for _, g := range given {
|
||||
logins[g.As] = true
|
||||
if !planned[g.As] {
|
||||
own = append(own, g)
|
||||
}
|
||||
}
|
||||
return own, logins, true
|
||||
}
|
||||
|
||||
// retire is one pass of the harness's retirement over what the plans bind and the grant file gives now; the logins
|
||||
// only the grant file gives are made and checked first, as the grant path does.
|
||||
func (m *Moved) retire(ctx context.Context) {
|
||||
h := m.h
|
||||
want, ok := m.asked(h.Node)
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
var given []contribution
|
||||
if ok && m.spec.WithGrants {
|
||||
var granted map[string]bool
|
||||
given, granted, ok = m.granted(want)
|
||||
for as := range granted {
|
||||
want[as] = true
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
// Not a result: the passes that must agree before anything is retired start again.
|
||||
h.r.key, h.r.count = "", 0
|
||||
return
|
||||
}
|
||||
h.seed(ctx, want)
|
||||
h.apply(ctx, given)
|
||||
h.retireUnasked(ctx, want)
|
||||
h.r.lastWant = want
|
||||
for as := range h.failing {
|
||||
if !want[as] {
|
||||
delete(h.failing, as)
|
||||
}
|
||||
}
|
||||
for as := range h.trouble {
|
||||
if _, held := h.applied[as]; !want[as] && !held {
|
||||
h.recovered(as, "no longer asked for")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// watch keeps the plans as the bus holds them, from the first read on.
|
||||
func (m *Moved) watch(ctx context.Context, bus provider.Bus) error {
|
||||
err := bus.Watch(ctx, provider.AssignmentsBucket, func(key string, value []byte) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if value == nil {
|
||||
delete(m.assignments, key)
|
||||
} else {
|
||||
m.assignments[key] = append([]byte(nil), value...)
|
||||
}
|
||||
})
|
||||
if err == nil {
|
||||
m.mu.Lock()
|
||||
m.loaded = true
|
||||
m.mu.Unlock()
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// Run answers the consumers through the helper and retires through the harness's rules, until ctx ends.
|
||||
//
|
||||
// The bus is reached through the runtime that launched this bundle, which answers only once Serve runs, and Run starts
|
||||
// beside Serve: a watch asked too early is asked again, after a wait that doubles from FirstRetry up to a minute,
|
||||
// until it is answered or ctx ends, so a start that loses the race never leaves the consumers unanswered (issue 943).
|
||||
func (m *Moved) Run(ctx context.Context, p *provider.Provider, bus provider.Bus) error {
|
||||
wait := m.FirstRetry
|
||||
if wait <= 0 {
|
||||
wait = time.Second
|
||||
}
|
||||
for tries := 1; ; tries++ {
|
||||
err := m.watch(ctx, bus)
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if tries == 4 || tries%20 == 0 {
|
||||
m.h.say("the plans cannot be watched yet (%d tries), asking again: %v", tries, err)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(wait):
|
||||
}
|
||||
wait = min(wait*2, time.Minute)
|
||||
}
|
||||
go func() {
|
||||
for {
|
||||
m.retire(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(m.h.Every):
|
||||
}
|
||||
}
|
||||
}()
|
||||
return p.Run(ctx)
|
||||
}
|
||||
|
||||
// NewMoved is a provider handed to the provisioner: the helper answering its consumers, and the harness (never run)
|
||||
// whose retirement and tools it shares. consumersFile is where its machine's client fills its consumers. Run the
|
||||
// result with Moved.Run, passing the provider and the bus returned with it.
|
||||
func NewMoved(consumersFile string, h *Harness, spec MovedSpec) (*Moved, *provider.Provider, provider.Bus, error) {
|
||||
if consumersFile == "" {
|
||||
return nil, nil, nil, errors.New("no consumers' file")
|
||||
}
|
||||
run := newMoved(h, spec)
|
||||
bus := provider.StateBus(spec.Module)
|
||||
return run, provider.New(provider.Options{
|
||||
Node: h.Node,
|
||||
Module: spec.Module,
|
||||
Offers: map[string]provider.Offer{spec.Provision: spec.Offer},
|
||||
Bus: bus,
|
||||
Vault: provider.ConsumersFile{Paths: []string{consumersFile}},
|
||||
Make: run.Make,
|
||||
Gone: run.Gone,
|
||||
Every: h.Every,
|
||||
Logf: func(f string, a ...any) { h.say(f, a...) },
|
||||
}), bus, nil
|
||||
}
|
||||
@@ -0,0 +1,480 @@
|
||||
package harness
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.novox.be/novox/mesh-sdk/go/provider"
|
||||
)
|
||||
|
||||
const (
|
||||
ProvisionName = "postgres-database"
|
||||
servedPort = 5432
|
||||
)
|
||||
|
||||
// postgresSpec is what a postgres-like provider says: a login per consumer, and the grant file's logins.
|
||||
var postgresSpec = MovedSpec{Module: "postgres", Provision: ProvisionName,
|
||||
Offer: provider.Offer{Credential: provider.PerConsumer, Serves: map[string]any{"port": servedPort}},
|
||||
Login: func(e provider.Entry) string { return e.User },
|
||||
Noun: "login", Made: "login", Backend: "the backend", WithGrants: true}
|
||||
|
||||
// bucketBus is the helper's bus over in-memory buckets.
|
||||
type bucketBus struct {
|
||||
mu sync.Mutex
|
||||
buckets map[string]map[string][]byte
|
||||
}
|
||||
|
||||
func (b *bucketBus) Watch(_ context.Context, bucket string, handle func(string, []byte)) error {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
for k, v := range b.buckets[bucket] {
|
||||
handle(k, v)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *bucketBus) Put(_ context.Context, bucket, key string, value []byte) error {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if b.buckets[bucket] == nil {
|
||||
b.buckets[bucket] = map[string][]byte{}
|
||||
}
|
||||
b.buckets[bucket][key] = append([]byte(nil), value...)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *bucketBus) Delete(_ context.Context, bucket, key string) error {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
delete(b.buckets[bucket], key)
|
||||
return nil
|
||||
}
|
||||
|
||||
// bind is one consumer's assignment, whose plan names this machine's postgres as the provider and carries what the
|
||||
// consumer contributes (issue 776).
|
||||
func (b *bucketBus) bind(node, module string, contributes map[string]any) {
|
||||
b.bindNamed("postgres", node, module, contributes, nil)
|
||||
}
|
||||
|
||||
// bindNamed is bind to a provider module, with the names the controller composed for the consumer (issue 792).
|
||||
func (b *bucketBus) bindNamed(provides, node, module string, contributes map[string]any, names map[string]string) {
|
||||
plan, _ := json.Marshal(map[string]any{"node": node, "module": module, "provisioning": map[string]any{
|
||||
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox", "provider-module": provides,
|
||||
"contributes": contributes, "names": names}}}})
|
||||
if err := b.Put(context.Background(), provider.AssignmentsBucket, node+"."+module, plan); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
|
||||
// consumersFile writes the file the machine's client fills: one credential per consumer.
|
||||
func consumersFile(t *testing.T, bound map[[2]string]string) string {
|
||||
var list []provider.ConsumerCredential
|
||||
for k, password := range bound {
|
||||
list = append(list, provider.ConsumerCredential{Consumer: k[0], Module: k[1],
|
||||
Credential: provider.PairName("novox", ProvisionName, k[0], k[1], ""), Value: password})
|
||||
}
|
||||
raw, _ := json.Marshal(list)
|
||||
at := filepath.Join(t.TempDir(), "consumers.json")
|
||||
if err := os.WriteFile(at, raw, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return at
|
||||
}
|
||||
|
||||
// movedWorld is postgres on the helper over an in-memory bus, with a recorder for a server and a clock to move.
|
||||
type movedWorld struct {
|
||||
t *testing.T
|
||||
a *recorder
|
||||
bus *bucketBus
|
||||
h *Harness
|
||||
spec MovedSpec
|
||||
run *Moved
|
||||
p *provider.Provider
|
||||
now time.Time
|
||||
file string
|
||||
}
|
||||
|
||||
func newMovedWorld(t *testing.T, bound map[[2]string]string) *movedWorld {
|
||||
return newMovedWorldOf(t, bound, postgresSpec)
|
||||
}
|
||||
|
||||
func newMovedWorldOf(t *testing.T, bound map[[2]string]string, spec MovedSpec) *movedWorld {
|
||||
w := &movedWorld{t: t, spec: spec, a: &recorder{}, bus: &bucketBus{buckets: map[string]map[string][]byte{}},
|
||||
now: time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)}
|
||||
w.file = consumersFile(t, bound)
|
||||
w.h = &Harness{Resource: ProvisionName, Adapter: w.a, Node: "novox", Now: func() time.Time { return w.now },
|
||||
Log: func(string, ...any) {}}
|
||||
var bus provider.Bus
|
||||
w.run, w.p, bus, _ = NewMoved(w.file, w.h, spec)
|
||||
_ = bus
|
||||
if err := w.run.watch(ctx, w.bus); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// pass is one helper pass and one retirement pass, five seconds after the last.
|
||||
func (w *movedWorld) pass() []provider.Outcome {
|
||||
w.now = w.now.Add(5 * time.Second)
|
||||
// the plans as the bus holds them now, as the watch would hand them
|
||||
w.run.assignments = map[string][]byte{}
|
||||
for k, v := range w.bus.buckets[provider.AssignmentsBucket] {
|
||||
w.run.assignments[k] = v
|
||||
}
|
||||
// the helper reads its buckets from the in-memory bus
|
||||
w.p = provider.New(provider.Options{Node: "novox", Module: w.spec.Module, Bus: w.bus,
|
||||
Vault: provider.ConsumersFile{Paths: []string{w.file}},
|
||||
Offers: map[string]provider.Offer{ProvisionName: w.spec.Offer},
|
||||
Make: w.run.Make, Gone: w.run.Gone, Logf: func(string, ...any) {}})
|
||||
out, err := w.p.Pass(ctx)
|
||||
if err != nil {
|
||||
w.t.Fatal(err)
|
||||
}
|
||||
w.run.retire(ctx)
|
||||
return out
|
||||
}
|
||||
|
||||
func bindings(n int) (map[[2]string]string, []string) {
|
||||
bound := map[[2]string]string{}
|
||||
var logins []string
|
||||
for i := 1; i <= n; i++ {
|
||||
module := fmt.Sprintf("app%d", i)
|
||||
bound[[2]string{"ace", module}] = "pw-" + module
|
||||
logins = append(logins, provider.ConsumerIdentity("ace", module))
|
||||
}
|
||||
return bound, logins
|
||||
}
|
||||
|
||||
// The consumers' file makes the same calls on the adapter as the equivalent grant files do (the harness).
|
||||
func TestTheConsumersFileMakesWhatTheGrantFilesMake(t *testing.T) {
|
||||
letta := provider.ConsumerIdentity("ace", "letta")
|
||||
plain := provider.ConsumerIdentity("ace", "plain")
|
||||
|
||||
// Today: the grant files, through the harness.
|
||||
g := newWorld(t)
|
||||
g.give(
|
||||
map[string]any{"as": letta, "node": "ace", "values": map[string]any{"name": "letta", "extensions": []any{"vector"}}},
|
||||
map[string]any{"as": plain, "node": "ace", "values": map[string]any{"name": "plain"}},
|
||||
)
|
||||
g.h.Reconcile(ctx)
|
||||
|
||||
// The consumers' file, through the provider helper.
|
||||
w := newMovedWorld(t, map[[2]string]string{{"ace", "letta"}: "pw-" + letta, {"ace", "plain"}: "pw-" + plain})
|
||||
w.bus.bind("ace", "letta", map[string]any{"name": "letta", "extensions": []any{"vector"}})
|
||||
w.bus.bind("ace", "plain", map[string]any{"name": "plain"})
|
||||
for _, o := range w.pass() {
|
||||
if o.Status != provider.Answered {
|
||||
t.Fatalf("%+v", o)
|
||||
}
|
||||
}
|
||||
byAs := map[string]Provision{}
|
||||
for _, c := range w.a.created {
|
||||
byAs[c.As] = c
|
||||
}
|
||||
if len(w.a.created) != 2 {
|
||||
t.Fatalf("created %d, want 2", len(w.a.created))
|
||||
}
|
||||
for _, want := range g.a.created {
|
||||
c := byAs[want.As]
|
||||
if c.As != want.As || c.Password != want.Password || c.Consumer != want.Consumer ||
|
||||
!reflect.DeepEqual(orEmpty(c.Values), orEmpty(want.Values)) {
|
||||
t.Fatalf("made %+v, the grant files make %+v", c, want)
|
||||
}
|
||||
}
|
||||
// Unchanged, and held: nothing is made again, within a minute or after it.
|
||||
w.a.held = true
|
||||
w.pass()
|
||||
w.now = w.now.Add(2 * time.Minute)
|
||||
w.pass()
|
||||
if len(w.a.created) != 2 {
|
||||
t.Fatalf("an unchanged pass created again: %d", len(w.a.created))
|
||||
}
|
||||
// No longer held: made again.
|
||||
w.a.held = false
|
||||
w.now = w.now.Add(2 * time.Minute)
|
||||
w.pass()
|
||||
if len(w.a.created) != 4 {
|
||||
t.Fatalf("not made again: %d", len(w.a.created))
|
||||
}
|
||||
}
|
||||
|
||||
// One read without a consumer retires nothing: it takes the harness's stable passes and time, and then the entry
|
||||
// goes and the login is retired, never dropped.
|
||||
func TestAConsumerMissingFromOneReadRetiresNothing(t *testing.T) {
|
||||
bound, logins := bindings(2)
|
||||
w := newMovedWorld(t, bound)
|
||||
w.bus.bind("ace", "app1", nil)
|
||||
w.bus.bind("ace", "app2", nil)
|
||||
w.pass()
|
||||
|
||||
// One read without app2.
|
||||
saved := w.bus.buckets[provider.AssignmentsBucket]["ace.app2"]
|
||||
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app2")
|
||||
w.pass()
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("retired %v on one read", w.a.removed)
|
||||
}
|
||||
if _, kept := w.bus.buckets[provider.BucketOf("postgres")]["ace.app2.postgres-database"]; !kept {
|
||||
t.Fatal("the entry went on one read")
|
||||
}
|
||||
// It is back: nothing was lost, and the entry never left the bucket.
|
||||
w.bus.Put(ctx, provider.AssignmentsBucket, "ace.app2", saved)
|
||||
for i := 0; i < 8; i++ {
|
||||
w.pass()
|
||||
}
|
||||
w.now = w.now.Add(15 * time.Minute)
|
||||
w.pass()
|
||||
if len(w.a.removed) != 0 || len(w.bus.buckets[provider.BucketOf("postgres")]) != 2 {
|
||||
t.Fatalf("retired %v, entries %d", w.a.removed, len(w.bus.buckets[provider.BucketOf("postgres")]))
|
||||
}
|
||||
|
||||
// Gone for good: not retired before StablePasses and StableFor, then retired and its entry removed.
|
||||
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app2")
|
||||
for i := 0; i < 6; i++ {
|
||||
w.pass()
|
||||
}
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("retired %v before the time was out", w.a.removed)
|
||||
}
|
||||
w.now = w.now.Add(11 * time.Minute)
|
||||
w.pass()
|
||||
w.pass()
|
||||
if !reflect.DeepEqual(w.a.removed, []string{logins[1]}) || len(w.a.deleted) != 0 {
|
||||
t.Fatalf("retired %v, deleted %v", w.a.removed, w.a.deleted)
|
||||
}
|
||||
if _, kept := w.bus.buckets[provider.BucketOf("postgres")]["ace.app2.postgres-database"]; kept {
|
||||
t.Fatal("the entry stayed after the login was retired")
|
||||
}
|
||||
}
|
||||
|
||||
// More than three at once waits for a person, as the harness's bound says.
|
||||
func TestMoreThanTheBoundWaitsForAPerson(t *testing.T) {
|
||||
bound, _ := bindings(5)
|
||||
w := newMovedWorld(t, bound)
|
||||
for i := 1; i <= 5; i++ {
|
||||
w.bus.bind("ace", fmt.Sprintf("app%d", i), nil)
|
||||
}
|
||||
w.pass()
|
||||
for i := 1; i <= 4; i++ {
|
||||
w.bus.Delete(ctx, provider.AssignmentsBucket, fmt.Sprintf("ace.app%d", i))
|
||||
}
|
||||
for i := 0; i < 6; i++ {
|
||||
w.pass()
|
||||
}
|
||||
w.now = w.now.Add(11 * time.Minute)
|
||||
w.pass()
|
||||
w.pass()
|
||||
if len(w.a.removed) != 0 || w.h.r.waiting == nil {
|
||||
t.Fatalf("retired %v, waiting %v", w.a.removed, w.h.r.waiting)
|
||||
}
|
||||
}
|
||||
|
||||
// A plan list that was never read is not nobody asking.
|
||||
func TestNoPlansReadRetiresNothing(t *testing.T) {
|
||||
bound, _ := bindings(1)
|
||||
w := newMovedWorld(t, bound)
|
||||
w.bus.bind("ace", "app1", nil)
|
||||
w.pass()
|
||||
w.bus.Delete(ctx, provider.AssignmentsBucket, "ace.app1")
|
||||
for i := 0; i < 8; i++ {
|
||||
w.pass()
|
||||
}
|
||||
w.now = w.now.Add(20 * time.Minute)
|
||||
w.pass()
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("retired %v with no plans read", w.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer whose credential is not in the file yet is waited for: nothing is made for it.
|
||||
func TestAConsumerWithoutACredentialIsWaitedFor(t *testing.T) {
|
||||
w := newMovedWorld(t, map[[2]string]string{})
|
||||
w.bus.bind("ace", "letta", map[string]any{"name": "letta"})
|
||||
outcomes := w.pass()
|
||||
if len(outcomes) != 1 || outcomes[0].Status != provider.Waiting || len(w.a.created) != 0 {
|
||||
t.Fatalf("%+v, created %v", outcomes, w.a.created)
|
||||
}
|
||||
}
|
||||
|
||||
// Where the file is not there, postgres takes the grant directory, as before; where it is, the helper.
|
||||
// The grant path restarts itself when the consumers file is first filled.
|
||||
// A consumer whose module has a slug keeps the login its plan names, the slug first (issue 788): the adapter makes it
|
||||
// under that login, and the retirement counts it as asked for, so a pass retires nothing.
|
||||
func TestASluggedConsumerKeepsTheLoginItsPlanNames(t *testing.T) {
|
||||
agent := provider.ConsumerIdentity("ace", "agent")
|
||||
w := newMovedWorld(t, map[[2]string]string{{"ace", "claude-code"}: "pw-agent"})
|
||||
plan, _ := json.Marshal(map[string]any{"node": "ace", "module": "claude-code", "provisioning": map[string]any{
|
||||
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
|
||||
"provider-module": "postgres", "as": agent}}}})
|
||||
if err := w.bus.Put(ctx, provider.AssignmentsBucket, "ace.claude-code", plan); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, o := range w.pass() {
|
||||
if o.Status != provider.Answered {
|
||||
t.Fatalf("%+v", o)
|
||||
}
|
||||
}
|
||||
if len(w.a.created) != 1 || w.a.created[0].As != agent {
|
||||
t.Fatalf("made %+v, want one under %s", w.a.created, agent)
|
||||
}
|
||||
for i := 0; i < 8; i++ {
|
||||
w.pass()
|
||||
}
|
||||
w.now = w.now.Add(15 * time.Minute)
|
||||
w.pass()
|
||||
w.pass()
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("retired %v: the plan's login was counted unasked", w.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
// A provider whose spec names the plan's `as` as the login and no grant file (the keycloak shape) makes the consumer
|
||||
// under that name with the values its spec composes, and a grant file that cannot be read does not hold up its
|
||||
// retirement pass.
|
||||
func TestASpecWithoutGrantsMakesUnderThePlansAsAndIgnoresTheGrantFile(t *testing.T) {
|
||||
a := &recorder{}
|
||||
bus := &bucketBus{buckets: map[string]map[string][]byte{}}
|
||||
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
|
||||
h := &Harness{Resource: ProvisionName, Adapter: a, Node: "novox", Now: func() time.Time { return now },
|
||||
Receives: filepath.Join(t.TempDir(), "unreadable-or-missing.json"), Log: func(string, ...any) {}}
|
||||
spec := MovedSpec{Module: "keycloak", Provision: ProvisionName,
|
||||
Offer: provider.Offer{Credential: provider.PerConsumer},
|
||||
Login: func(e provider.Entry) string { return e.As }, Noun: "client id", Made: "client", Backend: "Keycloak",
|
||||
Values: func(e provider.Entry) map[string]any { return map[string]any{"label": e.Contributes["label"]} }}
|
||||
file := consumersFile(t, map[[2]string]string{{"ace", "wiki"}: "pw-wiki"})
|
||||
run, _, _, err := NewMoved(file, h, spec)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
plan, _ := json.Marshal(map[string]any{"node": "ace", "module": "wiki", "provisioning": map[string]any{
|
||||
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox",
|
||||
"provider-module": "keycloak", "as": "wiki-client", "contributes": map[string]any{"label": "Wiki"}}}}})
|
||||
_ = bus.Put(ctx, provider.AssignmentsBucket, "ace.wiki", plan)
|
||||
if err := run.watch(ctx, bus); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := provider.New(provider.Options{Node: "novox", Module: "keycloak", Bus: bus,
|
||||
Vault: provider.ConsumersFile{Paths: []string{file}},
|
||||
Offers: map[string]provider.Offer{ProvisionName: spec.Offer}, Make: run.Make, Gone: run.Gone,
|
||||
Logf: func(string, ...any) {}})
|
||||
out, err := p.Pass(ctx)
|
||||
if err != nil || len(out) != 1 || out[0].Status != provider.Answered {
|
||||
t.Fatalf("%v %+v", err, out)
|
||||
}
|
||||
run.retire(ctx)
|
||||
if len(a.created) != 1 || a.created[0].As != "wiki-client" || a.created[0].Password != "pw-wiki" ||
|
||||
a.created[0].Values["label"] != "Wiki" {
|
||||
t.Fatalf("made %+v", a.created)
|
||||
}
|
||||
}
|
||||
|
||||
const testIssuer = "https://id.example.org/realms/Novox"
|
||||
|
||||
var (
|
||||
grafanaContributes = map[string]any{"label": "grafana", "endpoint": "web", "callback": "/login/generic_oauth"}
|
||||
grafanaNames = map[string]string{"name": "grafana.example.org", "internal-name": "grafana.home.internal"}
|
||||
)
|
||||
|
||||
// keycloakSpec is what a keycloak-like provider says: the client id is the plan's `as`, the values are what the
|
||||
// consumer contributes with the controller's names over them, and no grant file is consulted.
|
||||
var keycloakSpec = MovedSpec{Module: "keycloak", Provision: ProvisionName,
|
||||
Offer: provider.Offer{Credential: provider.PerConsumer, Serves: map[string]any{"issuer": testIssuer,
|
||||
"token-path": "/protocol/openid-connect/token"}},
|
||||
Login: func(e provider.Entry) string { return e.As },
|
||||
Values: func(e provider.Entry) map[string]any {
|
||||
values := map[string]any{}
|
||||
for k, v := range e.Contributes {
|
||||
values[k] = v
|
||||
}
|
||||
for k, name := range e.Names {
|
||||
values[k] = name
|
||||
}
|
||||
return values
|
||||
},
|
||||
Noun: "client id", Made: "client", Backend: "Keycloak"}
|
||||
|
||||
// The consumers' file makes the same call on the adapter as the equivalent grant file does (the harness), for a
|
||||
// provider of the keycloak shape: the same client id, secret and values; and the entry tells the consumer what the
|
||||
// provider serves.
|
||||
func TestTheConsumersFileMakesWhatTheGrantFileMakesForTheKeycloakShape(t *testing.T) {
|
||||
grafana := provider.ConsumerIdentity("home", "grafana")
|
||||
values := map[string]any{"name": grafanaNames["name"], "internal-name": grafanaNames["internal-name"]}
|
||||
for k, v := range grafanaContributes {
|
||||
values[k] = v
|
||||
}
|
||||
g := newWorld(t)
|
||||
g.give(map[string]any{"as": grafana, "node": "home", "values": values})
|
||||
g.h.Reconcile(ctx)
|
||||
|
||||
w := newMovedWorldOf(t, map[[2]string]string{{"home", "grafana"}: "pw-" + grafana}, keycloakSpec)
|
||||
w.bus.bindNamed("keycloak", "home", "grafana", grafanaContributes, grafanaNames)
|
||||
for _, o := range w.pass() {
|
||||
if o.Status != provider.Answered {
|
||||
t.Fatalf("%+v", o)
|
||||
}
|
||||
}
|
||||
if len(w.a.created) != 1 || len(g.a.created) != 1 {
|
||||
t.Fatalf("created %d through the helper, %d through the harness, want 1 each", len(w.a.created), len(g.a.created))
|
||||
}
|
||||
c, want := w.a.created[0], g.a.created[0]
|
||||
if c.As != want.As || c.Password != want.Password || c.Consumer != want.Consumer ||
|
||||
!reflect.DeepEqual(orEmpty(c.Values), orEmpty(want.Values)) {
|
||||
t.Fatalf("made %+v, the grant file makes %+v", c, want)
|
||||
}
|
||||
var entry provider.Entry
|
||||
if err := json.Unmarshal(w.bus.buckets[provider.BucketOf("keycloak")]["home.grafana."+ProvisionName], &entry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if entry.Values["issuer"] != testIssuer || entry.Values["token-path"] != "/protocol/openid-connect/token" {
|
||||
t.Fatalf("served %v", entry.Values)
|
||||
}
|
||||
w.a.held = true
|
||||
w.pass()
|
||||
w.now = w.now.Add(2 * time.Minute)
|
||||
w.pass()
|
||||
if len(w.a.created) != 1 {
|
||||
t.Fatalf("an unchanged pass created again: %d", len(w.a.created))
|
||||
}
|
||||
w.a.held = false
|
||||
w.now = w.now.Add(2 * time.Minute)
|
||||
w.pass()
|
||||
if len(w.a.created) != 2 {
|
||||
t.Fatalf("not made again: %d", len(w.a.created))
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer whose plan names its client id (a slugged module) keeps it, and the retirement counts it as asked for.
|
||||
func TestAConsumerNamedByItsPlanKeepsItsClientId(t *testing.T) {
|
||||
agent := provider.ConsumerIdentity("home", "dashboard")
|
||||
w := newMovedWorldOf(t, map[[2]string]string{{"home", "grafana-2"}: "pw"}, keycloakSpec)
|
||||
plan, _ := json.Marshal(map[string]any{"node": "home", "module": "grafana-2", "provisioning": map[string]any{
|
||||
"provisions": []any{map[string]any{"provision": ProvisionName, "provider": "novox", "provider-module": "keycloak",
|
||||
"as": agent, "contributes": grafanaContributes, "names": grafanaNames}}}})
|
||||
if err := w.bus.Put(ctx, provider.AssignmentsBucket, "home.grafana-2", plan); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, o := range w.pass() {
|
||||
if o.Status != provider.Answered {
|
||||
t.Fatalf("%+v", o)
|
||||
}
|
||||
}
|
||||
if len(w.a.created) != 1 || w.a.created[0].As != agent {
|
||||
t.Fatalf("made %+v, want one under %s", w.a.created, agent)
|
||||
}
|
||||
for i := 0; i < 8; i++ {
|
||||
w.pass()
|
||||
}
|
||||
w.now = w.now.Add(15 * time.Minute)
|
||||
w.pass()
|
||||
w.pass()
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("retired %v: the plan's client id was counted unasked", w.a.removed)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,623 @@
|
||||
package harness
|
||||
|
||||
// What becomes of a consumer the mesh no longer asks for (novox/hq ADR 0230 — the operator's model,
|
||||
// decided 2026-10-06; it replaces ADR 0229's withdrawal brake, which let one consumer go every hour).
|
||||
//
|
||||
// A consumer — a login, a client, a key, and the data behind it — is in one of three states:
|
||||
//
|
||||
// 1. ACTIVE.
|
||||
// 2. RETIRED. The mesh stopped asking for it: its access is disabled, reversibly, and its login and
|
||||
// data are marked "to delete" with when and why. Nothing is deleted. Asked for again, the ordinary
|
||||
// create re-enables it at once, as it was.
|
||||
// 3. DELETED, only through `cleanup delete`, a person's act, which this provider executes because it
|
||||
// owns its backend.
|
||||
//
|
||||
// **Stable removals.** A consumer is retired only once the same set of consumers has gone unasked BOTH
|
||||
// in StablePasses (5) consecutive passes that read the file AND for at least StableFor (10 minutes)
|
||||
// since the first pass that saw it. Five passes alone are twenty-five seconds — shorter than a
|
||||
// controller restart, a store reconnecting or a file half written. A pass that could not read the file
|
||||
// is not a result and starts both again; so does a different set. Additions and changes are never
|
||||
// delayed.
|
||||
//
|
||||
// **Too many is a person.** A stable set of more than RetireAtOnce (3), or — where more than one is
|
||||
// held — of more than RetireFraction (half) of those held, retires nothing: it WAITS, said in the
|
||||
// journal and announced `provisioner.retirement` `waiting` (again every SayAgainEvery), which the
|
||||
// controller raises as an urgent condition, until `retire approve <node> <module>` or `retire reject`.
|
||||
// An unassignment the controller made itself is no exception: many changes at once means a person is
|
||||
// at work, and a person confirms once. On 2026-10-04 one misread file withdrew seven consumers at once
|
||||
// (issue 241); under this rule that is a question, not an act.
|
||||
//
|
||||
// **The backend remembers, not this process.** What is retired, since when and why is read from the
|
||||
// backend's own mark (Adapter.Inventory), so a restart forgets nothing; a consumer the backend holds
|
||||
// active that the mesh no longer asks for is retired by the same rules after a restart as before one.
|
||||
// A consumer found disabled without the mark — withdrawn before this record — is ADOPTED as retired:
|
||||
// marked, said, announced `adopted`, and its clock starts then.
|
||||
//
|
||||
// **A rejection lives as long as this process.** Rejected, the set is kept active and not asked about
|
||||
// again while it stays the same set; a restart asks again — loudly, never silently.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// EventRetirement is the one event a provider says about retirement, its `change` saying what
|
||||
// happened. The controller derives the permission to emit it for every module that receives
|
||||
// contributions, as it does the standing events (novox/hq ADR 0224, 0230).
|
||||
const EventRetirement = "provisioner.retirement"
|
||||
|
||||
// The changes EventRetirement carries.
|
||||
const (
|
||||
ChangeWaiting = "waiting" // a stable set over the bound waits for a person
|
||||
ChangeSettled = "settled" // a waiting or rejected set ended without being retired
|
||||
ChangeApproved = "approved" // a person approved the waiting (or rejected) set
|
||||
ChangeRejected = "rejected" // a person kept the waiting set active
|
||||
ChangeRetired = "retired" // consumers disabled and marked to delete
|
||||
ChangeReenabled = "reenabled" // a retired consumer asked for again, enabled as it was
|
||||
ChangeDeleted = "deleted" // a retired consumer deleted, by a person
|
||||
ChangeAdopted = "adopted" // found disabled without the mark, now retired on record
|
||||
)
|
||||
|
||||
// StableFor is the least time the same unasked set must hold before it is retired (novox/hq ADR 0230):
|
||||
// longer than a controller restart, a store reconnecting or a file half written.
|
||||
const StableFor = 10 * time.Minute
|
||||
|
||||
// KindConsumer is a retired consumer. A backend may list other things set aside for deletion under a
|
||||
// word of its own (postgres: a database renamed aside).
|
||||
const KindConsumer = "consumer"
|
||||
|
||||
// Retired is one thing a provider holds retired, as its backend's mark says.
|
||||
type Retired struct {
|
||||
Consumer string
|
||||
// Node is the consumer's machine, where the mark records it.
|
||||
Node string
|
||||
// RetiredAt is when it was retired; zero for one found disabled without the mesh's mark.
|
||||
RetiredAt time.Time
|
||||
Why string
|
||||
// SizeBytes is what deleting it would free; -1 when the backend cannot say.
|
||||
SizeBytes int64
|
||||
Kind string
|
||||
}
|
||||
|
||||
// Inventory is what a backend holds that the mesh made.
|
||||
type Inventory struct {
|
||||
Active []string
|
||||
Retired []Retired
|
||||
// Sizes is what each active consumer keeps on the backend, in bytes, where the backend can say: what
|
||||
// the controller tells an empty replacement of a consumer's data by (novox/hq ADR 0233).
|
||||
Sizes map[string]int64
|
||||
}
|
||||
|
||||
// retirement is the loop's memory of the sets it is counting, waiting on and was told to keep.
|
||||
type retirement struct {
|
||||
key string // the unasked set, joined
|
||||
count int // consecutive passes that saw it
|
||||
firstSeen time.Time
|
||||
waiting *waitingSet
|
||||
rejected *rejectedSet
|
||||
retired map[string]retiredEntry
|
||||
lastWant map[string]bool
|
||||
seeded bool
|
||||
seedSaid string
|
||||
// noneSaid is whether this process has said that nothing waits: an earlier process may have left a set
|
||||
// waiting that this one never knew (issue 937).
|
||||
noneSaid bool
|
||||
}
|
||||
|
||||
type waitingSet struct {
|
||||
set []string
|
||||
since time.Time
|
||||
saidAt time.Time
|
||||
held int
|
||||
}
|
||||
|
||||
type rejectedSet struct {
|
||||
set []string
|
||||
by, why string
|
||||
at time.Time
|
||||
}
|
||||
|
||||
type retiredEntry struct {
|
||||
node string
|
||||
at time.Time
|
||||
why string
|
||||
}
|
||||
|
||||
// seed reads what the backend holds, once per process: an active consumer the mesh no longer asks
|
||||
// for is held, so it is retired by the same rules as one this process made; one found disabled
|
||||
// without the mark is adopted as retired.
|
||||
func (h *Harness) seed(ctx context.Context, want map[string]bool) {
|
||||
if h.r.seeded {
|
||||
return
|
||||
}
|
||||
inv, err := h.Adapter.Inventory(ctx)
|
||||
if err != nil {
|
||||
if said := err.Error(); said != h.r.seedSaid {
|
||||
h.say("cannot list what the backend holds (%v); a consumer the mesh stopped asking for while this "+
|
||||
"provider was down is not retired until it can", err)
|
||||
h.r.seedSaid = said
|
||||
}
|
||||
return
|
||||
}
|
||||
h.r.seeded = true
|
||||
for _, as := range inv.Active {
|
||||
if _, held := h.applied[as]; !held && !want[as] {
|
||||
// No hash: asked for again, it is applied again, which is harmless and marks it.
|
||||
h.applied[as] = appliedEntry{}
|
||||
h.say("%s: held by the backend and no longer asked for; retired once that holds for %d passes "+
|
||||
"and %s (novox/hq ADR 0230)", as, h.StablePasses, h.StableFor)
|
||||
}
|
||||
}
|
||||
now := h.Now()
|
||||
for _, r := range inv.Retired {
|
||||
if r.Kind != "" && r.Kind != KindConsumer {
|
||||
continue
|
||||
}
|
||||
if !r.RetiredAt.IsZero() {
|
||||
h.r.retired[r.Consumer] = retiredEntry{node: r.Node, at: r.RetiredAt, why: r.Why}
|
||||
continue
|
||||
}
|
||||
if want[r.Consumer] {
|
||||
continue // asked for: this pass's create enables it
|
||||
}
|
||||
why := "found disabled without the mesh's mark — withdrawn before retirement was recorded " +
|
||||
"(novox/hq ADR 0230); retired as found"
|
||||
if err := h.Adapter.Retire(ctx, r.Consumer, nil, why, now); err != nil {
|
||||
h.say("%s: found disabled and could not be marked retired, will try at the next start: %v", r.Consumer, err)
|
||||
continue
|
||||
}
|
||||
h.r.retired[r.Consumer] = retiredEntry{node: r.Node, at: now, why: why}
|
||||
h.say("%s: ADOPTED as retired — %s", r.Consumer, why)
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeAdopted, []map[string]any{
|
||||
{"consumer": r.Consumer, "node": r.Node, "retired_at": stamp(now), "why": why, "size_bytes": r.SizeBytes},
|
||||
}, map[string]any{"why": why}))
|
||||
}
|
||||
}
|
||||
|
||||
// retireUnasked counts the passes that saw the same set no longer asked for and, once it is stable,
|
||||
// retires it — or, past the bound, waits for a person.
|
||||
func (h *Harness) retireUnasked(ctx context.Context, want map[string]bool) {
|
||||
var unasked []string
|
||||
for as := range h.applied {
|
||||
if !want[as] {
|
||||
unasked = append(unasked, as)
|
||||
}
|
||||
}
|
||||
sort.Strings(unasked)
|
||||
key := strings.Join(unasked, "\x00")
|
||||
now := h.Now()
|
||||
switch {
|
||||
case len(unasked) == 0:
|
||||
h.settle("every consumer held is asked for again")
|
||||
h.r.key, h.r.count = "", 0
|
||||
return
|
||||
case key != h.r.key:
|
||||
if h.r.waiting != nil || h.r.rejected != nil {
|
||||
h.settle("the set the mesh no longer asks for changed")
|
||||
}
|
||||
h.r.key, h.r.count, h.r.firstSeen = key, 1, now
|
||||
h.say("no longer asked for: %s; retired once the same set holds for %d passes and %s",
|
||||
strings.Join(unasked, ", "), h.StablePasses, h.StableFor)
|
||||
default:
|
||||
h.r.count++
|
||||
}
|
||||
if h.r.rejected != nil || h.r.count < h.StablePasses || now.Sub(h.r.firstSeen) < h.StableFor {
|
||||
return
|
||||
}
|
||||
if h.overTheBound(len(unasked), len(h.applied)) {
|
||||
h.wait(unasked)
|
||||
return
|
||||
}
|
||||
why := fmt.Sprintf("the mesh stopped asking for it: the same result in %d consecutive passes over %s, from %s",
|
||||
h.r.count, now.Sub(h.r.firstSeen).Round(time.Second), stamp(h.r.firstSeen))
|
||||
h.retire(ctx, unasked, why, nil)
|
||||
}
|
||||
|
||||
// overTheBound says retiring n of the held consumers at once needs a person.
|
||||
func (h *Harness) overTheBound(n, held int) bool {
|
||||
if n == 0 {
|
||||
return false
|
||||
}
|
||||
return n > h.RetireAtOnce || (held > 1 && float64(n) > h.RetireFraction*float64(held))
|
||||
}
|
||||
|
||||
func (h *Harness) bound(held int) string {
|
||||
return fmt.Sprintf("more than %d at once, or more than %.0f%% of the %d held", h.RetireAtOnce,
|
||||
h.RetireFraction*100, held)
|
||||
}
|
||||
|
||||
// wait holds a stable set over the bound for a person, said once and announced again every
|
||||
// SayAgainEvery so a controller that missed the first hears the next.
|
||||
func (h *Harness) wait(set []string) {
|
||||
now := h.Now()
|
||||
w := h.r.waiting
|
||||
if w == nil {
|
||||
w = &waitingSet{set: set, since: now, held: len(h.applied)}
|
||||
h.r.waiting = w
|
||||
h.say("RETIREMENT WAITS FOR A PERSON: the mesh no longer asks for %d of the %d consumer(s) this provider "+
|
||||
"holds (%s), %s. Nothing is retired; each stays active until `retire approve %s <module>` or "+
|
||||
"`retire reject` (novox/hq ADR 0230)", len(set), w.held, strings.Join(set, ", "), h.bound(w.held), h.Node)
|
||||
} else if now.Sub(w.saidAt) < h.SayAgainEvery {
|
||||
return
|
||||
}
|
||||
w.saidAt = now
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeWaiting, h.named(set), map[string]any{
|
||||
"held": w.held, "bound": h.bound(w.held), "since": stamp(w.since),
|
||||
}))
|
||||
}
|
||||
|
||||
// settle ends a waiting or rejected set that the mesh's own answer made moot.
|
||||
func (h *Harness) settle(why string) {
|
||||
var set []string
|
||||
switch {
|
||||
case h.r.waiting != nil:
|
||||
set = h.r.waiting.set
|
||||
case h.r.rejected != nil:
|
||||
set = h.r.rejected.set
|
||||
case h.r.noneSaid:
|
||||
return
|
||||
default:
|
||||
// A restart forgets a set an earlier process waited on, and so would never say it ended: the
|
||||
// controller's condition would ask a person to approve what nobody waits on any more (issue 937).
|
||||
// The first time this process finds every consumer held asked for, it says so once, with no set.
|
||||
h.r.noneSaid = true
|
||||
h.say("nothing waits for a person: %s", why)
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeSettled, []map[string]any{}, map[string]any{"why": why}))
|
||||
return
|
||||
}
|
||||
h.r.waiting, h.r.rejected = nil, nil
|
||||
h.say("retirement of %s settled without retiring: %s", strings.Join(set, ", "), why)
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeSettled, h.named(set), map[string]any{"why": why}))
|
||||
}
|
||||
|
||||
// retire disables and marks each consumer of set; one that fails stays held and is tried again once
|
||||
// its set is stable again.
|
||||
func (h *Harness) retire(ctx context.Context, set []string, why string, extra map[string]any) []string {
|
||||
now := h.Now()
|
||||
held := len(h.applied)
|
||||
var done []string
|
||||
var said []map[string]any
|
||||
for _, as := range set {
|
||||
was, ok := h.applied[as]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if err := h.Adapter.Retire(ctx, as, was.derived, why, now); err != nil {
|
||||
h.say("%s: retiring failed, will try again: %v", as, err)
|
||||
continue
|
||||
}
|
||||
delete(h.applied, as)
|
||||
delete(h.lost, as)
|
||||
delete(h.failing, as)
|
||||
h.r.retired[as] = retiredEntry{node: was.node, at: now, why: why}
|
||||
h.recovered(as, "retired")
|
||||
h.say("%s: RETIRED — its access disabled and its data kept, marked to delete (%s). Asked for again "+
|
||||
"it is re-enabled as it was; it is deleted only by `cleanup delete` (novox/hq ADR 0230)", as, why)
|
||||
done = append(done, as)
|
||||
said = append(said, map[string]any{"consumer": as, "node": was.node, "retired_at": stamp(now), "why": why})
|
||||
}
|
||||
h.r.key, h.r.count, h.r.waiting, h.r.rejected = "", 0, nil, nil
|
||||
if len(done) > 0 {
|
||||
body := map[string]any{"held": held, "why": why}
|
||||
for k, v := range extra {
|
||||
body[k] = v
|
||||
}
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeRetired, said, body))
|
||||
}
|
||||
return done
|
||||
}
|
||||
|
||||
// reenabled says a retired consumer was asked for again and its create enabled it.
|
||||
func (h *Harness) reenabled(as, node string) {
|
||||
e, was := h.r.retired[as]
|
||||
if !was {
|
||||
return
|
||||
}
|
||||
delete(h.r.retired, as)
|
||||
h.say("%s: asked for again — re-enabled as it was, its mark to delete cleared (retired %s: %s)", as,
|
||||
stamp(e.at), e.why)
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeReenabled, []map[string]any{
|
||||
{"consumer": as, "node": node, "retired_at": stamp(e.at), "why": e.why},
|
||||
}, nil))
|
||||
}
|
||||
|
||||
// Approve retires exactly the set waiting (or the set rejected) — a person's confirmation.
|
||||
func (h *Harness) Approve(ctx context.Context, consumers []string, why, by, via string) ([]string, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.init()
|
||||
if strings.TrimSpace(why) == "" {
|
||||
return nil, errors.New("approve: say why")
|
||||
}
|
||||
set := sorted(consumers)
|
||||
var held []string
|
||||
switch {
|
||||
case h.r.waiting != nil && same(set, h.r.waiting.set):
|
||||
held = h.r.waiting.set
|
||||
case h.r.rejected != nil && same(set, h.r.rejected.set):
|
||||
held = h.r.rejected.set
|
||||
default:
|
||||
return nil, fmt.Errorf("approve: %s is not the set waiting here — %s", orNone(set), h.waitingWords())
|
||||
}
|
||||
h.say("retirement of %s APPROVED by %s: %s", strings.Join(held, ", "), orSomebody(by), why)
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeApproved, h.named(held),
|
||||
map[string]any{"why": why, "by": by, "via": via}))
|
||||
reason := fmt.Sprintf("the mesh stopped asking for it; approved by %s: %s", orSomebody(by), why)
|
||||
return h.retire(ctx, held, reason, map[string]any{"by": by, "via": via}), nil
|
||||
}
|
||||
|
||||
// Reject keeps the waiting set active; it is not asked about again while it stays the same set.
|
||||
func (h *Harness) Reject(consumers []string, why, by, via string) ([]string, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.init()
|
||||
if strings.TrimSpace(why) == "" {
|
||||
return nil, errors.New("reject: say why")
|
||||
}
|
||||
set := sorted(consumers)
|
||||
if h.r.waiting == nil || !same(set, h.r.waiting.set) {
|
||||
return nil, fmt.Errorf("reject: %s is not the set waiting here — %s", orNone(set), h.waitingWords())
|
||||
}
|
||||
h.r.rejected = &rejectedSet{set: h.r.waiting.set, by: by, why: why, at: h.Now()}
|
||||
h.r.waiting = nil
|
||||
h.say("retirement of %s REJECTED by %s: %s. Kept active, and not asked about again while the mesh goes on "+
|
||||
"not asking for exactly these (until this provider restarts)", strings.Join(set, ", "), orSomebody(by), why)
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeRejected, h.named(set),
|
||||
map[string]any{"why": why, "by": by, "via": via}))
|
||||
return set, nil
|
||||
}
|
||||
|
||||
// DeleteRetired deletes one retired consumer, by a person's word: never an active one, never one the
|
||||
// mesh asks for.
|
||||
func (h *Harness) DeleteRetired(ctx context.Context, consumer, why, by, via string) (int64, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.init()
|
||||
if strings.TrimSpace(why) == "" {
|
||||
return 0, errors.New("delete: say why")
|
||||
}
|
||||
if h.r.lastWant[consumer] {
|
||||
return 0, fmt.Errorf("delete: the mesh asks for %s — it is not retired", consumer)
|
||||
}
|
||||
if _, held := h.applied[consumer]; held {
|
||||
return 0, fmt.Errorf("delete: %s is active here — only a retired consumer is deleted", consumer)
|
||||
}
|
||||
inv, err := h.Adapter.Inventory(ctx)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("delete: what the backend holds cannot be read, so nothing is deleted: %w", err)
|
||||
}
|
||||
var found *Retired
|
||||
for i := range inv.Retired {
|
||||
if inv.Retired[i].Consumer == consumer {
|
||||
found = &inv.Retired[i]
|
||||
}
|
||||
}
|
||||
if found == nil {
|
||||
return 0, fmt.Errorf("delete: %s is not retired here — only a retired consumer is deleted", consumer)
|
||||
}
|
||||
freed, err := h.Adapter.Delete(ctx, *found)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
delete(h.r.retired, consumer)
|
||||
h.say("%s: DELETED by %s: %s (retired %s: %s; %d bytes freed)", consumer, orSomebody(by), why,
|
||||
stampOr(found.RetiredAt), found.Why, freed)
|
||||
h.announce(EventRetirement, h.retirementBody(ChangeDeleted, []map[string]any{{
|
||||
"consumer": consumer, "node": found.Node, "retired_at": stampOr(found.RetiredAt), "why": found.Why,
|
||||
"size_bytes": found.SizeBytes, "kind": kindOf(*found),
|
||||
}}, map[string]any{"why": why, "by": by, "via": via, "freed_bytes": freed}))
|
||||
return freed, nil
|
||||
}
|
||||
|
||||
// Retirement is what a person (and the controller's `cleanup list` and its probe) asks: what is
|
||||
// held, waiting, rejected and retired here.
|
||||
func (h *Harness) Retirement(ctx context.Context) (map[string]any, error) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
h.init()
|
||||
inv, err := h.Adapter.Inventory(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var held []string
|
||||
for as := range h.applied {
|
||||
held = append(held, as)
|
||||
}
|
||||
sort.Strings(held)
|
||||
retired := []map[string]any{}
|
||||
for _, r := range inv.Retired {
|
||||
retired = append(retired, map[string]any{"consumer": r.Consumer, "node": r.Node,
|
||||
"retired_at": stampOr(r.RetiredAt), "why": r.Why, "size_bytes": r.SizeBytes, "kind": kindOf(r)})
|
||||
}
|
||||
sizes := map[string]int64{}
|
||||
for _, as := range held {
|
||||
if size, ok := inv.Sizes[as]; ok && size >= 0 {
|
||||
sizes[as] = size
|
||||
}
|
||||
}
|
||||
out := map[string]any{"resource": h.Resource, "node": h.Node, "held": orEmptyList(held), "held_sizes": sizes,
|
||||
"stable_passes": h.StablePasses, "stable_for_seconds": int(h.StableFor.Seconds()), "bound": h.bound(len(h.applied)), "waiting": nil, "rejected": nil,
|
||||
"retired": retired}
|
||||
if w := h.r.waiting; w != nil {
|
||||
out["waiting"] = map[string]any{"consumers": h.named(w.set), "since": stamp(w.since), "held": w.held}
|
||||
}
|
||||
if r := h.r.rejected; r != nil {
|
||||
out["rejected"] = map[string]any{"consumers": h.named(r.set), "by": r.by, "why": r.why, "at": stamp(r.at)}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (h *Harness) waitingWords() string {
|
||||
switch {
|
||||
case h.r.waiting != nil:
|
||||
return "waiting: " + strings.Join(h.r.waiting.set, ", ")
|
||||
case h.r.rejected != nil:
|
||||
return "nothing waits; rejected and kept: " + strings.Join(h.r.rejected.set, ", ")
|
||||
}
|
||||
return "nothing waits for a person here"
|
||||
}
|
||||
|
||||
// named is a set with each consumer's machine, as the events and the tools say it.
|
||||
func (h *Harness) named(set []string) []map[string]any {
|
||||
out := make([]map[string]any, 0, len(set))
|
||||
for _, as := range set {
|
||||
out = append(out, map[string]any{"consumer": as, "node": h.applied[as].node})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *Harness) retirementBody(change string, consumers []map[string]any, extra map[string]any) map[string]any {
|
||||
body := map[string]any{"provider": h.Resource, "provider-node": h.Node, "change": change,
|
||||
"consumers": consumers, "at": stamp(h.Now())}
|
||||
for k, v := range extra {
|
||||
body[k] = v
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
// RetirementTools are the four tools every provider serves for retirement, the same names in every
|
||||
// module: the controller's `retire` and `cleanup` verbs ask them on the provider's machine.
|
||||
func RetirementTools(h *Harness) []stdio.Tool {
|
||||
if h == nil {
|
||||
return nil
|
||||
}
|
||||
ask := func() (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
}
|
||||
who := map[string]any{
|
||||
"why": map[string]any{"type": "string", "description": "why — required, kept in the hand-act log"},
|
||||
"by": map[string]any{"type": "string", "description": "who decided"},
|
||||
"via": map[string]any{"type": "string", "description": "mesh-controller when asked through its verbs"},
|
||||
}
|
||||
withSet := map[string]any{"consumers": map[string]any{"type": "array", "items": map[string]any{"type": "string"},
|
||||
"description": "the set, exactly as provisioner_retirement names it"}}
|
||||
for k, v := range who {
|
||||
withSet[k] = v
|
||||
}
|
||||
withOne := map[string]any{
|
||||
"consumer": map[string]any{"type": "string", "description": "the retired consumer to delete"},
|
||||
"confirm": map[string]any{"type": "string", "description": "the consumer's name again, to say this is meant"},
|
||||
}
|
||||
for k, v := range who {
|
||||
withOne[k] = v
|
||||
}
|
||||
return []stdio.Tool{
|
||||
{Name: "provisioner_retirement",
|
||||
Description: "What this provider holds, what waits for a person to approve its retirement, what was rejected, " +
|
||||
"and every retired consumer with when, why and its size (novox/hq ADR 0230).",
|
||||
Run: func(map[string]any) (any, error) {
|
||||
ctx, cancel := ask()
|
||||
defer cancel()
|
||||
return h.Retirement(ctx)
|
||||
}},
|
||||
{Name: "provisioner_retire_approve",
|
||||
Description: "Retire exactly the set waiting for a person (or the set rejected earlier): access disabled, data " +
|
||||
"kept and marked to delete. Use the controller's `retire approve`, which records the hand act.",
|
||||
Input: withSet,
|
||||
Run: func(args map[string]any) (any, error) {
|
||||
ctx, cancel := ask()
|
||||
defer cancel()
|
||||
done, err := h.Approve(ctx, strs(args["consumers"]), argString(args, "why"), argString(args, "by"), argString(args, "via"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return map[string]any{"retired": orEmptyList(done)}, nil
|
||||
}},
|
||||
{Name: "provisioner_retire_reject",
|
||||
Description: "Keep the set waiting for a person active. Use the controller's `retire reject`.",
|
||||
Input: withSet,
|
||||
Run: func(args map[string]any) (any, error) {
|
||||
kept, err := h.Reject(strs(args["consumers"]), argString(args, "why"), argString(args, "by"), argString(args, "via"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return map[string]any{"kept": kept}, nil
|
||||
}},
|
||||
{Name: "provisioner_delete",
|
||||
Description: "Delete one RETIRED consumer — its login and its data — for good. Refused for anything active or " +
|
||||
"asked for. Use the controller's `cleanup delete`, which records the hand act.",
|
||||
Input: withOne,
|
||||
Run: func(args map[string]any) (any, error) {
|
||||
consumer := argString(args, "consumer")
|
||||
if consumer == "" || argString(args, "confirm") != consumer {
|
||||
return nil, errors.New("delete: name the consumer, and repeat it in confirm")
|
||||
}
|
||||
ctx, cancel := ask()
|
||||
defer cancel()
|
||||
freed, err := h.DeleteRetired(ctx, consumer, argString(args, "why"), argString(args, "by"), argString(args, "via"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return map[string]any{"deleted": consumer, "freed_bytes": freed}, nil
|
||||
}},
|
||||
}
|
||||
}
|
||||
|
||||
func strs(v any) []string {
|
||||
list, _ := v.([]any)
|
||||
out := []string{}
|
||||
for _, x := range list {
|
||||
if s, ok := x.(string); ok && s != "" {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sorted(list []string) []string {
|
||||
out := append([]string(nil), list...)
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func same(a, b []string) bool {
|
||||
return strings.Join(sorted(a), "\x00") == strings.Join(sorted(b), "\x00")
|
||||
}
|
||||
|
||||
func orNone(set []string) string {
|
||||
if len(set) == 0 {
|
||||
return "an empty set"
|
||||
}
|
||||
return strings.Join(set, ", ")
|
||||
}
|
||||
|
||||
func orSomebody(by string) string {
|
||||
if by == "" {
|
||||
return "a person"
|
||||
}
|
||||
return by
|
||||
}
|
||||
|
||||
func orEmptyList(list []string) []string {
|
||||
if list == nil {
|
||||
return []string{}
|
||||
}
|
||||
return list
|
||||
}
|
||||
|
||||
func kindOf(r Retired) string {
|
||||
if r.Kind == "" {
|
||||
return KindConsumer
|
||||
}
|
||||
return r.Kind
|
||||
}
|
||||
|
||||
func stamp(t time.Time) string { return t.UTC().Format(time.RFC3339) }
|
||||
|
||||
func stampOr(t time.Time) string {
|
||||
if t.IsZero() {
|
||||
return ""
|
||||
}
|
||||
return stamp(t)
|
||||
}
|
||||
|
||||
func argString(args map[string]any, key string) string {
|
||||
s, _ := args[key].(string)
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,571 @@
|
||||
package harness
|
||||
|
||||
// Retirement (novox/hq ADR 0230), as the shared loop does it: a consumer the mesh stops asking for is
|
||||
// retired only after the same result in five consecutive passes, too many at once wait for a person,
|
||||
// asked for again it is re-enabled, and only a person deletes. The same file in every Go provider.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// recorder is a backend that remembers what it holds, active and retired, as a real one's mark does.
|
||||
type recorder struct {
|
||||
created []Provision
|
||||
removed []string // retired, in order
|
||||
deleted []string
|
||||
held bool
|
||||
failing error
|
||||
holdsErr error
|
||||
retErr error
|
||||
inv Inventory
|
||||
invErr error
|
||||
}
|
||||
|
||||
func (r *recorder) Create(_ context.Context, p Provision) error {
|
||||
if r.failing != nil {
|
||||
return r.failing
|
||||
}
|
||||
r.created = append(r.created, p)
|
||||
r.inv.Retired = withoutRetired(r.inv.Retired, p.As)
|
||||
if !listHas(r.inv.Active, p.As) {
|
||||
r.inv.Active = append(r.inv.Active, p.As)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *recorder) Retire(_ context.Context, as string, _ map[string]any, why string, at time.Time) error {
|
||||
if r.retErr != nil {
|
||||
return r.retErr
|
||||
}
|
||||
r.removed = append(r.removed, as)
|
||||
r.inv.Active = without(r.inv.Active, as)
|
||||
r.inv.Retired = append(withoutRetired(r.inv.Retired, as),
|
||||
Retired{Consumer: as, RetiredAt: at, Why: why, SizeBytes: 42, Kind: KindConsumer})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *recorder) Holds(context.Context, Provision) (bool, error) {
|
||||
if r.holdsErr != nil {
|
||||
return false, r.holdsErr
|
||||
}
|
||||
return r.held, nil
|
||||
}
|
||||
|
||||
func (r *recorder) Inventory(context.Context) (Inventory, error) { return r.inv, r.invErr }
|
||||
|
||||
func (r *recorder) Delete(_ context.Context, x Retired) (int64, error) {
|
||||
r.deleted = append(r.deleted, x.Consumer)
|
||||
r.inv.Retired = withoutRetired(r.inv.Retired, x.Consumer)
|
||||
return x.SizeBytes, nil
|
||||
}
|
||||
|
||||
func listHas(list []string, s string) bool {
|
||||
for _, x := range list {
|
||||
if x == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func without(list []string, s string) []string {
|
||||
var out []string
|
||||
for _, x := range list {
|
||||
if x != s {
|
||||
out = append(out, x)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func withoutRetired(list []Retired, s string) []Retired {
|
||||
var out []Retired
|
||||
for _, x := range list {
|
||||
if x.Consumer != s {
|
||||
out = append(out, x)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// holding gives the provider n consumers c1…cn and applies them.
|
||||
func holding(w *world, n int) []map[string]any {
|
||||
var given []map[string]any
|
||||
for i := 1; i <= n; i++ {
|
||||
given = append(given, map[string]any{"as": fmt.Sprintf("c%d", i), "node": "anchor"})
|
||||
}
|
||||
w.give(given...)
|
||||
w.h.Reconcile(ctx)
|
||||
return given
|
||||
}
|
||||
|
||||
// pass is one reconcile five seconds after the last.
|
||||
func (w *world) pass() {
|
||||
w.now = w.now.Add(5 * time.Second)
|
||||
w.h.Reconcile(ctx)
|
||||
}
|
||||
|
||||
func retirements(said []announced) []string {
|
||||
var out []string
|
||||
for _, a := range said {
|
||||
if a.event == EventRetirement {
|
||||
out = append(out, a.body["change"].(string))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func noneWaits(a announced) bool {
|
||||
c, _ := a.body["consumers"].([]map[string]any)
|
||||
return a.body["change"] == ChangeSettled && len(c) == 0
|
||||
}
|
||||
|
||||
func lastRetirement(t *testing.T, said []announced) map[string]any {
|
||||
t.Helper()
|
||||
for i := len(said) - 1; i >= 0; i-- {
|
||||
if said[i].event == EventRetirement {
|
||||
return said[i].body
|
||||
}
|
||||
}
|
||||
t.Fatal("nothing about retirement was announced")
|
||||
return nil
|
||||
}
|
||||
|
||||
func namesOf(body map[string]any) string {
|
||||
var out []string
|
||||
for _, c := range body["consumers"].([]map[string]any) {
|
||||
out = append(out, c["consumer"].(string))
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
// settle passes every five seconds until the same answer has held for StableFor, and one pass more.
|
||||
func (w *world) settle() { w.passes(StableFor + 5*time.Second) }
|
||||
|
||||
func TestATransientEmptyListForFourPassesRetiresNothing(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
given := holding(w, 1)
|
||||
w.give()
|
||||
for i := 0; i < 4; i++ {
|
||||
w.pass()
|
||||
}
|
||||
w.give(given...)
|
||||
w.pass()
|
||||
if len(w.a.removed) != 0 || len(retirements(*said)) != 0 {
|
||||
t.Fatalf("four passes retired %v and said %v", w.a.removed, retirements(*said))
|
||||
}
|
||||
}
|
||||
|
||||
// Five identical passes are twenty-five seconds — shorter than a controller restart. Both must hold:
|
||||
// five passes AND ten minutes of the same answer (novox/hq ADR 0230).
|
||||
func TestFivePassesInTwentyFiveSecondsRetireNothingTenMinutesDo(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
holding(w, 1)
|
||||
w.give()
|
||||
for i := 0; i < 5; i++ {
|
||||
w.pass()
|
||||
}
|
||||
if len(w.a.removed) != 0 || len(retirements(*said)) != 0 {
|
||||
t.Fatalf("five passes in 25 s retired %v", w.a.removed)
|
||||
}
|
||||
w.passes(StableFor - 30*time.Second)
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("retired before the set held %s: %v", StableFor, w.a.removed)
|
||||
}
|
||||
w.passes(time.Minute)
|
||||
if strings.Join(w.a.removed, ",") != "c1" {
|
||||
t.Fatalf("the same set held %s and was not retired: %v", StableFor, w.a.removed)
|
||||
}
|
||||
// Ten minutes in one slow pass are not five passes.
|
||||
w2 := newWorld(t)
|
||||
holding(w2, 1)
|
||||
w2.give()
|
||||
w2.pass()
|
||||
w2.now = w2.now.Add(StableFor)
|
||||
w2.pass()
|
||||
if len(w2.a.removed) != 0 {
|
||||
t.Fatalf("two passes ten minutes apart retired %v", w2.a.removed)
|
||||
}
|
||||
w2.passes(15 * time.Second)
|
||||
if len(w2.a.removed) != 1 {
|
||||
t.Fatalf("five passes over ten minutes did not retire: %v", w2.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStableSetIsRetiredAndAskedAgainReEnables(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
given := holding(w, 3)
|
||||
w.give(given[:2]...)
|
||||
w.settle()
|
||||
if strings.Join(w.a.removed, ",") != "c3" {
|
||||
t.Fatalf("retired %v", w.a.removed)
|
||||
}
|
||||
body := lastRetirement(t, *said)
|
||||
if body["change"] != ChangeRetired || namesOf(body) != "c3" || body["held"] != 3 || body["provider-node"] != "anchor" ||
|
||||
!strings.Contains(body["why"].(string), "consecutive passes over 10m") {
|
||||
t.Fatalf("%v", body)
|
||||
}
|
||||
if len(w.a.inv.Retired) != 1 || w.a.inv.Retired[0].Consumer != "c3" {
|
||||
t.Fatalf("the backend does not hold it retired: %+v", w.a.inv)
|
||||
}
|
||||
// Asked for again: the ordinary create, on the first pass, and said.
|
||||
created := len(w.a.created)
|
||||
w.give(given...)
|
||||
w.pass()
|
||||
if len(w.a.created) != created+1 || w.a.created[created].As != "c3" {
|
||||
t.Fatalf("not created again: %v", w.a.created)
|
||||
}
|
||||
if body := lastRetirement(t, *said); body["change"] != ChangeReenabled || namesOf(body) != "c3" {
|
||||
t.Fatalf("%v", body)
|
||||
}
|
||||
if len(w.a.inv.Retired) != 0 {
|
||||
t.Fatalf("still marked retired: %+v", w.a.inv.Retired)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnreadablePassStartsTheCountAgain(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
holding(w, 1)
|
||||
w.give()
|
||||
w.passes(StableFor - time.Minute)
|
||||
os.WriteFile(w.receives, []byte("{"), 0o600)
|
||||
w.pass()
|
||||
w.give()
|
||||
w.passes(StableFor - time.Minute)
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("an unreadable pass counted: %v", w.a.removed)
|
||||
}
|
||||
w.passes(2 * time.Minute)
|
||||
if len(w.a.removed) != 1 {
|
||||
t.Fatalf("not retired after ten minutes of readable passes: %v", w.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADifferentSetStartsTheCountAgain(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
given := holding(w, 5)
|
||||
w.give(given[:4]...)
|
||||
w.passes(StableFor - time.Minute)
|
||||
w.give(given[:3]...)
|
||||
w.passes(StableFor - time.Minute)
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("a changed set kept its count: %v", w.a.removed)
|
||||
}
|
||||
w.passes(2 * time.Minute)
|
||||
if strings.Join(w.a.removed, ",") != "c4,c5" {
|
||||
t.Fatalf("%v", w.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdditionsAndChangesAreNeverDelayed(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
holding(w, 1)
|
||||
w.give(map[string]any{"as": "c1", "node": "anchor"}, map[string]any{"as": "c2"})
|
||||
w.pass()
|
||||
if len(w.a.created) != 2 || w.a.created[1].As != "c2" {
|
||||
t.Fatalf("an addition waited: %v", w.a.created)
|
||||
}
|
||||
w.give(map[string]any{"as": "c1", "node": "anchor", "values": map[string]any{"name": "rotated"}}, map[string]any{"as": "c2"})
|
||||
w.pass()
|
||||
if len(w.a.created) != 3 || w.a.created[2].As != "c1" {
|
||||
t.Fatalf("a change waited: %v", w.a.created)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTooManyWaitsForAPersonAndApproveRetiresExactlyThatSet(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
holding(w, 4)
|
||||
w.give()
|
||||
w.passes(15 * time.Minute)
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("four of four were retired without a person: %v", w.a.removed)
|
||||
}
|
||||
if got := strings.Join(retirements(*said), ","); got != ChangeWaiting {
|
||||
t.Fatalf("said %q, want one waiting", got)
|
||||
}
|
||||
body := lastRetirement(t, *said)
|
||||
if namesOf(body) != "c1,c2,c3,c4" || body["held"] != 4 || body["bound"] == "" {
|
||||
t.Fatalf("%v", body)
|
||||
}
|
||||
if !strings.Contains(strings.Join(w.said, "\n"), "RETIREMENT WAITS FOR A PERSON") {
|
||||
t.Fatal("the wait was not said")
|
||||
}
|
||||
// Said again every quarter of an hour while it waits.
|
||||
w.passes(11 * time.Minute)
|
||||
if got := strings.Join(retirements(*said), ","); got != "waiting,waiting" {
|
||||
t.Fatalf("%q", got)
|
||||
}
|
||||
|
||||
if _, err := w.h.Approve(ctx, []string{"c1", "c2"}, "tidy", "operator", "mesh-controller"); err == nil {
|
||||
t.Fatal("approved a set that is not the one waiting")
|
||||
}
|
||||
if _, err := w.h.Approve(ctx, []string{"c4", "c3", "c2", "c1"}, "", "operator", ""); err == nil {
|
||||
t.Fatal("approved without a why")
|
||||
}
|
||||
done, err := w.h.Approve(ctx, []string{"c4", "c3", "c2", "c1"}, "the old machine is gone", "operator", "mesh-controller")
|
||||
if err != nil || strings.Join(done, ",") != "c1,c2,c3,c4" || strings.Join(w.a.removed, ",") != "c1,c2,c3,c4" {
|
||||
t.Fatal(done, err, w.a.removed)
|
||||
}
|
||||
changes := retirements(*said)
|
||||
if strings.Join(changes[len(changes)-2:], ",") != "approved,retired" {
|
||||
t.Fatalf("%v", changes)
|
||||
}
|
||||
if b := lastRetirement(t, *said); b["by"] != "operator" || b["via"] != "mesh-controller" ||
|
||||
!strings.Contains(b["why"].(string), "the old machine is gone") {
|
||||
t.Fatalf("%v", b)
|
||||
}
|
||||
// Nothing more waits.
|
||||
w.passes(time.Minute)
|
||||
if r, _ := w.h.Retirement(ctx); r["waiting"] != nil || len(r["retired"].([]map[string]any)) != 4 {
|
||||
t.Fatalf("%v", r)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRejectedIsKeptAndNotAskedAgainUntilTheSetChanges(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
given := holding(w, 4)
|
||||
w.give()
|
||||
w.settle()
|
||||
if _, err := w.h.Reject([]string{"c1"}, "no", "operator", ""); err == nil {
|
||||
t.Fatal("rejected a set that is not the one waiting")
|
||||
}
|
||||
kept, err := w.h.Reject([]string{"c1", "c2", "c3", "c4"}, "a person is moving them", "operator", "mesh-controller")
|
||||
if err != nil || len(kept) != 4 {
|
||||
t.Fatal(kept, err)
|
||||
}
|
||||
w.passes(time.Hour)
|
||||
if len(w.a.removed) != 0 {
|
||||
t.Fatalf("a rejected set was retired: %v", w.a.removed)
|
||||
}
|
||||
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected" {
|
||||
t.Fatalf("asked again after a rejection: %q", got)
|
||||
}
|
||||
r, _ := w.h.Retirement(ctx)
|
||||
if r["rejected"] == nil || r["waiting"] != nil {
|
||||
t.Fatalf("%v", r)
|
||||
}
|
||||
// One of them asked for again: a different answer, so the rejection is settled and counting
|
||||
// starts over — three of four is over the bound again, and waits again.
|
||||
w.give(given[0])
|
||||
w.pass()
|
||||
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected,settled" {
|
||||
t.Fatalf("%q", got)
|
||||
}
|
||||
w.settle()
|
||||
if got := strings.Join(retirements(*said), ","); got != "waiting,rejected,settled,waiting" {
|
||||
t.Fatalf("%q", got)
|
||||
}
|
||||
// A rejected set can still be approved later.
|
||||
w2, _ := standingWorld(t)
|
||||
holding(w2, 4)
|
||||
w2.give()
|
||||
w2.settle()
|
||||
w2.h.Reject([]string{"c1", "c2", "c3", "c4"}, "wait", "operator", "")
|
||||
if done, err := w2.h.Approve(ctx, []string{"c1", "c2", "c3", "c4"}, "now", "operator", ""); err != nil || len(done) != 4 {
|
||||
t.Fatal(done, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWaitingSetAskedForAgainSettles(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
given := holding(w, 4)
|
||||
w.give()
|
||||
w.settle()
|
||||
w.give(given...)
|
||||
w.pass()
|
||||
if got := strings.Join(retirements(*said), ","); got != "waiting,settled" || len(w.a.removed) != 0 {
|
||||
t.Fatalf("%q %v", got, w.a.removed)
|
||||
}
|
||||
if _, err := w.h.Approve(ctx, []string{"c1", "c2", "c3", "c4"}, "late", "operator", ""); err == nil {
|
||||
t.Fatal("approved a set that no longer waits")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteRemovesOnlyThatRetiredConsumer(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
given := holding(w, 5)
|
||||
w.give(given[:3]...)
|
||||
w.settle()
|
||||
if strings.Join(w.a.removed, ",") != "c4,c5" {
|
||||
t.Fatalf("%v", w.a.removed)
|
||||
}
|
||||
if _, err := w.h.DeleteRetired(ctx, "c1", "tidy", "operator", ""); err == nil {
|
||||
t.Fatal("deleted an active consumer")
|
||||
}
|
||||
if _, err := w.h.DeleteRetired(ctx, "c5", "", "operator", ""); err == nil {
|
||||
t.Fatal("deleted without a why")
|
||||
}
|
||||
if _, err := w.h.DeleteRetired(ctx, "nobody", "tidy", "operator", ""); err == nil {
|
||||
t.Fatal("deleted something not retired")
|
||||
}
|
||||
freed, err := w.h.DeleteRetired(ctx, "c5", "the experiment is over", "operator", "mesh-controller")
|
||||
if err != nil || freed != 42 || strings.Join(w.a.deleted, ",") != "c5" {
|
||||
t.Fatal(freed, err, w.a.deleted)
|
||||
}
|
||||
if b := lastRetirement(t, *said); b["change"] != ChangeDeleted || namesOf(b) != "c5" || b["freed_bytes"] != int64(42) {
|
||||
t.Fatalf("%v", b)
|
||||
}
|
||||
r, _ := w.h.Retirement(ctx)
|
||||
if left := r["retired"].([]map[string]any); len(left) != 1 || left[0]["consumer"] != "c4" {
|
||||
t.Fatalf("%v", r)
|
||||
}
|
||||
// Retired and asked for again before anybody deleted it: refused, it is the mesh's again.
|
||||
w.give(given[:4]...)
|
||||
w.pass()
|
||||
if _, err := w.h.DeleteRetired(ctx, "c4", "tidy", "operator", ""); err == nil {
|
||||
t.Fatal("deleted a consumer the mesh asks for")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheBound(t *testing.T) {
|
||||
h := &Harness{}
|
||||
h.init()
|
||||
for _, c := range []struct{ n, held int }{{1, 1}, {1, 2}, {1, 3}, {3, 7}, {3, 6}, {2, 5}} {
|
||||
if h.overTheBound(c.n, c.held) {
|
||||
t.Errorf("%d of %d waits for a person", c.n, c.held)
|
||||
}
|
||||
}
|
||||
for _, c := range []struct{ n, held int }{{2, 2}, {2, 3}, {4, 7}, {4, 10}, {7, 7}} {
|
||||
if !h.overTheBound(c.n, c.held) {
|
||||
t.Errorf("%d of %d is retired without a person", c.n, c.held)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARestartRetiresWhatTheBackendHoldsUnaskedAndAdoptsWhatItFindsDisabled(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
w.a.inv = Inventory{Active: []string{"kept", "orphan"}, Retired: []Retired{
|
||||
{Consumer: "locked-before", SizeBytes: 7, Kind: KindConsumer},
|
||||
{Consumer: "old_deleted_20261005", RetiredAt: time.Date(2026, 10, 5, 0, 0, 0, 0, time.UTC), Kind: "set-aside-database"},
|
||||
}}
|
||||
w.give(map[string]any{"as": "kept"})
|
||||
w.h.Reconcile(ctx)
|
||||
if b := lastRetirement(t, *said); b["change"] != ChangeAdopted || namesOf(b) != "locked-before" {
|
||||
t.Fatalf("%v", b)
|
||||
}
|
||||
if strings.Join(w.a.removed, ",") != "locked-before" {
|
||||
t.Fatalf("adopting did not mark it: %v", w.a.removed)
|
||||
}
|
||||
w.settle()
|
||||
if strings.Join(w.a.removed, ",") != "locked-before,orphan" {
|
||||
t.Fatalf("an orphan the backend holds was not retired: %v", w.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABackendThatCannotBeListedIsSaidAndAskedAgain(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
w.a.invErr = errors.New("connection refused")
|
||||
holding(w, 1)
|
||||
if !strings.Contains(strings.Join(w.said, "\n"), "cannot list what the backend holds") {
|
||||
t.Fatal(w.said)
|
||||
}
|
||||
w.a.invErr = nil
|
||||
w.a.inv = Inventory{Active: []string{"c1", "orphan"}}
|
||||
w.pass()
|
||||
w.settle()
|
||||
if strings.Join(w.a.removed, ",") != "orphan" {
|
||||
t.Fatalf("%v", w.a.removed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheToolsAnswer(t *testing.T) {
|
||||
w, _ := standingWorld(t)
|
||||
holding(w, 4)
|
||||
w.give()
|
||||
w.settle()
|
||||
tools := map[string]func(map[string]any) (any, error){}
|
||||
for _, tool := range RetirementTools(w.h) {
|
||||
tools[tool.Name] = tool.Run
|
||||
}
|
||||
if len(tools) != 4 {
|
||||
t.Fatalf("%d tools", len(tools))
|
||||
}
|
||||
got, err := tools["provisioner_retirement"](nil)
|
||||
if err != nil || got.(map[string]any)["waiting"] == nil {
|
||||
t.Fatal(got, err)
|
||||
}
|
||||
set := []any{"c1", "c2", "c3", "c4"}
|
||||
if _, err := tools["provisioner_retire_approve"](map[string]any{"consumers": set}); err == nil {
|
||||
t.Fatal("approved without a why")
|
||||
}
|
||||
if _, err := tools["provisioner_retire_approve"](map[string]any{"consumers": set, "why": "gone", "by": "operator"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tools["provisioner_delete"](map[string]any{"consumer": "c1", "why": "gone"}); err == nil {
|
||||
t.Fatal("deleted without confirm")
|
||||
}
|
||||
if _, err := tools["provisioner_delete"](map[string]any{"consumer": "c1", "confirm": "c1", "why": "gone"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Join(w.a.deleted, ",") != "c1" {
|
||||
t.Fatal(w.a.deleted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFailingConsumerRetiredIsSaidRecovered(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
given := holding(w, 2)
|
||||
w.a.held = false
|
||||
w.a.failing = errors.New("boom")
|
||||
w.passes(7 * time.Minute)
|
||||
w.give(given[0])
|
||||
w.settle()
|
||||
recovered := false
|
||||
for _, a := range *said {
|
||||
if a.event == EventRecovered && a.body["consumer"] == "c2" && a.body["why"] == "retired" {
|
||||
recovered = true
|
||||
}
|
||||
}
|
||||
if !recovered {
|
||||
t.Fatalf("%v", *said)
|
||||
}
|
||||
}
|
||||
|
||||
// everyWord is a world that keeps every word the provider announces.
|
||||
func everyWord(t *testing.T) (*world, *[]announced) {
|
||||
w := newWorld(t)
|
||||
var said []announced
|
||||
w.h.Announce = func(e string, b map[string]any) { said = append(said, announced{e, b}) }
|
||||
w.h.Node = "anchor"
|
||||
return w, &said
|
||||
}
|
||||
|
||||
// A restarted provider that finds every consumer it holds asked for says once that nothing waits, so the controller
|
||||
// clears a waiting condition an earlier process raised and this one never knew (issue 937); it says it no more.
|
||||
func TestARestartedProviderSaysOnceThatNothingWaits(t *testing.T) {
|
||||
w, said := everyWord(t)
|
||||
holding(w, 2)
|
||||
w.pass()
|
||||
w.pass()
|
||||
var words []announced
|
||||
for _, a := range *said {
|
||||
if a.event == EventRetirement {
|
||||
words = append(words, a)
|
||||
}
|
||||
}
|
||||
if len(words) != 1 || !noneWaits(words[0]) || words[0].body["provider-node"] != w.h.Node {
|
||||
t.Fatalf("said %+v, want one settled with no set", words)
|
||||
}
|
||||
}
|
||||
|
||||
// A provider whose own set waits does not say that nothing waits.
|
||||
func TestAProviderThatWaitsDoesNotSayNothingWaits(t *testing.T) {
|
||||
w, said := everyWord(t)
|
||||
w.a.inv.Active = []string{"c1", "c2", "c3", "c4", "c5"}
|
||||
w.give(map[string]any{"as": "c1", "node": "anchor"})
|
||||
w.settle()
|
||||
for _, a := range *said {
|
||||
if a.event == EventRetirement && noneWaits(a) {
|
||||
t.Fatalf("said nothing waits while %v waited", w.h.r.waiting)
|
||||
}
|
||||
}
|
||||
if w.h.r.waiting == nil {
|
||||
t.Fatal("nothing waited: the test does not hold the case")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package harness
|
||||
|
||||
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224): not on the first
|
||||
// failure, which may be a restart; after FailingAfter of failures with no success between; again
|
||||
// every SayAgainEvery while it lasts; and recovered on the first success, or when the consumer goes.
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type announced struct {
|
||||
event string
|
||||
body map[string]any
|
||||
}
|
||||
|
||||
func standingWorld(t *testing.T) (*world, *[]announced) {
|
||||
w := newWorld(t)
|
||||
var said []announced
|
||||
w.h.Announce = func(e string, b map[string]any) {
|
||||
// The first success since start, and the first word that nothing waits (issue 937), are their own
|
||||
// tests'; every other test reads past them.
|
||||
if c, _ := b["consumers"].([]map[string]any); b["why"] != "first-success" &&
|
||||
!(e == EventRetirement && b["change"] == ChangeSettled && len(c) == 0) {
|
||||
said = append(said, announced{e, b})
|
||||
}
|
||||
}
|
||||
w.h.Node = "anchor"
|
||||
return w, &said
|
||||
}
|
||||
|
||||
// passes reconciles every five seconds for d, as Run would.
|
||||
func (w *world) passes(d time.Duration) {
|
||||
for end := w.now.Add(d); w.now.Before(end); w.now = w.now.Add(5 * time.Second) {
|
||||
w.h.Reconcile(ctx)
|
||||
}
|
||||
}
|
||||
|
||||
func events(said []announced) string {
|
||||
var out []string
|
||||
for _, a := range said {
|
||||
out = append(out, a.event)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
func TestAConsumerFailedForMinutesIsAnnouncedNamingItAndTheClass(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
w.a.failing = errors.New(`token request failed: 401 {"error":"invalid_grant","error_description":"Invalid user credentials"}`)
|
||||
w.give(map[string]any{"as": "mesh_home_grafana", "node": "home-server"})
|
||||
|
||||
w.passes(4 * time.Minute)
|
||||
if len(*said) != 0 {
|
||||
t.Fatalf("announced before FailingAfter: %v", events(*said))
|
||||
}
|
||||
w.passes(2 * time.Minute)
|
||||
if events(*said) != EventFailing {
|
||||
t.Fatalf("want one %s, got %q", EventFailing, events(*said))
|
||||
}
|
||||
b := (*said)[0].body
|
||||
if b["consumer"] != "mesh_home_grafana" || b["node"] != "home-server" || b["class"] != ClassCredentials ||
|
||||
b["provider"] != "postgres-database" || b["provider-node"] != "anchor" || b["attempts"].(int) < 60 {
|
||||
t.Fatalf("%v", b)
|
||||
}
|
||||
|
||||
// Said again while it lasts, not every pass.
|
||||
w.passes(14 * time.Minute)
|
||||
if events(*said) != EventFailing {
|
||||
t.Fatalf("repeated too soon: %q", events(*said))
|
||||
}
|
||||
w.passes(2 * time.Minute)
|
||||
if events(*said) != EventFailing+","+EventFailing {
|
||||
t.Fatalf("not repeated: %q", events(*said))
|
||||
}
|
||||
|
||||
// The first success takes it back.
|
||||
w.a.failing = nil
|
||||
w.passes(5 * time.Second)
|
||||
if events(*said) != EventFailing+","+EventFailing+","+EventRecovered {
|
||||
t.Fatalf("no recovery: %q", events(*said))
|
||||
}
|
||||
if (*said)[2].body["consumer"] != "mesh_home_grafana" {
|
||||
t.Fatal((*said)[2].body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOneSuccessBetweenFailuresStartsTheRunAgain(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
w.a.failing = errors.New("connection refused")
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.passes(4 * time.Minute)
|
||||
w.a.failing = nil
|
||||
w.passes(5 * time.Second)
|
||||
w.give(map[string]any{"as": "a", "values": map[string]any{"name": "changed"}})
|
||||
w.a.failing = errors.New("connection refused")
|
||||
w.passes(4 * time.Minute)
|
||||
if len(*said) != 0 {
|
||||
t.Fatalf("two runs of four minutes are not one of eight: %q", events(*said))
|
||||
}
|
||||
}
|
||||
|
||||
// The check that failed for a day on 2026-10-05: clients already made, every minute's check refused
|
||||
// at the token. A check that cannot be asked is a failure too.
|
||||
func TestACheckThatKeepsFailingIsAFailureToo(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.h.Reconcile(ctx)
|
||||
w.a.holdsErr = errors.New("401 invalid_grant")
|
||||
w.passes(7 * time.Minute)
|
||||
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassCredentials {
|
||||
t.Fatalf("%q %v", events(*said), *said)
|
||||
}
|
||||
w.a.holdsErr = nil
|
||||
w.passes(time.Minute + 5*time.Second)
|
||||
if events(*said) != EventFailing+","+EventRecovered {
|
||||
t.Fatalf("%q", events(*said))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnreadableSecretIsAnnouncedAsSuch(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
w.give(map[string]any{"as": "a"})
|
||||
os.Remove(w.dir + "/a.secret")
|
||||
w.passes(6 * time.Minute)
|
||||
if events(*said) != EventFailing || (*said)[0].body["class"] != ClassSecret {
|
||||
t.Fatalf("%q %v", events(*said), *said)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAConsumerNoLongerAskedForIsNoLongerFailing(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
w.a.failing = errors.New("boom")
|
||||
w.give(map[string]any{"as": "a"}, map[string]any{"as": "b"})
|
||||
w.passes(6 * time.Minute)
|
||||
if events(*said) != EventFailing+","+EventFailing {
|
||||
t.Fatalf("%q", events(*said))
|
||||
}
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.passes(5 * time.Second)
|
||||
last := (*said)[len(*said)-1]
|
||||
if last.event != EventRecovered || last.body["consumer"] != "b" || last.body["why"] != "no longer asked for" {
|
||||
t.Fatalf("%v", *said)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnErrorIsClassedByItsWords(t *testing.T) {
|
||||
for text, want := range map[string]string{
|
||||
`Keycloak token request failed: 401 {"error":"invalid_grant"}`: ClassCredentials,
|
||||
`FATAL: password authentication failed for user "postgres"`: ClassCredentials,
|
||||
`dial tcp 127.0.0.1:5432: connect: connection refused`: ClassUnreachable,
|
||||
`context deadline exceeded`: ClassUnreachable,
|
||||
`extension "nope" is not available`: ClassRefused,
|
||||
} {
|
||||
if got := ClassOf(text); got != want {
|
||||
t.Errorf("%s: %s, want %s", text, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdapterThatClassesItsOwnErrorsIsBelieved(t *testing.T) {
|
||||
w, said := standingWorld(t)
|
||||
w.h.Adapter = classing{w.a}
|
||||
w.a.failing = errors.New("anything")
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.passes(6 * time.Minute)
|
||||
if (*said)[0].body["class"] != "its-own" {
|
||||
t.Fatal((*said)[0].body)
|
||||
}
|
||||
}
|
||||
|
||||
type classing struct{ *recorder }
|
||||
|
||||
func (classing) Class(error) string { return "its-own" }
|
||||
|
||||
// A provider restarted after announcing a failure has forgotten it; its first success for each
|
||||
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
|
||||
func TestTheFirstSuccessSinceStartIsAnnouncedOnce(t *testing.T) {
|
||||
w := newWorld(t)
|
||||
var said []announced
|
||||
w.h.Announce = func(e string, b map[string]any) {
|
||||
if e != EventRetirement {
|
||||
said = append(said, announced{e, b})
|
||||
}
|
||||
}
|
||||
w.give(map[string]any{"as": "a"})
|
||||
w.passes(3 * time.Minute)
|
||||
if events(said) != EventRecovered || said[0].body["why"] != "first-success" || said[0].body["consumer"] != "a" {
|
||||
t.Fatalf("%v", said)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,769 @@
|
||||
// Package provider is a providing module's half of a binding: it answers the provider's consumers itself, over the
|
||||
// bus (novox/hq ADR 0349 clause 12, issue 678): the provider module, not the controller, makes each bound
|
||||
// consumer's entry.
|
||||
//
|
||||
// **Who binds the provider.** Each node's provisioner client records the needs of the modules handed to it from
|
||||
// their plans, and the controller names in each plan, per required provision and local name, the provider that
|
||||
// resolution chose: `provisioning.provisions` in `<node>.<module>` of the assignments mirror (issue 760). A recorded
|
||||
// need names no provider (issue 677's note 7), so the plan's entry is where a bind is seen, as the vault sees it
|
||||
// (mesh-vault pair.go). A consumer binds this provider when an entry names this machine and this module, and
|
||||
// unbinds when the entry, or the assignment, goes. A consumer still filled the old way has no plan and is not seen
|
||||
// here; the controller answers it as before (ADR 0349 clause 17).
|
||||
//
|
||||
// **One entry per consumer, in the provider's own bucket.** For each bind the helper composes the consumer's
|
||||
// served values from the provider's offer (its config already composed into it, ADR 0202 §7) and the consumer's
|
||||
// identity (`${consumer:as}`, `${consumer:as:dns}`, ADR 0202 §1), and writes one entry, `<node>.<module>.
|
||||
// <provision>[.<local>]` of `<provider module>_bound`: the one statement both ends read. It never carries a
|
||||
// secret. It names the credential the vault keeps for the pair, which the consumer's client asks the vault for.
|
||||
//
|
||||
// **The credential is the vault's** (ADR 0349 clause 8). The helper mints nothing and reads nothing of the
|
||||
// provider's own secrets: it reads the value from the vault and hands it to the provider's adapter, which makes the
|
||||
// login its software keeps. An offer whose software holds one credential (ADR 0158) is the vault's one value per
|
||||
// provider and provision, `provision/<provider node>/<provision>`, with the provider's one user name; an offer with
|
||||
// a login per consumer (ADR 0048, ADR 0158 §11, the default) is each consumer's own value,
|
||||
// `pair/<provider node>/<provision>/<node>/<module>[/<local>]`, with the consumer's identity as its login (issue
|
||||
// 677's notes 12 and 13). A credential the vault does not hold yet is waited for: nothing is made and no entry is
|
||||
// written, so a consumer is never told of a login its provider has not made. A provider in the tool runner never asks
|
||||
// the vault itself: its Vault is ConsumersFile, the file its machine's provisioner client fills (ADR 0373).
|
||||
//
|
||||
// **A multi-holder consumer gets no derived value** (ADR 0202 §10 as ADR 0349 clause 12 changes it): a consumer
|
||||
// bound under a local name keeps several holders of the provision, each with its own login, while what the
|
||||
// provider derives names one; its entry refuses it, naming both ends, and its client reports it.
|
||||
//
|
||||
// The helper speaks to the bus and the vault through the two interfaces below, so a provider wires it to its
|
||||
// runtime and a test to a stand-in. Standard library only.
|
||||
package provider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// AssignmentsBucket is the controller's assignments mirror, `<node>.<module>`, whose value carries the plan of a
|
||||
// module handed to the provisioner.
|
||||
const AssignmentsBucket = "mesh-controller_assignments"
|
||||
|
||||
// BucketOf is a providing module's own bucket of entries, one per consumer: `<module>_bound`, the module's state
|
||||
// `bound`.
|
||||
func BucketOf(module string) string { return module + "_bound" }
|
||||
|
||||
// Bus is what the helper needs of the bus: a key-value watch and the provider's writes to its own bucket.
|
||||
type Bus interface {
|
||||
// Watch hands handle every current key and value of bucket, then each change as it comes, a removed key with a
|
||||
// nil value; it returns once the current values are handed, and the watch lasts until ctx ends.
|
||||
Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error
|
||||
Put(ctx context.Context, bucket, key string, value []byte) error
|
||||
Delete(ctx context.Context, bucket, key string) error
|
||||
}
|
||||
|
||||
// ConditionalBus is a Bus that can also delete a key only while it still holds what the caller read. The helper uses
|
||||
// it when the Bus offers it, so a pass that lets a consumer go never removes an entry another instance of the
|
||||
// provider wrote after the pass read it; a Bus without it deletes unconditionally. DeleteIf returns an error that
|
||||
// is stdio.ErrChanged when the key holds something else now.
|
||||
type ConditionalBus interface {
|
||||
Bus
|
||||
DeleteIf(ctx context.Context, bucket, key string, held []byte) error
|
||||
}
|
||||
|
||||
// Vault is how the provider reads a credential the vault keeps, by its name, opened.
|
||||
type Vault interface {
|
||||
Value(ctx context.Context, name string) ([]byte, error)
|
||||
}
|
||||
|
||||
// CredentialForm is which credential an offer's consumers present.
|
||||
type CredentialForm int
|
||||
|
||||
const (
|
||||
// PerConsumer is a login of each consumer's own (ADR 0048, ADR 0158 §11): the default.
|
||||
PerConsumer CredentialForm = iota
|
||||
// Shared is the one credential the provider's software holds, shared with every consumer (ADR 0158).
|
||||
Shared
|
||||
// NoCredential is an offer whose consumers present none.
|
||||
NoCredential
|
||||
)
|
||||
|
||||
// Offer is what the provider serves one provision with.
|
||||
type Offer struct {
|
||||
// Serves are the values served to each consumer, the provider's config already composed into them; a string
|
||||
// may name the consumer's identity, `${consumer:as}` or `${consumer:as:dns}`.
|
||||
Serves map[string]any
|
||||
// Credential is the form of the credential its consumers present. A consumer's plan entry that names a
|
||||
// credential form of its own wins over it (the plan's `credential-form`).
|
||||
Credential CredentialForm
|
||||
// User is the provider's one user name, for a Shared credential.
|
||||
User string
|
||||
}
|
||||
|
||||
// Entry is one consumer's entry in the provider's bucket: the one statement both ends read. Never a secret.
|
||||
type Entry struct {
|
||||
Provider string `json:"provider"`
|
||||
ProviderModule string `json:"provider-module"`
|
||||
Provision string `json:"provision"`
|
||||
Consumer string `json:"consumer"`
|
||||
Module string `json:"module"`
|
||||
Local string `json:"local,omitempty"`
|
||||
// As is the consumer's identity, the mesh's (ADR 0202 §1).
|
||||
As string `json:"as,omitempty"`
|
||||
// User is the login the consumer presents: its identity, or the provider's one user name for a shared credential.
|
||||
User string `json:"user,omitempty"`
|
||||
// Values are the served values, composed for this consumer: what `${bound:<provision>:<key>}` reads.
|
||||
Values map[string]any `json:"values,omitempty"`
|
||||
// VaultName is the name under which the vault keeps this consumer's credential, which the consumer's client asks
|
||||
// the vault for: a name, never the credential. Its key is not "credential" because the runtime's state guard
|
||||
// refuses any key ending in that word, whatever the value (issue 945).
|
||||
VaultName string `json:"vault-name,omitempty"`
|
||||
// Refused, when set, is why the consumer is not served, naming both ends.
|
||||
Refused string `json:"refused,omitempty"`
|
||||
// Contributes is what the consumer's module tells its provider about this provision (a database's name, its
|
||||
// extensions), from its plan's entry (issue 776). Handed to Make and Gone; never written to the bucket.
|
||||
Contributes map[string]any `json:"-"`
|
||||
// At and From are where the consumer's plan says the provider is and which machine it is (issue 788): the values
|
||||
// today's binding file gives that consumer. Written into Values as `at` and `from`; never alone to the bucket.
|
||||
At, From string `json:"-"`
|
||||
// Names are the names the controller composed for this consumer on this provision (issue 792): its own name, its
|
||||
// internal name, and one of each per local name it contributes under, keyed as its binding file names them.
|
||||
// Written into Values over anything the provider serves under the key; never alone to the bucket.
|
||||
Names map[string]string `json:"-"`
|
||||
// CredentialForm and Name are from the consumer's plan entry (issue 825): the form of the credential the
|
||||
// controller chose for this bind (`shared`, `pair`, `issued` or `given`) and the holder's name. When set, they
|
||||
// win over the offer's form. Never written to the bucket.
|
||||
CredentialForm, Name string `json:"-"`
|
||||
}
|
||||
|
||||
// Key is the entry's key in the provider's bucket: `<node>.<module>.<provision>`, and `.<local>` for a bind under a
|
||||
// local name.
|
||||
func (e Entry) Key() string {
|
||||
k := e.Consumer + "." + e.Module + "." + e.Provision
|
||||
if e.Local != "" {
|
||||
k += "." + e.Local
|
||||
}
|
||||
return k
|
||||
}
|
||||
|
||||
// Consumer is what the provider's adapter makes: the consumer's entry and the credential the vault keeps for it.
|
||||
type Consumer struct {
|
||||
Entry Entry
|
||||
Credential []byte
|
||||
}
|
||||
|
||||
// Options are the provider's.
|
||||
type Options struct {
|
||||
// Node and Module are this provider: the machine it runs on and the module serving.
|
||||
Node, Module string
|
||||
// Offers are the provisions it serves, by name.
|
||||
Offers map[string]Offer
|
||||
Bus Bus
|
||||
Vault Vault
|
||||
// Make brings a consumer into being in the provider's software under the vault's credential. Asked on every
|
||||
// pass for every consumer answered, so it must leave one that is already as asked alone; a rotated credential
|
||||
// reaches the software this way.
|
||||
Make func(ctx context.Context, c Consumer) error
|
||||
// Gone, when set, hears that a consumer no longer binds the provider, before its entry is removed; an error
|
||||
// keeps the entry, and it is heard again on the next pass. Retiring what was kept for it is the provider's
|
||||
// (ADR 0230).
|
||||
Gone func(ctx context.Context, e Entry) error
|
||||
// Identity is the consumer's identity for an entry whose plan names none; ConsumerIdentity of the node and the
|
||||
// module when not set.
|
||||
Identity func(node, module string) string
|
||||
// Every is how often a pass runs without a change watched; a minute when not set.
|
||||
Every time.Duration
|
||||
// Logf says what a pass did that differs from the last; the standard log when not set.
|
||||
Logf func(format string, args ...any)
|
||||
}
|
||||
|
||||
// Provider answers its consumers.
|
||||
type Provider struct {
|
||||
opts Options
|
||||
|
||||
mu sync.Mutex
|
||||
assignments map[string][]byte
|
||||
entries map[string][]byte
|
||||
said map[string]string
|
||||
poke chan struct{}
|
||||
}
|
||||
|
||||
// New is a provider answering its consumers as opts say.
|
||||
func New(opts Options) *Provider {
|
||||
if opts.Identity == nil {
|
||||
opts.Identity = ConsumerIdentity
|
||||
}
|
||||
if opts.Every <= 0 {
|
||||
opts.Every = time.Minute
|
||||
}
|
||||
if opts.Logf == nil {
|
||||
opts.Logf = func(f string, a ...any) { fmt.Printf("[provider] "+f+"\n", a...) }
|
||||
}
|
||||
return &Provider{opts: opts, assignments: map[string][]byte{}, entries: map[string][]byte{},
|
||||
said: map[string]string{}, poke: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
// What a pass did for one bind.
|
||||
const (
|
||||
Answered = "answered"
|
||||
Refused = "refused"
|
||||
Waiting = "waiting"
|
||||
Failed = "failed"
|
||||
Removed = "removed"
|
||||
// Superseded is an entry no bind asks for any more that another instance of the provider wrote after this pass
|
||||
// read it: it is left standing.
|
||||
Superseded = "superseded"
|
||||
)
|
||||
|
||||
// Outcome is what a pass did for one consumer's entry.
|
||||
type Outcome struct {
|
||||
Key string `json:"key"`
|
||||
Status string `json:"status"`
|
||||
Why string `json:"why,omitempty"`
|
||||
}
|
||||
|
||||
// Run watches the binds and the provider's own entries, and passes once at start, on every change and every
|
||||
// Every, until ctx ends.
|
||||
func (p *Provider) Run(ctx context.Context) error {
|
||||
if err := p.watch(ctx, BucketOf(p.opts.Module), p.entries); err != nil {
|
||||
return fmt.Errorf("watching %s: %w", BucketOf(p.opts.Module), err)
|
||||
}
|
||||
if err := p.watch(ctx, AssignmentsBucket, p.assignments); err != nil {
|
||||
return fmt.Errorf("watching %s: %w", AssignmentsBucket, err)
|
||||
}
|
||||
tick := time.NewTicker(p.opts.Every)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
p.mu.Lock()
|
||||
assignments, entries := clone(p.assignments), clone(p.entries)
|
||||
p.mu.Unlock()
|
||||
if _, err := p.pass(ctx, assignments, entries); err != nil {
|
||||
p.say("pass", err.Error())
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-p.poke:
|
||||
case <-tick.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Provider) watch(ctx context.Context, bucket string, into map[string][]byte) error {
|
||||
return p.opts.Bus.Watch(ctx, bucket, func(key string, value []byte) {
|
||||
p.mu.Lock()
|
||||
if value == nil {
|
||||
delete(into, key)
|
||||
} else {
|
||||
into[key] = append([]byte(nil), value...)
|
||||
}
|
||||
p.mu.Unlock()
|
||||
if bucket == AssignmentsBucket {
|
||||
select {
|
||||
case p.poke <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Pass reads the binds and the provider's entries from the bus and answers once: each bind of this provider gets
|
||||
// its entry, and each entry of this provider no bind asks for is removed. An assignment that cannot be read, or that
|
||||
// is not the one its key names, is skipped and said (an outcome of status Failed under the assignment's key, and a
|
||||
// line of the log), and the entries of its consumer are left as they are, since a bind missed would read as an
|
||||
// unbind; the pass goes on for the rest. Run passes by itself; Pass is for a provider that passes when it chooses.
|
||||
//
|
||||
// Over StateBus each call to Pass leaves two watches open for the life of the process, silenced once the call has
|
||||
// ended (see StateBus): a provider that passes often should use Run, which watches once.
|
||||
func (p *Provider) Pass(ctx context.Context) ([]Outcome, error) {
|
||||
watchCtx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
var mu sync.Mutex
|
||||
read := func(bucket string) (map[string][]byte, error) {
|
||||
got := map[string][]byte{}
|
||||
err := p.opts.Bus.Watch(watchCtx, bucket, func(key string, value []byte) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if value == nil {
|
||||
delete(got, key)
|
||||
} else {
|
||||
got[key] = append([]byte(nil), value...)
|
||||
}
|
||||
})
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return clone(got), err
|
||||
}
|
||||
entries, err := read(BucketOf(p.opts.Module))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading %s: %w", BucketOf(p.opts.Module), err)
|
||||
}
|
||||
assignments, err := read(AssignmentsBucket)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading %s: %w", AssignmentsBucket, err)
|
||||
}
|
||||
return p.pass(ctx, assignments, entries)
|
||||
}
|
||||
|
||||
// planned is as much of an assignment as the helper reads: lenient, since the plan is the client's to act on.
|
||||
type planned struct {
|
||||
Node string `json:"node"`
|
||||
Module string `json:"module"`
|
||||
Provisioning *struct {
|
||||
Provisions []struct {
|
||||
Provision string `json:"provision"`
|
||||
Local string `json:"local"`
|
||||
Provider string `json:"provider"`
|
||||
ProviderModule string `json:"provider-module"`
|
||||
Contributes map[string]any `json:"contributes"`
|
||||
At string `json:"at"`
|
||||
From string `json:"from"`
|
||||
As string `json:"as"`
|
||||
Names map[string]string `json:"names"`
|
||||
CredentialForm string `json:"credential-form"`
|
||||
Name string `json:"name"`
|
||||
} `json:"provisions"`
|
||||
} `json:"provisioning"`
|
||||
}
|
||||
|
||||
func (p *Provider) pass(ctx context.Context, assignments, entries map[string][]byte) ([]Outcome, error) {
|
||||
var binds []Entry
|
||||
var out []Outcome
|
||||
unknown := map[string]bool{} // consumers whose assignment was skipped: their entries are not judged
|
||||
for _, key := range sortedKeys(assignments) {
|
||||
if !Standing(assignments[key]) {
|
||||
continue // ended or pending: read as a removed key (issue 1098)
|
||||
}
|
||||
var a planned
|
||||
if err := json.Unmarshal(assignments[key], &a); err != nil || a.Node+"."+a.Module != key {
|
||||
o := Outcome{Key: key, Status: Failed, Why: fmt.Sprintf("the assignment %s is not the controller's, so "+
|
||||
"who it binds is not known; it is skipped, and its entries are left as they are", key)}
|
||||
out = append(out, o)
|
||||
p.say(key, o.Status+" "+o.Why)
|
||||
unknown[key] = true
|
||||
continue
|
||||
}
|
||||
if a.Provisioning == nil {
|
||||
continue
|
||||
}
|
||||
for _, b := range a.Provisioning.Provisions {
|
||||
if b.Provider != p.opts.Node || b.ProviderModule != p.opts.Module {
|
||||
continue
|
||||
}
|
||||
binds = append(binds, Entry{Provider: p.opts.Node, ProviderModule: p.opts.Module,
|
||||
Provision: b.Provision, Consumer: a.Node, Module: a.Module, Local: b.Local,
|
||||
Contributes: b.Contributes, At: b.At, From: b.From, As: b.As, Names: b.Names,
|
||||
CredentialForm: b.CredentialForm, Name: b.Name})
|
||||
}
|
||||
}
|
||||
|
||||
want := map[string]bool{}
|
||||
for _, b := range binds {
|
||||
key := b.Key()
|
||||
want[key] = true
|
||||
o := p.answer(ctx, b, entries[key])
|
||||
out = append(out, o)
|
||||
p.say(key, o.Status+" "+o.Why)
|
||||
}
|
||||
for _, key := range sortedKeys(entries) {
|
||||
if want[key] || unknown[consumerOf(key)] {
|
||||
continue
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(entries[key], &e); err != nil || e.Provider != p.opts.Node ||
|
||||
e.ProviderModule != p.opts.Module {
|
||||
continue // another machine's instance of this provider wrote it, or nothing of this helper's
|
||||
}
|
||||
if p.opts.Gone != nil {
|
||||
if err := p.opts.Gone(ctx, e); err != nil {
|
||||
o := Outcome{Key: key, Status: Failed, Why: "no longer bound, and the provider could not let it go: " +
|
||||
err.Error()}
|
||||
out = append(out, o)
|
||||
p.say(key, o.Status+" "+o.Why)
|
||||
continue
|
||||
}
|
||||
}
|
||||
if err := p.remove(ctx, key, entries[key]); err != nil {
|
||||
if errors.Is(err, stdio.ErrChanged) {
|
||||
o := Outcome{Key: key, Status: Superseded, Why: "another instance of the provider wrote its entry " +
|
||||
"since this pass read it, so it stands"}
|
||||
out = append(out, o)
|
||||
p.say(key, o.Status+" "+o.Why)
|
||||
continue
|
||||
}
|
||||
o := Outcome{Key: key, Status: Failed, Why: "no longer bound, and its entry could not be removed: " +
|
||||
err.Error()}
|
||||
out = append(out, o)
|
||||
p.say(key, o.Status+" "+o.Why)
|
||||
continue
|
||||
}
|
||||
out = append(out, Outcome{Key: key, Status: Removed})
|
||||
p.say(key, Removed)
|
||||
p.mu.Lock()
|
||||
delete(p.said, key)
|
||||
p.mu.Unlock()
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// consumerOf is the `<node>.<module>` an entry's key starts with: its first two parts.
|
||||
func consumerOf(key string) string {
|
||||
parts := strings.SplitN(key, ".", 3)
|
||||
if len(parts) < 3 {
|
||||
return key
|
||||
}
|
||||
return parts[0] + "." + parts[1]
|
||||
}
|
||||
|
||||
// remove deletes an entry this instance read as held, and only while it is still as read when the Bus can say so.
|
||||
func (p *Provider) remove(ctx context.Context, key string, held []byte) error {
|
||||
if c, ok := p.opts.Bus.(ConditionalBus); ok {
|
||||
return c.DeleteIf(ctx, BucketOf(p.opts.Module), key, held)
|
||||
}
|
||||
return p.opts.Bus.Delete(ctx, BucketOf(p.opts.Module), key)
|
||||
}
|
||||
|
||||
// answer makes one bind's entry and writes it when it differs from what the bucket holds.
|
||||
func (p *Provider) answer(ctx context.Context, e Entry, have []byte) Outcome {
|
||||
key := e.Key()
|
||||
offer, offered := p.opts.Offers[e.Provision]
|
||||
switch {
|
||||
case !offered:
|
||||
e.Refused = fmt.Sprintf("%s on %s is bound to %s on %s for %s, which %s does not serve", e.Module,
|
||||
e.Consumer, e.ProviderModule, e.Provider, e.Provision, e.ProviderModule)
|
||||
case e.Local != "" && derives(offer.Serves):
|
||||
e.Refused = fmt.Sprintf("%s on %s keeps several holders of %s (this one is %q), and %s on %s derives %s for "+
|
||||
"each consumer from its login: each holder has a login of its own and a consumer is told one value per "+
|
||||
"requirement, so the two ends would name different things (novox/hq ADR 0202 §10, ADR 0349 clause 12)",
|
||||
e.Module, e.Consumer, e.Provision, e.Local, e.ProviderModule, e.Provider,
|
||||
strings.Join(derivedKeys(offer.Serves), ", "))
|
||||
}
|
||||
if e.Refused != "" {
|
||||
return p.write(ctx, e, have, Outcome{Key: key, Status: Refused, Why: e.Refused})
|
||||
}
|
||||
|
||||
// The plan's identity is the controller's, slug first; Identity only answers for an entry without one.
|
||||
if e.As == "" {
|
||||
e.As = p.opts.Identity(e.Consumer, e.Module)
|
||||
}
|
||||
values, err := ServedTo(offer.Serves, e.As)
|
||||
if err != nil {
|
||||
e.Refused = fmt.Sprintf("%s on %s serving %s to %s on %s: %v", e.ProviderModule, e.Provider, e.Provision,
|
||||
e.Module, e.Consumer, err)
|
||||
e.As = ""
|
||||
return p.write(ctx, e, have, Outcome{Key: key, Status: Refused, Why: e.Refused})
|
||||
}
|
||||
if e.At != "" || e.From != "" {
|
||||
if values == nil {
|
||||
values = map[string]any{}
|
||||
}
|
||||
if e.At != "" {
|
||||
values["at"] = e.At
|
||||
}
|
||||
if e.From != "" {
|
||||
values["from"] = e.From
|
||||
}
|
||||
}
|
||||
if len(e.Names) > 0 {
|
||||
if values == nil {
|
||||
values = map[string]any{}
|
||||
}
|
||||
for key, name := range e.Names {
|
||||
values[key] = name
|
||||
}
|
||||
}
|
||||
e.Values = values
|
||||
var credential []byte
|
||||
// The plan's form wins; the offer's is the default.
|
||||
// An offer that declares no credential has none whatever the plan says: the controller writes "pair" for it.
|
||||
form := e.CredentialForm
|
||||
if offer.Credential == NoCredential {
|
||||
form = ""
|
||||
} else if form == "" {
|
||||
form = map[CredentialForm]string{Shared: "shared", PerConsumer: "pair"}[offer.Credential]
|
||||
}
|
||||
switch form {
|
||||
case "shared":
|
||||
e.User = offer.User
|
||||
e.VaultName = SharedName(e.Provider, e.Provision)
|
||||
case "pair":
|
||||
e.User = e.As
|
||||
e.VaultName = PairName(e.Provider, e.Provision, e.Consumer, e.Module, e.Local)
|
||||
case "issued", "given":
|
||||
// A per-holder value, as mesh-vault names it (perholder.go).
|
||||
e.User = e.As
|
||||
e.VaultName = HolderName(e.Provider, e.Provision, e.Consumer, e.Module)
|
||||
}
|
||||
if e.VaultName != "" {
|
||||
credential, err = p.opts.Vault.Value(ctx, e.VaultName)
|
||||
if err != nil {
|
||||
return Outcome{Key: key, Status: Waiting, Why: fmt.Sprintf("the vault's %s: %v", e.VaultName, err)}
|
||||
}
|
||||
}
|
||||
if p.opts.Make != nil {
|
||||
if err := p.opts.Make(ctx, Consumer{Entry: e, Credential: credential}); err != nil {
|
||||
why := err.Error()
|
||||
why = redact(why, credential)
|
||||
return Outcome{Key: key, Status: Failed, Why: "the provider could not make it: " + why}
|
||||
}
|
||||
}
|
||||
return p.write(ctx, e, have, Outcome{Key: key, Status: Answered})
|
||||
}
|
||||
|
||||
// redact takes a credential out of text in the forms it can come back in: as it is, in base64 (standard and URL,
|
||||
// padded and not) and as a JSON string says it (escapes included).
|
||||
func redact(text string, credential []byte) string {
|
||||
if len(credential) == 0 {
|
||||
return text
|
||||
}
|
||||
quoted, _ := json.Marshal(string(credential))
|
||||
forms := []string{string(credential), base64.StdEncoding.EncodeToString(credential),
|
||||
base64.RawStdEncoding.EncodeToString(credential), base64.URLEncoding.EncodeToString(credential),
|
||||
base64.RawURLEncoding.EncodeToString(credential), string(quoted[1 : len(quoted)-1])}
|
||||
sort.SliceStable(forms, func(i, j int) bool { return len(forms[i]) > len(forms[j]) })
|
||||
for _, f := range forms {
|
||||
text = strings.ReplaceAll(text, f, "<credential>")
|
||||
}
|
||||
return text
|
||||
}
|
||||
|
||||
func (p *Provider) write(ctx context.Context, e Entry, have []byte, o Outcome) Outcome {
|
||||
body, err := json.Marshal(e)
|
||||
if err != nil {
|
||||
return Outcome{Key: o.Key, Status: Failed, Why: err.Error()}
|
||||
}
|
||||
if sameJSON(have, body) {
|
||||
return o
|
||||
}
|
||||
if err := p.opts.Bus.Put(ctx, BucketOf(p.opts.Module), o.Key, body); err != nil {
|
||||
return Outcome{Key: o.Key, Status: Failed, Why: "its entry could not be written: " + err.Error()}
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// say logs what a key's pass said when it differs from the last.
|
||||
func (p *Provider) say(key, what string) {
|
||||
p.mu.Lock()
|
||||
same := p.said[key] == what
|
||||
p.said[key] = what
|
||||
p.mu.Unlock()
|
||||
if !same {
|
||||
p.opts.Logf("%s: %s", key, strings.TrimSpace(what))
|
||||
}
|
||||
}
|
||||
|
||||
// SharedName is the vault's one value per provider and provision (ADR 0158, issue 677).
|
||||
func SharedName(provider, provision string) string { return "provision/" + provider + "/" + provision }
|
||||
|
||||
// HolderName is the vault's value of one holder's own credential, issued or given for it (mesh-vault perholder.go).
|
||||
func HolderName(provider, provision, node, module string) string {
|
||||
return "provision/" + provider + "/" + provision + "/" + node + "/" + module
|
||||
}
|
||||
|
||||
// PairName is the vault's value of one consumer's own login (ADR 0048, ADR 0158 §11, issue 677's note 12).
|
||||
func PairName(provider, provision, node, module, local string) string {
|
||||
n := "pair/" + provider + "/" + provision + "/" + node + "/" + module
|
||||
if local != "" {
|
||||
n += "/" + local
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// identityUnusable is every character a consumer's identity does not hold (mesh-controller internal/catalogue
|
||||
// identity.go, the one derivation).
|
||||
var identityUnusable = regexp.MustCompile(`[^a-z0-9_]+`)
|
||||
|
||||
// ConsumerIdentity is what one module on one machine is called wherever it authenticates: `mesh_<node>_<module>`,
|
||||
// lower case, anything but letters, digits and `_` written `_` (mesh-controller ConsumerIdentity). module is the
|
||||
// identity source: the module's slug when it declares one, otherwise its name (ADR 0049).
|
||||
func ConsumerIdentity(node, module string) string {
|
||||
clean := func(s string) string {
|
||||
return strings.Trim(identityUnusable.ReplaceAllString(strings.ToLower(s), "_"), "_")
|
||||
}
|
||||
return "mesh_" + clean(node) + "_" + clean(module)
|
||||
}
|
||||
|
||||
// consumerFact is `${consumer:<fact>}` or `${consumer:<fact>:<alphabet>}`.
|
||||
var consumerFact = regexp.MustCompile(`\$\{consumer:([a-z][a-z0-9-]*)(?::([a-z][a-z0-9-]*))?\}`)
|
||||
|
||||
// ServedTo fills an offer's served values for one consumer whose identity is as (ADR 0202 §1): `${consumer:as}` as
|
||||
// it is, `${consumer:as:dns}` with `-` for `_`. Only strings carry placeholders; any other fact or alphabet is
|
||||
// refused, never left standing.
|
||||
func ServedTo(serves map[string]any, as string) (map[string]any, error) {
|
||||
if serves == nil {
|
||||
return nil, nil
|
||||
}
|
||||
out := make(map[string]any, len(serves))
|
||||
for _, key := range sortedKeys(serves) {
|
||||
text, ok := serves[key].(string)
|
||||
if !ok || !strings.Contains(text, "${consumer:") {
|
||||
out[key] = serves[key]
|
||||
continue
|
||||
}
|
||||
var failed error
|
||||
out[key] = consumerFact.ReplaceAllStringFunc(text, func(match string) string {
|
||||
parts := consumerFact.FindStringSubmatch(match)
|
||||
switch {
|
||||
case parts[1] == "as" && parts[2] == "":
|
||||
return as
|
||||
case parts[1] == "as" && parts[2] == "dns":
|
||||
return strings.ReplaceAll(as, "_", "-")
|
||||
}
|
||||
if failed == nil {
|
||||
failed = fmt.Errorf("the value served as %q says %s, and the mesh states only ${consumer:as} and "+
|
||||
"${consumer:as:dns}", key, match)
|
||||
}
|
||||
return match
|
||||
})
|
||||
if failed != nil {
|
||||
return nil, failed
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func derives(serves map[string]any) bool { return len(derivedKeys(serves)) > 0 }
|
||||
|
||||
func derivedKeys(serves map[string]any) []string {
|
||||
var out []string
|
||||
for _, k := range sortedKeys(serves) {
|
||||
if text, ok := serves[k].(string); ok && strings.Contains(text, "${consumer:") {
|
||||
out = append(out, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func clone(m map[string][]byte) map[string][]byte {
|
||||
out := make(map[string][]byte, len(m))
|
||||
for k, v := range m {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sameJSON(a, b []byte) bool {
|
||||
if a == nil || b == nil {
|
||||
return false
|
||||
}
|
||||
var x, y any
|
||||
if json.Unmarshal(a, &x) != nil || json.Unmarshal(b, &y) != nil {
|
||||
return false
|
||||
}
|
||||
ja, _ := json.Marshal(x)
|
||||
jb, _ := json.Marshal(y)
|
||||
return string(ja) == string(jb)
|
||||
}
|
||||
|
||||
func sortedKeys[V any](m map[string]V) []string {
|
||||
if len(m) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// StateBus is the Bus of a provider that runs as a module bundle over the runtime's stdio: the provider's own bucket,
|
||||
// BucketOf(module), is the module's state `bound`, and AssignmentsBucket is the state it reads,
|
||||
// `mesh-controller.assignments` (both declared in the module's manifest). Any other bucket is refused.
|
||||
//
|
||||
// A stdio state watch lasts as long as the process and cannot be ended (stdio.KeptState.Watch), so Watch honours the
|
||||
// Bus contract by silencing instead: once ctx has ended the handler is never called again, though the runtime keeps
|
||||
// sending changes that are dropped. A Pass, which watches twice per call, is therefore correct, only the dropped
|
||||
// changes of its two spent watches remain.
|
||||
func StateBus(module string) Bus { return stateBus{module: module} }
|
||||
|
||||
type stateBus struct{ module string }
|
||||
|
||||
func (b stateBus) state(bucket string) (stdio.KeptState, error) {
|
||||
switch bucket {
|
||||
case BucketOf(b.module):
|
||||
return stdio.State("bound"), nil
|
||||
case AssignmentsBucket:
|
||||
return stdio.State("mesh-controller.assignments"), nil
|
||||
}
|
||||
return stdio.KeptState{}, fmt.Errorf("%s is not a bucket this provider keeps or reads", bucket)
|
||||
}
|
||||
|
||||
func (b stateBus) Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error {
|
||||
s, err := b.state(bucket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.Watch("", func(ch stdio.StateChange) error {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
if ch.Op == "delete" {
|
||||
handle(ch.Key, nil)
|
||||
} else {
|
||||
handle(ch.Key, ch.Value)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func (b stateBus) Put(_ context.Context, bucket, key string, value []byte) error {
|
||||
s, err := b.state(bucket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = s.Put(key, json.RawMessage(value))
|
||||
return err
|
||||
}
|
||||
|
||||
// DeleteIf removes the key only while it still holds what the caller read; it reads the key first, so another
|
||||
// writer's change in that short while can still be lost.
|
||||
func (b stateBus) DeleteIf(_ context.Context, bucket, key string, held []byte) error {
|
||||
s, err := b.state(bucket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cur, err := s.Get(key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if cur == nil {
|
||||
return nil
|
||||
}
|
||||
if !sameJSON(cur.Value, held) {
|
||||
return fmt.Errorf("%s is not as it was read: %w", key, stdio.ErrChanged)
|
||||
}
|
||||
return s.Delete(key)
|
||||
}
|
||||
|
||||
func (b stateBus) Delete(_ context.Context, bucket, key string) error {
|
||||
s, err := b.state(bucket)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.Delete(key)
|
||||
}
|
||||
|
||||
// Standing is whether an assignment's value is a standing assignment: false when `ended` is set (the module was
|
||||
// taken off its node, novox/hq ADR 0408 clause 4) or `state` is not empty (a pending one). Such a value is read
|
||||
// exactly as a removed key. Every other field, known or not, changes nothing (issue 1098).
|
||||
func Standing(value []byte) bool {
|
||||
var v struct {
|
||||
Ended json.RawMessage `json:"ended"`
|
||||
State json.RawMessage `json:"state"`
|
||||
}
|
||||
if json.Unmarshal(value, &v) != nil {
|
||||
return true // not decided here: the parse that follows refuses it
|
||||
}
|
||||
return !fieldSet(v.Ended) && !fieldSet(v.State)
|
||||
}
|
||||
|
||||
// fieldSet is whether a field carries something: not absent, null or empty.
|
||||
func fieldSet(f json.RawMessage) bool {
|
||||
s := strings.TrimSpace(string(f))
|
||||
return s != "" && s != "null" && s != `""`
|
||||
}
|
||||
@@ -0,0 +1,803 @@
|
||||
package provider
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// standIn is a stand-in bus: buckets of keys and values, and watches that hand the current values first and then
|
||||
// every change, as the bus's key-value watch does.
|
||||
type standIn struct {
|
||||
mu sync.Mutex
|
||||
buckets map[string]map[string][]byte
|
||||
watchers map[string][]func(string, []byte)
|
||||
}
|
||||
|
||||
func newStandIn() *standIn {
|
||||
return &standIn{buckets: map[string]map[string][]byte{}, watchers: map[string][]func(string, []byte){}}
|
||||
}
|
||||
|
||||
func (s *standIn) Watch(ctx context.Context, bucket string, handle func(key string, value []byte)) error {
|
||||
s.mu.Lock()
|
||||
current := map[string][]byte{}
|
||||
for k, v := range s.buckets[bucket] {
|
||||
current[k] = v
|
||||
}
|
||||
s.watchers[bucket] = append(s.watchers[bucket], handle)
|
||||
s.mu.Unlock()
|
||||
for _, k := range sortedKeys(current) {
|
||||
handle(k, current[k])
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *standIn) Put(_ context.Context, bucket, key string, value []byte) error {
|
||||
s.mu.Lock()
|
||||
if s.buckets[bucket] == nil {
|
||||
s.buckets[bucket] = map[string][]byte{}
|
||||
}
|
||||
s.buckets[bucket][key] = append([]byte(nil), value...)
|
||||
ws := slices.Clone(s.watchers[bucket])
|
||||
s.mu.Unlock()
|
||||
for _, w := range ws {
|
||||
w(key, value)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *standIn) Delete(_ context.Context, bucket, key string) error {
|
||||
s.mu.Lock()
|
||||
delete(s.buckets[bucket], key)
|
||||
ws := slices.Clone(s.watchers[bucket])
|
||||
s.mu.Unlock()
|
||||
for _, w := range ws {
|
||||
w(key, nil)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteIf removes the key only while it still holds what the caller read.
|
||||
func (s *standIn) DeleteIf(ctx context.Context, bucket, key string, held []byte) error {
|
||||
s.mu.Lock()
|
||||
cur := s.buckets[bucket][key]
|
||||
s.mu.Unlock()
|
||||
if !bytes.Equal(cur, held) {
|
||||
return fmt.Errorf("%s: %w", key, stdio.ErrChanged)
|
||||
}
|
||||
return s.Delete(ctx, bucket, key)
|
||||
}
|
||||
|
||||
func (s *standIn) get(bucket, key string) []byte {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.buckets[bucket][key]
|
||||
}
|
||||
|
||||
func (s *standIn) keys(bucket string) []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return sortedKeys(s.buckets[bucket])
|
||||
}
|
||||
|
||||
// everything is every value the bus holds, for a test that no secret is in a bucket.
|
||||
func (s *standIn) everything() []byte {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var all []byte
|
||||
for _, b := range s.buckets {
|
||||
for k, v := range b {
|
||||
all = append(all, k...)
|
||||
all = append(all, v...)
|
||||
}
|
||||
}
|
||||
return all
|
||||
}
|
||||
|
||||
// vault is a stand-in vault: a value per name, or nothing yet.
|
||||
type vault struct {
|
||||
mu sync.Mutex
|
||||
values map[string]string
|
||||
asked []string
|
||||
}
|
||||
|
||||
func (v *vault) Value(_ context.Context, name string) ([]byte, error) {
|
||||
v.mu.Lock()
|
||||
defer v.mu.Unlock()
|
||||
v.asked = append(v.asked, name)
|
||||
val, ok := v.values[name]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("the vault holds no %s yet", name)
|
||||
}
|
||||
return []byte(val), nil
|
||||
}
|
||||
|
||||
// maker records what the provider's adapter was asked to make, with the credential it was handed.
|
||||
type maker struct {
|
||||
mu sync.Mutex
|
||||
made map[string]string
|
||||
// contributed is what each consumer's entry carried of its plan's contribution.
|
||||
contributed map[string]map[string]any
|
||||
// as is the identity each consumer's entry was made under.
|
||||
as map[string]string
|
||||
gone []string
|
||||
fail error
|
||||
calls int
|
||||
}
|
||||
|
||||
func (m *maker) Make(_ context.Context, c Consumer) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.calls++
|
||||
if m.fail != nil {
|
||||
return m.fail
|
||||
}
|
||||
if m.made == nil {
|
||||
m.made = map[string]string{}
|
||||
}
|
||||
m.made[c.Entry.Key()] = string(c.Credential)
|
||||
if m.contributed == nil {
|
||||
m.contributed = map[string]map[string]any{}
|
||||
}
|
||||
m.contributed[c.Entry.Key()] = c.Entry.Contributes
|
||||
if m.as == nil {
|
||||
m.as = map[string]string{}
|
||||
}
|
||||
m.as[c.Entry.Key()] = c.Entry.As
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *maker) Gone(_ context.Context, e Entry) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.gone = append(m.gone, e.Key())
|
||||
return nil
|
||||
}
|
||||
|
||||
// assign puts a consumer's assignment with its plan's provisions into the stand-in mirror.
|
||||
func assign(t *testing.T, bus *standIn, node, module string, provisions ...map[string]string) {
|
||||
t.Helper()
|
||||
ps := []map[string]string{}
|
||||
ps = append(ps, provisions...)
|
||||
raw, err := json.Marshal(map[string]any{"node": node, "module": module, "assigned": "2026-10-15T00:00:00Z",
|
||||
"provisioning": map[string]any{"templates": map[string]string{"source": "x", "digest": "y"},
|
||||
"provisions": ps}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := bus.Put(context.Background(), AssignmentsBucket, node+"."+module, raw); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func bind(provision, local, provider, providerModule string) map[string]string {
|
||||
b := map[string]string{"provision": provision, "provider": provider, "provider-module": providerModule}
|
||||
if local != "" {
|
||||
b["local"] = local
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func entryOf(t *testing.T, bus *standIn, key string) Entry {
|
||||
t.Helper()
|
||||
raw := bus.get(BucketOf("postgres"), key)
|
||||
if raw == nil {
|
||||
t.Fatalf("no entry %s in %s; it holds %v", key, BucketOf("postgres"), bus.keys(BucketOf("postgres")))
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(raw, &e); err != nil {
|
||||
t.Fatalf("entry %s: %v", key, err)
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
// postgres is a provider of postgres-database on novox with a login per consumer (ADR 0048, ADR 0158 §11), its
|
||||
// served values derived from each consumer's identity (ADR 0202 §1).
|
||||
func postgres(bus *standIn, v *vault, m *maker) *Provider {
|
||||
return New(Options{
|
||||
Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Gone: m.Gone,
|
||||
Offers: map[string]Offer{
|
||||
"postgres-database": {Credential: PerConsumer, Serves: map[string]any{
|
||||
"host": "postgres.novox.mesh", "port": 5432, "database": "${consumer:as}"}},
|
||||
},
|
||||
Logf: func(string, ...any) {},
|
||||
})
|
||||
}
|
||||
|
||||
// A provider answers each consumer that binds it with an entry of its own in its own bucket: composed from its
|
||||
// served values and the consumer's identity, naming the credential the vault keeps for that consumer, and never
|
||||
// the credential itself; its adapter is handed each consumer's own value read from the vault.
|
||||
func TestEachConsumerGetsAnEntryOfItsOwnAndTheVaultsValue(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{
|
||||
"pair/novox/postgres-database/ace/letta": "letta-secret-1",
|
||||
"pair/novox/postgres-database/g14/umami": "umami-secret-2",
|
||||
}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
|
||||
p := postgres(bus, v, m)
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.postgres-database",
|
||||
"g14.umami.postgres-database"}) {
|
||||
t.Fatalf("entries %v", got)
|
||||
}
|
||||
e := entryOf(t, bus, "ace.letta.postgres-database")
|
||||
if e.Provider != "novox" || e.ProviderModule != "postgres" || e.Consumer != "ace" || e.Module != "letta" ||
|
||||
e.As != "mesh_ace_letta" || e.User != "mesh_ace_letta" ||
|
||||
e.VaultName != "pair/novox/postgres-database/ace/letta" || e.Refused != "" {
|
||||
t.Fatalf("entry %+v", e)
|
||||
}
|
||||
if e.Values["database"] != "mesh_ace_letta" || e.Values["host"] != "postgres.novox.mesh" ||
|
||||
e.Values["port"] != float64(5432) {
|
||||
t.Fatalf("values %v", e.Values)
|
||||
}
|
||||
if m.made["ace.letta.postgres-database"] != "letta-secret-1" ||
|
||||
m.made["g14.umami.postgres-database"] != "umami-secret-2" {
|
||||
t.Fatalf("made %v", m.made)
|
||||
}
|
||||
if bytes.Contains(bus.everything(), []byte("secret-")) {
|
||||
t.Fatal("a credential is in a bucket")
|
||||
}
|
||||
for _, o := range out {
|
||||
if o.Status != Answered {
|
||||
t.Fatalf("outcome %+v", o)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A provider whose offer shares one credential (ADR 0158) names the vault's one value for the provision, and its
|
||||
// one user name, to every consumer.
|
||||
func TestASharedCredentialIsTheVaultsOneValueAndTheProvidersOneUser(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"provision/novox/sonarr-api": "one-value"}}
|
||||
assign(t, bus, "ace", "overseerr", bind("sonarr-api", "", "novox", "sonarr"))
|
||||
assign(t, bus, "g14", "homepage", bind("sonarr-api", "", "novox", "sonarr"))
|
||||
p := New(Options{Node: "novox", Module: "sonarr", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
|
||||
Offers: map[string]Offer{"sonarr-api": {Credential: Shared, User: "sonarr",
|
||||
Serves: map[string]any{"url": "http://sonarr.novox.mesh:8989"}}}})
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, key := range []string{"ace.overseerr.sonarr-api", "g14.homepage.sonarr-api"} {
|
||||
var e Entry
|
||||
if err := json.Unmarshal(bus.get(BucketOf("sonarr"), key), &e); err != nil {
|
||||
t.Fatalf("%s: %v", key, err)
|
||||
}
|
||||
if e.VaultName != "provision/novox/sonarr-api" || e.User != "sonarr" || e.Values["url"] == nil {
|
||||
t.Fatalf("%s: %+v", key, e)
|
||||
}
|
||||
if m.made[key] != "one-value" {
|
||||
t.Fatalf("%s made with %q", key, m.made[key])
|
||||
}
|
||||
}
|
||||
if bytes.Contains(bus.everything(), []byte("one-value")) {
|
||||
t.Fatal("the shared credential is in a bucket")
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer that keeps several holders of one provision is served no derived value: its entry refuses it, naming
|
||||
// both ends, so its client can report it (ADR 0349 clause 12, ADR 0202 §10); nothing is made for it.
|
||||
func TestAMultiHolderConsumerIsRefusedADerivedValueNamingBothEnds(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta/archive": "x-secret"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "archive", "novox", "postgres"))
|
||||
out, err := postgres(bus, v, m).Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e := entryOf(t, bus, "ace.letta.postgres-database.archive")
|
||||
for _, says := range []string{"letta on ace", "postgres on novox", "postgres-database", "archive"} {
|
||||
if !strings.Contains(e.Refused, says) {
|
||||
t.Fatalf("the refusal does not name %q: %s", says, e.Refused)
|
||||
}
|
||||
}
|
||||
if e.Values != nil || e.VaultName != "" {
|
||||
t.Fatalf("a refused entry carries values: %+v", e)
|
||||
}
|
||||
if m.calls != 0 {
|
||||
t.Fatal("the adapter made a refused consumer")
|
||||
}
|
||||
if len(out) != 1 || out[0].Status != Refused {
|
||||
t.Fatalf("outcomes %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer bound under a local name is served when the offer derives nothing from it: its credential is its own
|
||||
// pair row under that local name.
|
||||
func TestALocalNameIsServedWhenNothingIsDerived(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/redis-cache/ace/letta/queue": "q-secret"}}
|
||||
assign(t, bus, "ace", "letta", bind("redis-cache", "queue", "novox", "redis"))
|
||||
p := New(Options{Node: "novox", Module: "redis", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
|
||||
Offers: map[string]Offer{"redis-cache": {Credential: PerConsumer, Serves: map[string]any{"port": 6379}}}})
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(bus.get(BucketOf("redis"), "ace.letta.redis-cache.queue"), &e); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e.Refused != "" || e.VaultName != "pair/novox/redis-cache/ace/letta/queue" || e.Local != "queue" {
|
||||
t.Fatalf("entry %+v", e)
|
||||
}
|
||||
if m.made["ace.letta.redis-cache.queue"] != "q-secret" {
|
||||
t.Fatalf("made %v", m.made)
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer that unbinds loses its entry and the adapter hears it is gone; an entry another machine's instance of
|
||||
// the same provider wrote is not this one's to remove.
|
||||
func TestAnUnbindRemovesTheEntryAndLeavesAnotherProvidersAlone(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
other, _ := json.Marshal(Entry{Provider: "shanks", ProviderModule: "postgres", Provision: "postgres-database",
|
||||
Consumer: "g14", Module: "gitea"})
|
||||
_ = bus.Put(context.Background(), BucketOf("postgres"), "g14.gitea.postgres-database", other)
|
||||
p := postgres(bus, v, m)
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"g14.gitea.postgres-database"}) {
|
||||
t.Fatalf("entries left %v", got)
|
||||
}
|
||||
if !slices.Equal(m.gone, []string{"ace.letta.postgres-database"}) {
|
||||
t.Fatalf("gone %v", m.gone)
|
||||
}
|
||||
if len(out) != 1 || out[0].Status != Removed {
|
||||
t.Fatalf("outcomes %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A bind whose credential the vault does not hold yet is waited for: nothing is made and no entry is written, so a
|
||||
// consumer is never told of a login its provider has not made; once the vault holds it, it is answered.
|
||||
func TestACredentialTheVaultDoesNotHoldYetIsWaitedFor(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
p := postgres(bus, v, m)
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(out) != 1 || out[0].Status != Waiting || !strings.Contains(out[0].Why, "pair/novox/postgres-database/ace/letta") {
|
||||
t.Fatalf("outcomes %+v", out)
|
||||
}
|
||||
if bus.keys(BucketOf("postgres")) != nil || m.calls != 0 {
|
||||
t.Fatal("a consumer was answered before its credential was there")
|
||||
}
|
||||
v.values["pair/novox/postgres-database/ace/letta"] = "late"
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if entryOf(t, bus, "ace.letta.postgres-database").Refused != "" || m.made["ace.letta.postgres-database"] != "late" {
|
||||
t.Fatal("not answered once the vault held the value")
|
||||
}
|
||||
}
|
||||
|
||||
// Only the binds naming this provider, on this machine, are its own; a bind naming it for a provision it does not
|
||||
// offer is refused, naming both ends.
|
||||
func TestOnlyBindsNamingThisProviderAreAnswered(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "shanks", "postgres"),
|
||||
bind("mongodb-database", "", "novox", "postgres"))
|
||||
assign(t, bus, "ace", "other", bind("postgres-database", "", "novox", "mssql"))
|
||||
out, err := postgres(bus, v, m).Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.mongodb-database"}) {
|
||||
t.Fatalf("entries %v", got)
|
||||
}
|
||||
e := entryOf(t, bus, "ace.letta.mongodb-database")
|
||||
if !strings.Contains(e.Refused, "postgres on novox") || !strings.Contains(e.Refused, "letta on ace") {
|
||||
t.Fatalf("refusal %q", e.Refused)
|
||||
}
|
||||
if len(out) != 1 || m.calls != 0 {
|
||||
t.Fatalf("outcomes %+v, %d made", out, m.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// An assignment that cannot be read is skipped and said, by its key and why; the rest of the pass goes on (issue 825,
|
||||
// finding 3). Its consumer's own entry is left alone, since a bind missed would read as an unbind.
|
||||
func TestAnUnreadableAssignmentIsSkippedAndSaidAndTheRestGoesOn(t *testing.T) {
|
||||
for name, bad := range map[string]string{"not json": "{not json",
|
||||
"another key": `{"node":"g14","module":"umami"}`} {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s",
|
||||
"pair/novox/postgres-database/g14/umami": "u"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
p := postgres(bus, v, m)
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = bus.Put(context.Background(), AssignmentsBucket, "ace.letta", []byte(bad))
|
||||
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
|
||||
var logged []string
|
||||
p.opts.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
if bus.get(BucketOf("postgres"), "g14.umami.postgres-database") == nil {
|
||||
t.Fatalf("%s: the good consumer was not answered", name)
|
||||
}
|
||||
if bus.get(BucketOf("postgres"), "ace.letta.postgres-database") == nil || len(m.gone) != 0 {
|
||||
t.Fatalf("%s: the entry of the unreadable assignment's consumer was removed", name)
|
||||
}
|
||||
said := false
|
||||
for _, o := range out {
|
||||
if o.Key == "ace.letta" && o.Status == Failed && o.Why != "" {
|
||||
said = true
|
||||
}
|
||||
}
|
||||
if !said || !strings.Contains(strings.Join(logged, "\n"), "ace.letta") {
|
||||
t.Fatalf("%s: the skipped assignment is not said: %+v, %q", name, out, logged)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An adapter that fails is said with the credential taken out of its words, and the entry is not written.
|
||||
func TestAFailedMakeIsSaidWithoutTheCredential(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{fail: errors.New(`password "hunter2-secret" rejected`)}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "hunter2-secret"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
var logged []string
|
||||
p := postgres(bus, v, m)
|
||||
p.opts.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(out) != 1 || out[0].Status != Failed || strings.Contains(out[0].Why, "hunter2") {
|
||||
t.Fatalf("outcomes %+v", out)
|
||||
}
|
||||
if strings.Contains(strings.Join(logged, "\n"), "hunter2") || len(logged) == 0 {
|
||||
t.Fatalf("logged %q", logged)
|
||||
}
|
||||
if bus.keys(BucketOf("postgres")) != nil {
|
||||
t.Fatal("an entry was written for a consumer the adapter could not make")
|
||||
}
|
||||
}
|
||||
|
||||
// Run watches the binds on the stand-in bus: a consumer that binds is answered, and one that unbinds is removed,
|
||||
// without a pass asked by hand.
|
||||
func TestRunAnswersBindsAndUnbindsAsTheyAreWatched(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
|
||||
p := postgres(bus, v, m)
|
||||
p.opts.Every = time.Hour
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- p.Run(ctx) }()
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
waitFor(t, func() bool { return bus.get(BucketOf("postgres"), "ace.letta.postgres-database") != nil })
|
||||
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
|
||||
waitFor(t, func() bool { return bus.get(BucketOf("postgres"), "ace.letta.postgres-database") == nil })
|
||||
cancel()
|
||||
if err := <-done; err != nil && !errors.Is(err, context.Canceled) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func waitFor(t *testing.T, ok func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for !ok() {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("not within 3s")
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// The consumer's identity is the mesh's (ADR 0202 §1): mesh_<node>_<module>, lower case, `_` for anything else,
|
||||
// and its DNS form writes `-` for `_`.
|
||||
func TestTheIdentityIsTheMeshsAndItsDNSForm(t *testing.T) {
|
||||
if got := ConsumerIdentity("Home.Server", "my-app"); got != "mesh_home_server_my_app" {
|
||||
t.Fatal(got)
|
||||
}
|
||||
got, err := ServedTo(map[string]any{"bucket": "b-${consumer:as:dns}", "n": 3}, "mesh_ace_letta")
|
||||
if err != nil || got["bucket"] != "b-mesh-ace-letta" || got["n"] != 3 {
|
||||
t.Fatalf("%v %v", got, err)
|
||||
}
|
||||
if _, err := ServedTo(map[string]any{"x": "${consumer:node}"}, "mesh_ace_letta"); err == nil {
|
||||
t.Fatal("a fact the mesh does not state was filled")
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer's plan entry may carry what its module contributes to the provider (issue 776): the helper hands it to
|
||||
// the provider's adapter with the consumer, and does not write it to the bucket.
|
||||
func TestAConsumersContributionIsHandedToTheProviderPerConsumer(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{
|
||||
"pair/novox/postgres-database/ace/nextcloud": "a", "pair/novox/postgres-database/g14/umami": "b"}}
|
||||
raw := `{"node":"ace","module":"nextcloud","provisioning":{"templates":{"source":"x","digest":"y"},` +
|
||||
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
|
||||
`"contributes":{"name":"nextcloud","extensions":["pg_trgm"]}}]}}`
|
||||
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.nextcloud", []byte(raw)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
|
||||
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]any{"name": "nextcloud", "extensions": []any{"pg_trgm"}}
|
||||
if got := m.contributed["ace.nextcloud.postgres-database"]; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("the adapter was handed the contribution %v, want %v", got, want)
|
||||
}
|
||||
if got := m.contributed["g14.umami.postgres-database"]; got != nil {
|
||||
t.Fatalf("a consumer contributing nothing was handed %v", got)
|
||||
}
|
||||
if bytes.Contains(bus.get(BucketOf("postgres"), "ace.nextcloud.postgres-database"), []byte("pg_trgm")) {
|
||||
t.Fatal("a contribution is in the provider's bucket")
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer's plan entry carries where its provider is, which machine, and who the consumer is (issue 788): the
|
||||
// entry's values carry `at` and `from` as the binding file gives them, and its `as` is the plan's, so a module with a
|
||||
// slug keeps the login it has today. An entry of a plan without them is as it was.
|
||||
func TestAConsumersEntryCarriesAtFromAndTheAsOfItsPlan(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{
|
||||
"pair/novox/postgres-database/ace/claude-code": "a", "pair/novox/postgres-database/g14/umami": "b"}}
|
||||
raw := `{"node":"ace","module":"claude-code","provisioning":{"templates":{"source":"x","digest":"y"},` +
|
||||
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
|
||||
`"at":"10.7.0.1","from":"novox","as":"mesh_ace_agent"}]}}`
|
||||
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.claude-code", []byte(raw)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
|
||||
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.claude-code.postgres-database"), &e); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e.Values["at"] != "10.7.0.1" || e.Values["from"] != "novox" || e.Values["port"] == nil {
|
||||
t.Fatalf("the entry's values are %v", e.Values)
|
||||
}
|
||||
if e.As != "mesh_ace_agent" || e.User != "mesh_ace_agent" || m.as["ace.claude-code.postgres-database"] != "mesh_ace_agent" {
|
||||
t.Fatalf("the entry is %+v, made as %q", e, m.as["ace.claude-code.postgres-database"])
|
||||
}
|
||||
var plain Entry
|
||||
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "g14.umami.postgres-database"), &plain); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, has := plain.Values["at"]; has || plain.As != "mesh_g14_umami" {
|
||||
t.Fatalf("an entry of a plan without them changed: %+v", plain)
|
||||
}
|
||||
}
|
||||
|
||||
// A consumer's plan entry may carry the names the controller composed for it (issue 792): each is copied into the
|
||||
// entry's values under its key, over anything the provider serves under it. An entry without them is as it was.
|
||||
func TestAConsumersEntryCarriesTheNamesOfItsPlan(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{
|
||||
"pair/novox/postgres-database/ace/gitea": "a", "pair/novox/postgres-database/g14/umami": "b"}}
|
||||
raw := `{"node":"ace","module":"gitea","provisioning":{"templates":{"source":"x","digest":"y"},` +
|
||||
`"provisions":[{"provision":"postgres-database","provider":"novox","provider-module":"postgres",` +
|
||||
`"names":{"name":"git.example.org","internal-name":"git.mesh","name-web":"web.example.org"}}]}}`
|
||||
if err := bus.Put(context.Background(), AssignmentsBucket, "ace.gitea", []byte(raw)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assign(t, bus, "g14", "umami", bind("postgres-database", "", "novox", "postgres"))
|
||||
if _, err := postgres(bus, v, m).Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.gitea.postgres-database"), &e); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if e.Values["name"] != "git.example.org" || e.Values["internal-name"] != "git.mesh" ||
|
||||
e.Values["name-web"] != "web.example.org" || e.Values["port"] == nil {
|
||||
t.Fatalf("the entry's values are %v", e.Values)
|
||||
}
|
||||
var plain Entry
|
||||
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "g14.umami.postgres-database"), &plain); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, has := plain.Values["name"]; has {
|
||||
t.Fatalf("an entry of a plan without names changed: %+v", plain)
|
||||
}
|
||||
}
|
||||
|
||||
// refusedByTheStateGuard mirrors the key rule of the runtime's state guard (mesh-tools node-tools internal/bus
|
||||
// state.go): a JSON key whose lowercased name, with `-`, `_` and spaces taken out, ends in one of these is refused.
|
||||
func refusedByTheStateGuard(key string) bool {
|
||||
k := strings.NewReplacer("-", "", "_", "", " ", "").Replace(strings.ToLower(key))
|
||||
for _, end := range []string{"password", "passwd", "secret", "token", "credential", "credentials",
|
||||
"authorization", "apikey", "privatekey", "accesskey", "cookie"} {
|
||||
if strings.HasSuffix(k, end) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func keysOf(v any, into *[]string) {
|
||||
switch x := v.(type) {
|
||||
case map[string]any:
|
||||
for k, c := range x {
|
||||
*into = append(*into, k)
|
||||
keysOf(c, into)
|
||||
}
|
||||
case []any:
|
||||
for _, c := range x {
|
||||
keysOf(c, into)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Every entry the helper writes passes the state guard's key rule (issue 945), and names where the vault keeps the
|
||||
// credential under `vault-name`, for a login per consumer and for a shared one.
|
||||
func TestAnEntryHasNoKeyTheStateGuardRefuses(t *testing.T) {
|
||||
for name, form := range map[string]CredentialForm{"pair": PerConsumer, "shared": Shared} {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1",
|
||||
"provision/novox/postgres-database": "s2"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
|
||||
Offers: map[string]Offer{"postgres-database": {Credential: form, User: "shared-user"}}})
|
||||
if _, err := p.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var doc map[string]any
|
||||
if err := json.Unmarshal(bus.get(BucketOf("postgres"), "ace.letta.postgres-database"), &doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var keys []string
|
||||
keysOf(doc, &keys)
|
||||
for _, k := range keys {
|
||||
if refusedByTheStateGuard(k) {
|
||||
t.Fatalf("%s: the entry carries the key %q, which the state guard refuses", name, k)
|
||||
}
|
||||
}
|
||||
want := map[string]string{"pair": "pair/novox/postgres-database/ace/letta",
|
||||
"shared": "provision/novox/postgres-database"}[name]
|
||||
if doc["vault-name"] != want {
|
||||
t.Fatalf("%s: vault-name is %v, want %s", name, doc["vault-name"], want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The plan's credential form and holder name win over the offer's; the offer's form is the default (issue 825,
|
||||
// finding 4).
|
||||
func TestThePlansCredentialFormWinsOverTheOffers(t *testing.T) {
|
||||
run := func(offer CredentialForm, plan map[string]string) Entry {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "p",
|
||||
"provision/novox/postgres-database": "s", "provision/novox/postgres-database/ace/letta": "h"}}
|
||||
b := bind("postgres-database", "", "novox", "postgres")
|
||||
for k, val := range plan {
|
||||
b[k] = val
|
||||
}
|
||||
assign(t, bus, "ace", "letta", b)
|
||||
p := New(Options{Node: "novox", Module: "postgres", Bus: bus, Vault: v, Make: m.Make, Logf: func(string, ...any) {},
|
||||
Offers: map[string]Offer{"postgres-database": {Credential: offer, User: "shared-user"}}})
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil || len(out) != 1 || out[0].Status != Answered {
|
||||
t.Fatalf("%v %+v", err, out)
|
||||
}
|
||||
return entryOf(t, bus, "ace.letta.postgres-database")
|
||||
}
|
||||
if e := run(PerConsumer, map[string]string{"credential-form": "shared"}); e.VaultName !=
|
||||
"provision/novox/postgres-database" || e.User != "shared-user" {
|
||||
t.Fatalf("shared over pair: %+v", e)
|
||||
}
|
||||
if e := run(Shared, map[string]string{"credential-form": "pair"}); e.VaultName !=
|
||||
"pair/novox/postgres-database/ace/letta" || e.User != "mesh_ace_letta" {
|
||||
t.Fatalf("pair over shared: %+v", e)
|
||||
}
|
||||
for _, form := range []string{"issued", "given"} {
|
||||
if e := run(PerConsumer, map[string]string{"credential-form": form, "name": "main"}); e.VaultName !=
|
||||
"provision/novox/postgres-database/ace/letta" || e.User != "mesh_ace_letta" {
|
||||
t.Fatalf("%s holder: %+v", form, e)
|
||||
}
|
||||
}
|
||||
if e := run(Shared, nil); e.VaultName != "provision/novox/postgres-database" {
|
||||
t.Fatalf("the offer's form is the default: %+v", e)
|
||||
}
|
||||
}
|
||||
|
||||
// An instance deletes only an entry it wrote, and only while the entry is as it read it: one that another instance of
|
||||
// the provider wrote after the consumer moved stands (issue 825, finding 5).
|
||||
func TestAnEntryAnotherInstanceWroteAfterTheMoveStands(t *testing.T) {
|
||||
bus, m := newStandIn(), &maker{}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s"}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
a := postgres(bus, v, m)
|
||||
if _, err := a.Pass(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The consumer moves to the instance on shanks: A's pass reads its own entry and the unbind, and while it lets
|
||||
// the consumer go, B writes the consumer's entry.
|
||||
_ = bus.Delete(context.Background(), AssignmentsBucket, "ace.letta")
|
||||
theirs, _ := json.Marshal(Entry{Provider: "shanks", ProviderModule: "postgres", Provision: "postgres-database",
|
||||
Consumer: "ace", Module: "letta", VaultName: "pair/shanks/postgres-database/ace/letta"})
|
||||
a.opts.Gone = func(ctx context.Context, e Entry) error {
|
||||
return bus.Put(ctx, BucketOf("postgres"), e.Key(), theirs)
|
||||
}
|
||||
out, err := a.Pass(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := bus.get(BucketOf("postgres"), "ace.letta.postgres-database"); !bytes.Equal(got, theirs) {
|
||||
t.Fatalf("the other instance's entry was removed: %s", got)
|
||||
}
|
||||
for _, o := range out {
|
||||
if o.Status == Removed {
|
||||
t.Fatalf("outcomes %+v", out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A failed Make is said with the credential taken out in the forms it can come back in: as it is, in base64 (standard
|
||||
// and URL, with and without padding) and as a JSON string says it (issue 825, finding 6).
|
||||
func TestAFailedMakeIsSaidWithoutTheCredentialInAnyOfItsForms(t *testing.T) {
|
||||
cred := "pa/ss+w\"rd?>>x"
|
||||
b := []byte(cred)
|
||||
jsonQuoted, _ := json.Marshal(cred)
|
||||
forms := map[string]string{
|
||||
"raw": cred, "std": base64.StdEncoding.EncodeToString(b), "std unpadded": base64.RawStdEncoding.EncodeToString(b),
|
||||
"url": base64.URLEncoding.EncodeToString(b), "url unpadded": base64.RawURLEncoding.EncodeToString(b),
|
||||
"json": string(jsonQuoted[1 : len(jsonQuoted)-1]),
|
||||
}
|
||||
if forms["std"] == forms["url"] || forms["json"] == forms["raw"] {
|
||||
t.Fatal("the test credential does not tell the forms apart")
|
||||
}
|
||||
for name, form := range forms {
|
||||
bus, m := newStandIn(), &maker{fail: errors.New("login failed for " + form + " at the server")}
|
||||
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": cred}}
|
||||
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
|
||||
out, err := postgres(bus, v, m).Pass(context.Background())
|
||||
if err != nil || len(out) != 1 {
|
||||
t.Fatalf("%v %+v", err, out)
|
||||
}
|
||||
if strings.Contains(out[0].Why, form) || !strings.Contains(out[0].Why, "<credential>") {
|
||||
t.Fatalf("%s form: %q", name, out[0].Why)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An offer that declares no credential is answered with none whatever form the plan says (the controller writes
|
||||
// "pair" for such an offer): no vault name, and the Vault is never asked, so a provider with no Vault works (issue 825,
|
||||
// finding 4; issue 923).
|
||||
func TestAnOfferWithNoCredentialIgnoresThePlansForm(t *testing.T) {
|
||||
bus := newStandIn()
|
||||
b := bind("route", "", "novox", "route-proxy")
|
||||
b["credential-form"] = "pair"
|
||||
assign(t, bus, "ace", "letta", b)
|
||||
p := New(Options{Node: "novox", Module: "route-proxy", Bus: bus, Logf: func(string, ...any) {},
|
||||
Offers: map[string]Offer{"route": {Credential: NoCredential}}})
|
||||
out, err := p.Pass(context.Background())
|
||||
if err != nil || len(out) != 1 || out[0].Status != Answered {
|
||||
t.Fatalf("%v %+v", err, out)
|
||||
}
|
||||
var e Entry
|
||||
if err := json.Unmarshal(bus.get(BucketOf("route-proxy"), "ace.letta.route"), &e); err != nil || e.VaultName != "" ||
|
||||
e.User != "" {
|
||||
t.Fatalf("%v %+v", err, e)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package provider
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
stdio "git.novox.be/novox/mesh-sdk/go"
|
||||
)
|
||||
|
||||
// runtime drives a bundle over pipes as the runtime does: it reads what the bundle says and sends what the runtime would.
|
||||
type runtime struct {
|
||||
t *testing.T
|
||||
lines *bufio.Scanner
|
||||
in io.Writer
|
||||
}
|
||||
|
||||
func (r *runtime) next() map[string]any {
|
||||
done := make(chan map[string]any, 1)
|
||||
go func() {
|
||||
if r.lines.Scan() {
|
||||
var m map[string]any
|
||||
_ = json.Unmarshal(r.lines.Bytes(), &m)
|
||||
done <- m
|
||||
}
|
||||
}()
|
||||
select {
|
||||
case m := <-done:
|
||||
return m
|
||||
case <-time.After(3 * time.Second):
|
||||
r.t.Fatal("the bundle said nothing")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func (r *runtime) send(m map[string]any) {
|
||||
b, _ := json.Marshal(m)
|
||||
_, _ = r.in.Write(append(b, '\n'))
|
||||
}
|
||||
|
||||
func (r *runtime) call(id int, tool string, args map[string]any) {
|
||||
r.send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "tools/call", "params": map[string]any{"name": tool, "arguments": args}})
|
||||
}
|
||||
|
||||
func serve(t *testing.T, tools []stdio.Tool) *runtime {
|
||||
t.Setenv(stdio.ServedAs, "keeper")
|
||||
inR, inW := io.Pipe()
|
||||
outR, outW := io.Pipe()
|
||||
go func() { _ = stdio.ServeOn("", tools, inR, outW) }()
|
||||
t.Cleanup(func() { _ = inW.Close() })
|
||||
return &runtime{t: t, lines: bufio.NewScanner(outR), in: inW}
|
||||
}
|
||||
|
||||
// The bus of a moved provider maps its own bucket to the state `bound` and the assignments to the state
|
||||
// `mesh-controller.assignments`, writes only the own bucket's state, and refuses any other bucket.
|
||||
func TestStateBusMapsTheProvidersBucketsToTheRuntimesStates(t *testing.T) {
|
||||
bus := StateBus("postgres")
|
||||
rt := serve(t, []stdio.Tool{
|
||||
{Name: "watch", Run: func(a map[string]any) (any, error) {
|
||||
return nil, bus.Watch(context.Background(), a["bucket"].(string), func(string, []byte) {})
|
||||
}},
|
||||
{Name: "put", Run: func(map[string]any) (any, error) {
|
||||
return "put", bus.Put(context.Background(), BucketOf("postgres"), "k", []byte(`{"a":1}`))
|
||||
}},
|
||||
{Name: "delete", Run: func(map[string]any) (any, error) {
|
||||
return "deleted", bus.Delete(context.Background(), BucketOf("postgres"), "k")
|
||||
}},
|
||||
})
|
||||
for i, c := range []struct{ bucket, state string }{
|
||||
{BucketOf("postgres"), "bound"},
|
||||
{AssignmentsBucket, "mesh-controller.assignments"},
|
||||
} {
|
||||
rt.call(i+1, "watch", map[string]any{"bucket": c.bucket})
|
||||
w := rt.next()
|
||||
if w["method"] != "mesh/state.watch" || w["params"].(map[string]any)["state"] != c.state {
|
||||
t.Fatalf("bucket %s asked %v, want the state %s", c.bucket, w, c.state)
|
||||
}
|
||||
rt.send(map[string]any{"jsonrpc": "2.0", "id": w["id"], "result": map[string]any{}})
|
||||
rt.next()
|
||||
}
|
||||
rt.call(3, "put", nil)
|
||||
p := rt.next()
|
||||
if p["method"] != "mesh/state.put" || p["params"].(map[string]any)["state"] != "bound" || p["params"].(map[string]any)["key"] != "k" {
|
||||
t.Fatalf("put asked %v", p)
|
||||
}
|
||||
rt.send(map[string]any{"jsonrpc": "2.0", "id": p["id"], "result": map[string]any{"revision": 1}})
|
||||
rt.next()
|
||||
rt.call(4, "delete", nil)
|
||||
d := rt.next()
|
||||
if d["method"] != "mesh/state.delete" || d["params"].(map[string]any)["state"] != "bound" {
|
||||
t.Fatalf("delete asked %v", d)
|
||||
}
|
||||
rt.send(map[string]any{"jsonrpc": "2.0", "id": d["id"], "result": map[string]any{}})
|
||||
rt.next()
|
||||
|
||||
for _, bucket := range []string{"other_bound", "elsewhere"} {
|
||||
if err := bus.Watch(context.Background(), bucket, func(string, []byte) {}); err == nil || !strings.Contains(err.Error(), "not a bucket") {
|
||||
t.Fatalf("a watch of %s gave %v", bucket, err)
|
||||
}
|
||||
if err := bus.Put(context.Background(), bucket, "k", nil); err == nil {
|
||||
t.Fatalf("a put to %s was allowed", bucket)
|
||||
}
|
||||
if err := bus.Delete(context.Background(), bucket, "k"); err == nil {
|
||||
t.Fatalf("a delete in %s was allowed", bucket)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A watch whose ctx has ended calls its handler no more, though the runtime keeps sending changes: the Bus contract
|
||||
// says a watch lasts until ctx ends, and a stdio state watch lasts as long as the process.
|
||||
func TestStateBusWatchCallsItsHandlerNoMoreOnceItsContextEnded(t *testing.T) {
|
||||
bus := StateBus("postgres")
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
handled := make(chan string, 8)
|
||||
rt := serve(t, []stdio.Tool{
|
||||
{Name: "start", Run: func(map[string]any) (any, error) {
|
||||
return "started", bus.Watch(ctx, AssignmentsBucket, func(key string, value []byte) {
|
||||
handled <- key + "=" + string(value)
|
||||
})
|
||||
}},
|
||||
{Name: "stop", Run: func(map[string]any) (any, error) { cancel(); return "stopped", nil }},
|
||||
})
|
||||
change := func(id int, watch any, key, op string) {
|
||||
rt.send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "mesh/state", "params": map[string]any{"watch": watch,
|
||||
"change": map[string]any{"state": "mesh-controller.assignments", "key": key, "op": op, "value": "v"}}})
|
||||
if r := rt.next(); r["id"] != float64(id) || r["error"] != nil {
|
||||
t.Fatalf("a change was answered %v", r)
|
||||
}
|
||||
}
|
||||
rt.call(1, "start", nil)
|
||||
w := rt.next()
|
||||
rt.send(map[string]any{"jsonrpc": "2.0", "id": w["id"], "result": map[string]any{}})
|
||||
rt.next()
|
||||
|
||||
change(50, w["id"], "before", "put")
|
||||
if got := <-handled; got != `before="v"` {
|
||||
t.Fatalf("handled %q", got)
|
||||
}
|
||||
rt.call(2, "stop", nil)
|
||||
rt.next()
|
||||
change(51, w["id"], "after", "put")
|
||||
change(52, w["id"], "after", "delete")
|
||||
select {
|
||||
case got := <-handled:
|
||||
t.Fatalf("the handler was called after its context ended: %q", got)
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package stdio
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A module's code reads its values at runtime (novox/hq ADR 0349 clause 14, ADR 0370, and the part of the feature 57 design that lets a running module read, rather than be handed in a file, its provisioned values): its config
|
||||
// values and its bindings' values that are not secret, for what its manifest declares, and watches them. The
|
||||
// runtime that launched this bundle asks its machine's provisioner client **as this bundle's module** — the module it
|
||||
// launched, never a name this code passes — and the client answers only what that module declares, only while the
|
||||
// mesh places it on that machine. A secret is refused while the module's code runs in the tool runner: the vault
|
||||
// hands a secret only to a caller the bus proves, and the bus proves only the tool runner there, so a module's
|
||||
// secrets reach it through its files until it runs under a bus account of its own (novox/hq issue 569).
|
||||
|
||||
// ProvisionChange is what a watch hands over: the value now, or — once it is no longer answered, the module taken
|
||||
// off the machine or the key no longer declared — Refused, why.
|
||||
type ProvisionChange struct {
|
||||
Kind string `json:"kind"`
|
||||
Key string `json:"key"`
|
||||
Value json.RawMessage `json:"value,omitempty"`
|
||||
Refused string `json:"refused,omitempty"`
|
||||
}
|
||||
|
||||
const provisionTimeout = 30 * time.Second
|
||||
|
||||
func provisionGet(kind, key string) (json.RawMessage, error) {
|
||||
c, err := live()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.ask("mesh/provision.get", map[string]any{"kind": kind, "key": key}, provisionTimeout)
|
||||
}
|
||||
|
||||
func provisionWatch(kind, key string, handle func(ProvisionChange) error) error {
|
||||
if handle == nil {
|
||||
return errors.New("a watch needs a handler")
|
||||
}
|
||||
c, err := live()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
id := c.newID()
|
||||
c.mu.Lock()
|
||||
c.provisioned[id] = handle
|
||||
c.mu.Unlock()
|
||||
if _, err := c.askAs(id, "mesh/provision.watch", map[string]any{"kind": kind, "key": key}, provisionTimeout); err != nil {
|
||||
c.mu.Lock()
|
||||
delete(c.provisioned, id)
|
||||
c.mu.Unlock()
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Config is one of this module's config values as mesh-config resolved it for this machine, as JSON: the key as
|
||||
// its manifest declares it under `config`. A key it does not declare is refused, with the reason.
|
||||
func Config(key string) (json.RawMessage, error) { return provisionGet("config", key) }
|
||||
|
||||
// WatchConfig hands handle the config value now and then every change, in order, and returns once the value now is
|
||||
// handled. The watch lasts as long as the process.
|
||||
func WatchConfig(key string, handle func(ProvisionChange) error) error {
|
||||
return provisionWatch("config", key, handle)
|
||||
}
|
||||
|
||||
// Bound is one value, not a secret, of what this module is bound to: the provision as its manifest requires it,
|
||||
// and the key its provider answers (`${bound:<provision>:<key>}`).
|
||||
func Bound(provision, key string) (json.RawMessage, error) {
|
||||
return provisionGet("bound", provision+":"+key)
|
||||
}
|
||||
|
||||
// WatchBound is WatchConfig for a bound value.
|
||||
func WatchBound(provision, key string, handle func(ProvisionChange) error) error {
|
||||
return provisionWatch("bound", provision+":"+key, handle)
|
||||
}
|
||||
|
||||
// Secret is one of this module's secrets. Refused, with the reason, while the module's code runs in the tool
|
||||
// runner (novox/hq ADR 0349 §11, ADR 0370 §2): its secrets reach it through its files.
|
||||
func Secret(name string) ([]byte, error) {
|
||||
raw, err := provisionGet("secret", name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var s string
|
||||
if err := json.Unmarshal(raw, &s); err != nil {
|
||||
return nil, errors.New("the runtime answered a secret that is not a string")
|
||||
}
|
||||
return []byte(s), nil
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package stdio
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A Go bundle reads its values at runtime through the runtime (novox/hq ADR 0349 §14, ADR 0370, issue 685): Config,
|
||||
// Bound and their watches ask the runtime, which asks as the module; a watch's value now is handled before it
|
||||
// returns and each change after is named by the watch's id; what the module does not declare and every secret come
|
||||
// back refused, in the runtime's words; a change for no watch of this bundle is refused.
|
||||
func TestABundleReadsAndWatchesItsValuesThroughTheRuntime(t *testing.T) {
|
||||
t.Setenv(ServedAs, "claude-code")
|
||||
inR, inW := io.Pipe()
|
||||
outR, outW := io.Pipe()
|
||||
seen := make(chan string, 16)
|
||||
watched := make(chan error, 1)
|
||||
answers := make(chan string, 4)
|
||||
go func() {
|
||||
_ = ServeOn("", []Tool{
|
||||
{Name: "start", Run: func(map[string]any) (any, error) {
|
||||
go func() {
|
||||
watched <- WatchConfig("managed_settings", func(c ProvisionChange) error {
|
||||
seen <- c.Kind + " " + c.Key + " " + string(c.Value) + c.Refused
|
||||
return nil
|
||||
})
|
||||
}()
|
||||
return "started", nil
|
||||
}},
|
||||
{Name: "read", Run: func(a map[string]any) (any, error) {
|
||||
var v json.RawMessage
|
||||
var err error
|
||||
switch a["what"] {
|
||||
case "config":
|
||||
v, err = Config(a["key"].(string))
|
||||
case "bound":
|
||||
v, err = Bound("mcp-endpoint", "url")
|
||||
case "secret":
|
||||
var s []byte
|
||||
s, err = Secret("gitea-read")
|
||||
v = json.RawMessage(s)
|
||||
}
|
||||
if err != nil {
|
||||
answers <- "refused: " + err.Error()
|
||||
} else {
|
||||
answers <- string(v)
|
||||
}
|
||||
return "read", nil
|
||||
}},
|
||||
}, inR, outW)
|
||||
}()
|
||||
lines := bufio.NewScanner(outR)
|
||||
next := func() map[string]any {
|
||||
done := make(chan map[string]any, 1)
|
||||
go func() {
|
||||
if lines.Scan() {
|
||||
var m map[string]any
|
||||
_ = json.Unmarshal(lines.Bytes(), &m)
|
||||
done <- m
|
||||
}
|
||||
}()
|
||||
select {
|
||||
case m := <-done:
|
||||
return m
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("the bundle said nothing")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
send := func(m map[string]any) { b, _ := json.Marshal(m); _, _ = inW.Write(append(b, '\n')) }
|
||||
// asked is the bundle's next ask of the runtime, beside its answers to this tool call and those before.
|
||||
asked := func(callID int) map[string]any {
|
||||
t.Helper()
|
||||
var ask map[string]any
|
||||
for ask == nil {
|
||||
m := next()
|
||||
if m["method"] != nil {
|
||||
ask = m
|
||||
} else if id, ok := m["id"].(float64); !ok || id > float64(callID) || m["error"] != nil {
|
||||
t.Fatalf("said %v", m)
|
||||
}
|
||||
}
|
||||
return ask
|
||||
}
|
||||
|
||||
// The watch: asked of the runtime with the kind and the key and no module, the module being the runtime's word.
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": map[string]any{"name": "start"}})
|
||||
watch := asked(1)
|
||||
if watch["method"] != "mesh/provision.watch" {
|
||||
t.Fatalf("asked %v", watch)
|
||||
}
|
||||
if p := watch["params"].(map[string]any); p["kind"] != "config" || p["key"] != "managed_settings" || len(p) != 2 {
|
||||
t.Fatalf("asked to watch %v", p)
|
||||
}
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": 50, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
|
||||
"change": map[string]any{"kind": "config", "key": "managed_settings", "value": map[string]any{"spinnerTipsEnabled": false}}}})
|
||||
for {
|
||||
if r := next(); r["id"] == float64(50) {
|
||||
if r["error"] != nil {
|
||||
t.Fatalf("the value now was answered %v", r)
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
if s := <-seen; s != `config managed_settings {"spinnerTipsEnabled":false}` {
|
||||
t.Fatalf("handled %q", s)
|
||||
}
|
||||
select {
|
||||
case err := <-watched:
|
||||
t.Fatalf("the watch returned before it was answered: %v", err)
|
||||
default:
|
||||
}
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": watch["id"], "result": map[string]any{}})
|
||||
if err := <-watched; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A change, and then the value no longer answered.
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": 51, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
|
||||
"change": map[string]any{"kind": "config", "key": "managed_settings", "value": map[string]any{"spinnerTipsEnabled": true}}}})
|
||||
if r := next(); r["id"] != float64(51) || r["error"] != nil {
|
||||
t.Fatalf("a change was answered %v", r)
|
||||
}
|
||||
if s := <-seen; s != `config managed_settings {"spinnerTipsEnabled":true}` {
|
||||
t.Fatalf("handled %q", s)
|
||||
}
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": 52, "method": "mesh/provision", "params": map[string]any{"watch": watch["id"],
|
||||
"change": map[string]any{"kind": "config", "key": "managed_settings", "refused": "claude-code is not placed on g14"}}})
|
||||
if r := next(); r["id"] != float64(52) || r["error"] != nil {
|
||||
t.Fatalf("a refusal was answered %v", r)
|
||||
}
|
||||
if s := <-seen; s != `config managed_settings claude-code is not placed on g14` {
|
||||
t.Fatalf("handled %q", s)
|
||||
}
|
||||
// A change for a watch this bundle never asked for is refused.
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": 53, "method": "mesh/provision", "params": map[string]any{"watch": "bundle-999",
|
||||
"change": map[string]any{"kind": "config", "key": "x"}}})
|
||||
if r := next(); r["id"] != float64(53) || !strings.Contains(r["error"].(map[string]any)["message"].(string), "names no watch") {
|
||||
t.Fatalf("a change for no watch was answered %v", r)
|
||||
}
|
||||
|
||||
// Each reader asks the runtime, and hands back its answer or its refusal.
|
||||
for i, c := range []struct {
|
||||
what, key, kind, askedKey string
|
||||
result any
|
||||
refusal string
|
||||
want string
|
||||
}{
|
||||
{what: "config", key: "role", kind: "config", askedKey: "role", result: "the build machine", want: `"the build machine"`},
|
||||
{what: "config", key: "nothing", kind: "config", askedKey: "nothing",
|
||||
refusal: "claude-code declares no config value nothing", want: "refused: claude-code declares no config value nothing"},
|
||||
{what: "bound", kind: "bound", askedKey: "mcp-endpoint:url", result: "http://127.0.0.1:9000", want: `"http://127.0.0.1:9000"`},
|
||||
{what: "secret", kind: "secret", askedKey: "gitea-read",
|
||||
refusal: "the vault hands a secret only to a caller the bus proves", want: "refused: the vault hands a secret only to a caller the bus proves"},
|
||||
} {
|
||||
id := 10 + i
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": id, "method": "tools/call", "params": map[string]any{"name": "read",
|
||||
"arguments": map[string]any{"what": c.what, "key": c.key}}})
|
||||
get := asked(id)
|
||||
if p := get["params"].(map[string]any); get["method"] != "mesh/provision.get" || p["kind"] != c.kind || p["key"] != c.askedKey || len(p) != 2 {
|
||||
t.Fatalf("%s: asked %v", c.what, get)
|
||||
}
|
||||
if c.refusal != "" {
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": get["id"], "error": map[string]any{"code": -32000, "message": c.refusal}})
|
||||
} else {
|
||||
send(map[string]any{"jsonrpc": "2.0", "id": get["id"], "result": c.result})
|
||||
}
|
||||
if got := <-answers; got != c.want {
|
||||
t.Errorf("%s %s: %q, want %q", c.what, c.key, got, c.want)
|
||||
}
|
||||
}
|
||||
_ = inW.Close()
|
||||
}
|
||||
+203
@@ -0,0 +1,203 @@
|
||||
// Package seal is the mesh's sealed box (novox/hq ADR 0183, ADR 0206): a value sealed to one recipient's
|
||||
// public key, which only that recipient's private key opens. X25519 for the agreement, HKDF-SHA256 for the
|
||||
// key, AES-256-GCM for the box — the box the licence manager made and opened before it lived here, byte
|
||||
// for byte, and the one the SDK's TypeScript `seal` makes and opens.
|
||||
//
|
||||
// A box is `{ v: 1, eph, iv, tag, ct }`, every field standard base64; `eph` is the one-time public key as
|
||||
// SPKI DER. The key is derived with the salt eph-DER ‖ the recipient's raw public key and the info
|
||||
// "novox-mesh sealed box v1", so a box cannot be re-addressed to another key.
|
||||
//
|
||||
// Keys travel as PEM: the public key as SPKI ("PUBLIC KEY"), the private key as PKCS #8 ("PRIVATE KEY").
|
||||
// Standard library only.
|
||||
package seal
|
||||
|
||||
import (
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"crypto/ecdh"
|
||||
"crypto/hmac"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/x509"
|
||||
"encoding/base64"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Box is a value sealed to one recipient, as it travels in JSON.
|
||||
type Box struct {
|
||||
V int `json:"v"`
|
||||
Eph string `json:"eph"`
|
||||
IV string `json:"iv"`
|
||||
Tag string `json:"tag"`
|
||||
Ct string `json:"ct"`
|
||||
}
|
||||
|
||||
// KeyPair is a recipient's keypair as the two PEM strings it is kept and sent as.
|
||||
type KeyPair struct {
|
||||
PublicKey string `json:"publicKey"`
|
||||
PrivateKey string `json:"privateKey"`
|
||||
}
|
||||
|
||||
// Version is the only box version Open opens.
|
||||
const Version = 1
|
||||
|
||||
const info = "novox-mesh sealed box v1"
|
||||
|
||||
// GenerateKeyPair makes a fresh X25519 keypair, PEM-encoded.
|
||||
func GenerateKeyPair() (KeyPair, error) {
|
||||
priv, err := ecdh.X25519().GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return KeyPair{}, err
|
||||
}
|
||||
pubDER, err := x509.MarshalPKIXPublicKey(priv.PublicKey())
|
||||
if err != nil {
|
||||
return KeyPair{}, err
|
||||
}
|
||||
privDER, err := x509.MarshalPKCS8PrivateKey(priv)
|
||||
if err != nil {
|
||||
return KeyPair{}, err
|
||||
}
|
||||
return KeyPair{
|
||||
PublicKey: string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: pubDER})),
|
||||
PrivateKey: string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: privDER})),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Seal seals plaintext to the recipient's PEM public key, with a fresh one-time key and iv.
|
||||
func Seal(plaintext, recipientPEM string) (Box, error) {
|
||||
eph, err := ecdh.X25519().GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return Box{}, err
|
||||
}
|
||||
iv := make([]byte, 12)
|
||||
if _, err := rand.Read(iv); err != nil {
|
||||
return Box{}, err
|
||||
}
|
||||
return sealWith(plaintext, recipientPEM, eph, iv)
|
||||
}
|
||||
|
||||
// sealWith is Seal with its two sources of randomness given, for the byte-for-byte tests.
|
||||
func sealWith(plaintext, recipientPEM string, eph *ecdh.PrivateKey, iv []byte) (Box, error) {
|
||||
recipient, err := publicFromPEM(recipientPEM)
|
||||
if err != nil {
|
||||
return Box{}, err
|
||||
}
|
||||
secret, err := eph.ECDH(recipient)
|
||||
if err != nil {
|
||||
return Box{}, err
|
||||
}
|
||||
ephDER, err := x509.MarshalPKIXPublicKey(eph.PublicKey())
|
||||
if err != nil {
|
||||
return Box{}, err
|
||||
}
|
||||
gcm, err := newGCM(boxKey(secret, ephDER, recipient.Bytes()))
|
||||
if err != nil {
|
||||
return Box{}, err
|
||||
}
|
||||
out := gcm.Seal(nil, iv, []byte(plaintext), nil)
|
||||
ct, tag := out[:len(out)-gcm.Overhead()], out[len(out)-gcm.Overhead():]
|
||||
b64 := base64.StdEncoding.EncodeToString
|
||||
return Box{V: Version, Eph: b64(ephDER), IV: b64(iv), Tag: b64(tag), Ct: b64(ct)}, nil
|
||||
}
|
||||
|
||||
// Open opens a box with the recipient's PEM private key. It fails for a box sealed to another key, one
|
||||
// changed in any field, or one of another version.
|
||||
func Open(box Box, privatePEM string) (string, error) {
|
||||
if box.V != Version {
|
||||
return "", fmt.Errorf("a sealed box of version %d, not %d", box.V, Version)
|
||||
}
|
||||
priv, err := privateFromPEM(privatePEM)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
d := base64.StdEncoding.DecodeString
|
||||
ephDER, err := d(box.Eph)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the box's eph: %w", err)
|
||||
}
|
||||
ephKey, err := x509.ParsePKIXPublicKey(ephDER)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the box's eph: %w", err)
|
||||
}
|
||||
eph, ok := ephKey.(*ecdh.PublicKey)
|
||||
if !ok || eph.Curve() != ecdh.X25519() {
|
||||
return "", errors.New("the box's eph is not an X25519 key")
|
||||
}
|
||||
secret, err := priv.ECDH(eph)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
iv, err1 := d(box.IV)
|
||||
tag, err2 := d(box.Tag)
|
||||
ct, err3 := d(box.Ct)
|
||||
if err := errors.Join(err1, err2, err3); err != nil {
|
||||
return "", fmt.Errorf("the box's fields: %w", err)
|
||||
}
|
||||
gcm, err := newGCM(boxKey(secret, ephDER, priv.PublicKey().Bytes()))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(iv) != gcm.NonceSize() || len(tag) != gcm.Overhead() {
|
||||
return "", errors.New("the box does not open with this key")
|
||||
}
|
||||
plain, err := gcm.Open(nil, iv, append(append([]byte{}, ct...), tag...), nil)
|
||||
if err != nil {
|
||||
return "", errors.New("the box does not open with this key")
|
||||
}
|
||||
return string(plain), nil
|
||||
}
|
||||
|
||||
func publicFromPEM(p string) (*ecdh.PublicKey, error) {
|
||||
block, _ := pem.Decode([]byte(p))
|
||||
if block == nil {
|
||||
return nil, errors.New("not a PEM public key")
|
||||
}
|
||||
k, err := x509.ParsePKIXPublicKey(block.Bytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pub, ok := k.(*ecdh.PublicKey)
|
||||
if !ok || pub.Curve() != ecdh.X25519() {
|
||||
return nil, errors.New("not an X25519 public key")
|
||||
}
|
||||
return pub, nil
|
||||
}
|
||||
|
||||
func privateFromPEM(p string) (*ecdh.PrivateKey, error) {
|
||||
block, _ := pem.Decode([]byte(p))
|
||||
if block == nil {
|
||||
return nil, errors.New("not a PEM private key")
|
||||
}
|
||||
k, err := x509.ParsePKCS8PrivateKey(block.Bytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
priv, ok := k.(*ecdh.PrivateKey)
|
||||
if !ok || priv.Curve() != ecdh.X25519() {
|
||||
return nil, errors.New("not an X25519 private key")
|
||||
}
|
||||
return priv, nil
|
||||
}
|
||||
|
||||
// boxKey is HKDF-SHA256 (RFC 5869) for one 32-byte block: written out because crypto/hkdf needs Go 1.24
|
||||
// and this module promises 1.22; the vectors prove it equals the licence manager's hkdf.Key.
|
||||
func boxKey(secret, ephDER, recipientRaw []byte) []byte {
|
||||
salt := append(append([]byte{}, ephDER...), recipientRaw...)
|
||||
extract := hmac.New(sha256.New, salt)
|
||||
extract.Write(secret)
|
||||
prk := extract.Sum(nil)
|
||||
expand := hmac.New(sha256.New, prk)
|
||||
expand.Write([]byte(info))
|
||||
expand.Write([]byte{1})
|
||||
return expand.Sum(nil)
|
||||
}
|
||||
|
||||
func newGCM(key []byte) (cipher.AEAD, error) {
|
||||
block, err := aes.NewCipher(key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return cipher.NewGCM(block)
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
package seal
|
||||
|
||||
import (
|
||||
"crypto/ecdh"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fixture struct {
|
||||
Recipient KeyPair `json:"recipient"`
|
||||
Sealed []struct {
|
||||
Plaintext string `json:"plaintext"`
|
||||
Box Box `json:"box"`
|
||||
} `json:"sealed"`
|
||||
Vectors []struct {
|
||||
Plaintext string `json:"plaintext"`
|
||||
EphPrivate string `json:"ephPrivate"`
|
||||
IV string `json:"iv"`
|
||||
Box Box `json:"box"`
|
||||
} `json:"vectors"`
|
||||
}
|
||||
|
||||
func readFixture(t *testing.T) fixture {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile("testdata/sealed-by-licence-manager.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var f fixture
|
||||
if err := json.Unmarshal(raw, &f); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(f.Sealed) == 0 || len(f.Vectors) == 0 {
|
||||
t.Fatal("the fixture holds no boxes")
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// Every box the licence manager's own Seal made opens here to its plaintext.
|
||||
func TestABoxTheLicenceManagerSealedOpens(t *testing.T) {
|
||||
f := readFixture(t)
|
||||
for _, s := range f.Sealed {
|
||||
got, err := Open(s.Box, f.Recipient.PrivateKey)
|
||||
if err != nil || got != s.Plaintext {
|
||||
t.Fatalf("opened %q, %v; want %q", got, err, s.Plaintext)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Given the same one-time key and iv, this package seals byte for byte what the licence manager sealed:
|
||||
// the reverse direction, so whatever opens the licence manager's boxes opens these.
|
||||
func TestSealIsByteForByteTheLicenceManagersBox(t *testing.T) {
|
||||
f := readFixture(t)
|
||||
for _, v := range f.Vectors {
|
||||
d := base64.StdEncoding.DecodeString
|
||||
ephRaw, err1 := d(v.EphPrivate)
|
||||
iv, err2 := d(v.IV)
|
||||
if err1 != nil || err2 != nil {
|
||||
t.Fatal(err1, err2)
|
||||
}
|
||||
eph, err := ecdh.X25519().NewPrivateKey(ephRaw)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := sealWith(v.Plaintext, f.Recipient.PublicKey, eph, iv)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != v.Box {
|
||||
t.Fatalf("sealed %+v\nthe licence manager sealed %+v", got, v.Box)
|
||||
}
|
||||
if plain, err := Open(got, f.Recipient.PrivateKey); err != nil || plain != v.Plaintext {
|
||||
t.Fatalf("reopened %q, %v", plain, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A box the agent module's TypeScript sealed (the fixture the licence manager keeps) opens here too.
|
||||
func TestABoxSealedInTypeScriptOpens(t *testing.T) {
|
||||
raw, err := os.ReadFile("testdata/sealed-by-typescript.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var f struct {
|
||||
PrivateKey string `json:"privateKey"`
|
||||
Box Box `json:"box"`
|
||||
Plaintext string `json:"plaintext"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &f); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, err := Open(f.Box, f.PrivateKey); err != nil || got != f.Plaintext {
|
||||
t.Fatalf("opened %q, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// What Seal makes opens with its recipient's key and no other, and a fresh box differs every time.
|
||||
func TestABoxOpensOnlyForItsRecipient(t *testing.T) {
|
||||
a, err := GenerateKeyPair()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := GenerateKeyPair()
|
||||
box, err := Seal("a token", a.PublicKey)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, err := Open(box, a.PrivateKey); err != nil || got != "a token" {
|
||||
t.Fatalf("opened %q, %v", got, err)
|
||||
}
|
||||
if _, err := Open(box, b.PrivateKey); err == nil {
|
||||
t.Fatal("a box opened for another key")
|
||||
}
|
||||
again, _ := Seal("a token", a.PublicKey)
|
||||
if again.Eph == box.Eph || again.IV == box.IV {
|
||||
t.Fatal("two boxes share a one-time key or iv")
|
||||
}
|
||||
}
|
||||
|
||||
// A box changed in any field, or of another version, does not open.
|
||||
func TestATamperedBoxDoesNotOpen(t *testing.T) {
|
||||
f := readFixture(t)
|
||||
good := f.Sealed[1].Box
|
||||
flip := func(s string) string {
|
||||
b, _ := base64.StdEncoding.DecodeString(s)
|
||||
b[len(b)-1] ^= 1
|
||||
return base64.StdEncoding.EncodeToString(b)
|
||||
}
|
||||
for name, box := range map[string]Box{
|
||||
"ct": {V: 1, Eph: good.Eph, IV: good.IV, Tag: good.Tag, Ct: flip(good.Ct)},
|
||||
"tag": {V: 1, Eph: good.Eph, IV: good.IV, Tag: flip(good.Tag), Ct: good.Ct},
|
||||
"iv": {V: 1, Eph: good.Eph, IV: flip(good.IV), Tag: good.Tag, Ct: good.Ct},
|
||||
"eph": {V: 1, Eph: flip(good.Eph), IV: good.IV, Tag: good.Tag, Ct: good.Ct},
|
||||
"version": {V: 2, Eph: good.Eph, IV: good.IV, Tag: good.Tag, Ct: good.Ct},
|
||||
} {
|
||||
if _, err := Open(box, f.Recipient.PrivateKey); err == nil {
|
||||
t.Fatalf("a box with a changed %s opened", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A key that is not X25519 PEM is refused, not misread.
|
||||
func TestAKeyOfAnotherKindIsRefused(t *testing.T) {
|
||||
if _, err := Seal("x", "not a key"); err == nil {
|
||||
t.Fatal("sealed to text")
|
||||
}
|
||||
f := readFixture(t)
|
||||
if _, err := Open(f.Sealed[0].Box, f.Recipient.PublicKey); err == nil {
|
||||
t.Fatal("opened with a public key")
|
||||
}
|
||||
if _, err := Seal("x", f.Recipient.PrivateKey); err == nil {
|
||||
t.Fatal("sealed to a private key")
|
||||
}
|
||||
}
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"about": "Made by the claude-licence-manager's seal.go (mesh-catalog feat/57-config-and-provisioning at 8aa37e8), unchanged but for its two sources of randomness, which the vectors fix. The recipient is a test key derived from a fixed label; it guards nothing.",
|
||||
"recipient": {
|
||||
"publicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VuAyEAkZXnghyfA1BR5rWtvBLmjdZMCbC99xxF3WKHe1E+3XM=\n-----END PUBLIC KEY-----\n",
|
||||
"privateKey": "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VuBCIEIHjV954xXPa72FyGUw5PsWCbsHrliVp/7xoAgG9fGs7q\n-----END PRIVATE KEY-----\n"
|
||||
},
|
||||
"sealed": [
|
||||
{
|
||||
"plaintext": "",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAwN2gt32hv+9VyFh0oQ8oetPBHsZ9dt80DzwPwcB0bHg=",
|
||||
"iv": "0xHCegxP6o48EGLq",
|
||||
"tag": "VYQZm/lBZB0YMPUqYVPKaQ==",
|
||||
"ct": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "a grant sealed by the licence manager",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEA54v96LEb8SwT2uJZCa90/2jcLvEbK/BZpbMS6TxAAW0=",
|
||||
"iv": "xm6Jg4oOnRpRpDGK",
|
||||
"tag": "cmbOzIV63Nv0lPfpTgP2kg==",
|
||||
"ct": "InWe8PZ8hkd7F9QaHujjhdiBECKzLAK22d2aIEXwBhtxIowc/Q=="
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "sk-ant-api03-test-0123456789abcdefghij\n",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAIXdum2FYLzvLDj/Of2253hH8Lu0szU4tLxJzfXTahDs=",
|
||||
"iv": "KMPNa60C/Qjhtp0Z",
|
||||
"tag": "PPa82ZdFyWz9kbLSSGkl3g==",
|
||||
"ct": "Phu3ZsxIWGlBWbYOGzPopncEbc0VhTAloxzeHs4KDBMFZU9Pet2V"
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "ünïcødé ✓ and a NUL \u0000 inside",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAo1WaRyrAJFM6If/cpFgdpZjsC6XebuJtVSugbI+m8RU=",
|
||||
"iv": "H9+WvM6ZYbCYJj6D",
|
||||
"tag": "K5p986nSpeItYK1m4bcVPQ==",
|
||||
"ct": "eyZs3uVeZguNbqSmrMc1rWgxhW3nP0GABH6Ek06/8UuBqw=="
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; ",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAzjZdTMu08I0Z+FWXSr9f++EADfqg3qr2FVt+3Q/hxm0=",
|
||||
"iv": "lTG2VwPA5SwHbZE/",
|
||||
"tag": "Il6ICxoZgZr9dOgeIZOPrg==",
|
||||
"ct": "ZWQ8m1nxrqq0oIGRoeNlUu9/JpTMrxlncTCL5cDqsJrx9NjS0nZX292JJ5p0+BYSVLUN+yGUuBOnH860ywvKK7XhPOc0Yo2nq7X2O3OYn9Za4y+HK6dVDM8zhsS8P3Di/MJCwS+AnremVCaj2k3xHL3oc331RBZZ5w2JfgSX3ouwl2+VbIPyJVSDsxnLLoHsiIR7nB0GNCe0fynhVkoIEwC8IXLnL6m3rjjN3YCQDmVP2UoftniLAypd5ioc9GCtpvUL6ktjs82JkvEtFbHtc+YzIjP6ZnjTj/bvbRHkIqN4MDiu+fvGOr0jtGWO24ecGC+6KtrKGOaEYHDqOLaK3c+UcrDsFYC+cuKCdCNhs5UNhYYylVXE648KFa488eFk"
|
||||
}
|
||||
}
|
||||
],
|
||||
"vectors": [
|
||||
{
|
||||
"plaintext": "",
|
||||
"ephPrivate": "W8iT0SrRDnazANHYsXHC72sM1wzx+/Y8vowREeGGz/A=",
|
||||
"iv": "DD7WV6Y8hLGvaG0b",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAdCCqlFhxbKBpv1JClXceH43AqEvcaM6SGfyXx05UJUs=",
|
||||
"iv": "DD7WV6Y8hLGvaG0b",
|
||||
"tag": "ctb+ZQjx8FyynvT88rfwkQ==",
|
||||
"ct": ""
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "a grant sealed by the licence manager",
|
||||
"ephPrivate": "lzIbkYmIbkKTYDL4dZiAFUFv2h+GWBiqlQfnsVD2An8=",
|
||||
"iv": "xz2FIBXNM3scOAJb",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAS3L49p4KrGMo/NmHZq8osoPbk2ZCQWLoNIKnpB4T5hg=",
|
||||
"iv": "xz2FIBXNM3scOAJb",
|
||||
"tag": "7dC8bYqzsJ/17EC7y6lhCA==",
|
||||
"ct": "FEMfh4HMvMwqx4QE4YMMR3d3g5rNuxa/+u6u+i/aRLed3q8kBw=="
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "sk-ant-api03-test-0123456789abcdefghij\n",
|
||||
"ephPrivate": "tAjRQTQHk6nNJLblzdslDbWmyYm8cE6/G4ipXHR0zlE=",
|
||||
"iv": "BPF1JDQ5mDNhKKr3",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEA1GianODxSXUHYMdUmZLcQztp5ry2/xtwUSeHZWrGFAo=",
|
||||
"iv": "BPF1JDQ5mDNhKKr3",
|
||||
"tag": "uiIIFD7LgPmLV3Y6uleSzg==",
|
||||
"ct": "+maa0GDDOkcyzD0chFQ4a/0NvBrvl23NVqcGv1BEKgmMUenf9HHd"
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "ünïcødé ✓ and a NUL \u0000 inside",
|
||||
"ephPrivate": "QBYUxuk4jwiWKOPEWeE7v6WxY8BDmZZCDwlY1WJ1PSA=",
|
||||
"iv": "u76JO8baqokfnjcK",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAePakOoCmmXLtnN3EykbpqfdbfXpkg04F2djxnJWH220=",
|
||||
"iv": "u76JO8baqokfnjcK",
|
||||
"tag": "sfBPRnwhfP/p7vlsUx+e9w==",
|
||||
"ct": "B9yc0FbdKYDtDeX1jmKa+DBP8fPN1O9SbeYBDn9b3+Bq2Q=="
|
||||
}
|
||||
},
|
||||
{
|
||||
"plaintext": "a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; a longer value, past one AES block; ",
|
||||
"ephPrivate": "2VYOyzqLrb+zWUhVpHA9aV3vMAyRWlNPEq2ASf60F9M=",
|
||||
"iv": "0Gv6ejq07LwbM8cY",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEARSOBczrJ6X1MkxT7Mrd1ryDLmXAMBgYIgpfwBjHz1GA=",
|
||||
"iv": "0Gv6ejq07LwbM8cY",
|
||||
"tag": "Vm+E0myXgdMwA6knJDg/+w==",
|
||||
"ct": "tJwZpUXeQYOu2PG1g9AwIV8RhKKDorUHWQSygEus3xHOpuTwh3+D4j2htIerMB/3AKToQYABeZMyEAOzhL2x+Cg06WlUKrggZjvdX5W7X813KQBgzg0YnnCZTKTYlviDWe1E9TZ3aAauhlcgpBaQXUh/akkDiqeq2Ua5ruY6IGH5HgqWWPYJo3b4zkbArhvdP2HJX9Pn1mQa0BtGMinyutyD0am5vZuemdKGYS1Ao00M81sMXYc2CBy9K48Ofw8GapcmYNHsE0s6NSgNZumnVaR8TQRy/oqEdL3Q6z31hAZMzTdS0ckXg1NCgcgXVuzwNov/D4320E3DxFS3v8Z5p74ZkqAmo/1JcCNJwwQyRCWk/4iJFjP1OKEavOGo+bcp"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"privateKey": "-----BEGIN PRIVATE KEY-----\nMC4CAQAwBQYDK2VuBCIEIAi2NK/bN+p7cqYUwv/kz72TgLdmJUfOHCDZTrMpQzpS\n-----END PRIVATE KEY-----\n",
|
||||
"publicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VuAyEARYvD/w+9ah0KWS9T9pd6Ea6CBymUE48vEVk983QPKyY=\n-----END PUBLIC KEY-----\n",
|
||||
"box": {
|
||||
"v": 1,
|
||||
"eph": "MCowBQYDK2VuAyEAIYlSGrJvF8qSjR1aTFWbEQM/NFbZXrarglN0aRLZZ0E=",
|
||||
"iv": "ABqT/hMukn6+xpjh",
|
||||
"tag": "POzmsWTPHSn9xLMMJJ9Akg==",
|
||||
"ct": "m2u0kuQ0PwrsGelM99Z5aBw6FCd6lFISUbwiK5TzzDw4ZrGtsHEvxytoIeFC"
|
||||
},
|
||||
"plaintext": "a grant sealed by the TypeScript agent module"
|
||||
}
|
||||
+25
@@ -31,6 +31,16 @@ type StateChange struct {
|
||||
Current bool `json:"current"`
|
||||
}
|
||||
|
||||
// StateRevision is one value a bucket still keeps for a key: what was written, at which revision, by which
|
||||
// operation ("put" or "delete", whose value is null) and when (novox/hq ADR 0307).
|
||||
type StateRevision struct {
|
||||
Key string `json:"key"`
|
||||
Value json.RawMessage `json:"value"`
|
||||
Revision uint64 `json:"revision"`
|
||||
Op string `json:"op"`
|
||||
At time.Time `json:"at"`
|
||||
}
|
||||
|
||||
// KeptState is one state this module keeps, or reads.
|
||||
type KeptState struct{ name string }
|
||||
|
||||
@@ -58,6 +68,21 @@ func (s KeptState) Get(key string) (*StateEntry, error) {
|
||||
return &e, json.Unmarshal(raw, &e)
|
||||
}
|
||||
|
||||
// History is every value the bucket still keeps for key, oldest first — as many as the state declared it
|
||||
// keeps, deletes included — and empty for a key it holds nothing for (novox/hq ADR 0307). A read, like Get:
|
||||
// the owner's instances and its readers may ask it, and it writes nothing.
|
||||
func (s KeptState) History(key string) ([]StateRevision, error) {
|
||||
raw, err := s.ask("history", map[string]any{"key": key})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
revisions := []StateRevision{}
|
||||
if len(raw) == 0 || string(raw) == "null" {
|
||||
return revisions, nil
|
||||
}
|
||||
return revisions, json.Unmarshal(raw, &revisions)
|
||||
}
|
||||
|
||||
// Put writes a key — the owner's instances only — and returns the revision written.
|
||||
func (s KeptState) Put(key string, value any) (uint64, error) {
|
||||
raw, err := s.ask("put", map[string]any{"key": key, "value": value})
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
package stdio
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// historying serves a bundle whose one tool, "history", runs run, answering each mesh/state ask with
|
||||
// answer beside the loop, as the runtime does; it returns what the bundle asked.
|
||||
func historying(t *testing.T, answer func(message) map[string]any, run func()) []message {
|
||||
t.Helper()
|
||||
t.Setenv(ServedAs, "issues")
|
||||
inR, inW := io.Pipe()
|
||||
outR, outW := io.Pipe()
|
||||
go func() {
|
||||
_ = ServeOn("", []Tool{{Name: "history", Run: func(map[string]any) (any, error) { run(); return "done", nil }}}, inR, outW)
|
||||
}()
|
||||
var wmu, amu sync.Mutex
|
||||
var asked []message
|
||||
send := func(m map[string]any) {
|
||||
m["jsonrpc"] = "2.0"
|
||||
b, _ := json.Marshal(m)
|
||||
wmu.Lock()
|
||||
defer wmu.Unlock()
|
||||
_, _ = inW.Write(append(b, '\n'))
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
lines := bufio.NewScanner(outR)
|
||||
for lines.Scan() {
|
||||
var m message
|
||||
if json.Unmarshal(lines.Bytes(), &m) != nil {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(m.Method, "mesh/state.") {
|
||||
amu.Lock()
|
||||
asked = append(asked, m)
|
||||
amu.Unlock()
|
||||
a := answer(m)
|
||||
a["id"] = m.ID
|
||||
send(a)
|
||||
continue
|
||||
}
|
||||
if m.Method == "" {
|
||||
close(done)
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
send(map[string]any{"id": 1, "method": "tools/call", "params": map[string]any{"name": "history"}})
|
||||
t.Cleanup(func() { _ = inW.Close() })
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("the bundle did not answer")
|
||||
}
|
||||
amu.Lock()
|
||||
defer amu.Unlock()
|
||||
return asked
|
||||
}
|
||||
|
||||
// History asks mesh/state.history with the state and the key alone, and hands back every kept value oldest
|
||||
// first, a delete with its null value; nothing kept is an empty history; a refusal is the runtime's error
|
||||
// (novox/hq ADR 0307).
|
||||
func TestHistoryIsEveryKeptValueOldestFirst(t *testing.T) {
|
||||
kept := []map[string]any{
|
||||
{"key": "401", "value": map[string]any{"status": "open"}, "revision": 4, "op": "put", "at": "2026-10-10T19:46:00Z"},
|
||||
{"key": "401", "value": map[string]any{"status": "fixing"}, "revision": 9, "op": "put", "at": "2026-10-11T08:01:02Z"},
|
||||
{"key": "401", "value": nil, "revision": 12, "op": "delete", "at": "2026-10-11T09:30:00+02:00"},
|
||||
}
|
||||
var got, none, null []StateRevision
|
||||
var refused error
|
||||
asked := historying(t, func(m message) map[string]any {
|
||||
var p map[string]any
|
||||
_ = json.Unmarshal(m.Params, &p)
|
||||
switch {
|
||||
case p["state"] != "issues":
|
||||
return map[string]any{"error": map[string]any{"code": -32000, "message": "issues was issued no state " + p["state"].(string)}}
|
||||
case p["key"] == "401":
|
||||
return map[string]any{"result": kept}
|
||||
case p["key"] == "402":
|
||||
return map[string]any{"result": nil}
|
||||
}
|
||||
return map[string]any{"result": []any{}}
|
||||
}, func() {
|
||||
var err error
|
||||
if got, err = State("issues").History("401"); err != nil {
|
||||
t.Errorf("history of 401: %v", err)
|
||||
}
|
||||
if none, err = State("issues").History("999"); err != nil {
|
||||
t.Errorf("history of 999: %v", err)
|
||||
}
|
||||
if null, err = State("issues").History("402"); err != nil {
|
||||
t.Errorf("history of 402: %v", err)
|
||||
}
|
||||
var h []StateRevision
|
||||
h, refused = State("elsewhere.issues").History("401")
|
||||
if h != nil {
|
||||
t.Errorf("a refused history handed back %v", h)
|
||||
}
|
||||
})
|
||||
|
||||
if len(asked) != 4 {
|
||||
t.Fatalf("asked %d times, not 4", len(asked))
|
||||
}
|
||||
var p map[string]any
|
||||
_ = json.Unmarshal(asked[0].Params, &p)
|
||||
if asked[0].Method != "mesh/state.history" || p["state"] != "issues" || p["key"] != "401" || len(p) != 2 {
|
||||
t.Fatalf("asked %s %v", asked[0].Method, p)
|
||||
}
|
||||
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("history of 401 is %d values, not 3: %+v", len(got), got)
|
||||
}
|
||||
want := []struct {
|
||||
rev uint64
|
||||
op string
|
||||
value string
|
||||
at time.Time
|
||||
}{
|
||||
{4, "put", `{"status":"open"}`, time.Date(2026, 10, 10, 19, 46, 0, 0, time.UTC)},
|
||||
{9, "put", `{"status":"fixing"}`, time.Date(2026, 10, 11, 8, 1, 2, 0, time.UTC)},
|
||||
{12, "delete", `null`, time.Date(2026, 10, 11, 7, 30, 0, 0, time.UTC)},
|
||||
}
|
||||
for i, w := range want {
|
||||
r := got[i]
|
||||
if r.Key != "401" || r.Revision != w.rev || r.Op != w.op || string(r.Value) != w.value || !r.At.Equal(w.at) {
|
||||
t.Fatalf("value %d is %+v, not %+v", i, r, w)
|
||||
}
|
||||
}
|
||||
if none == nil || len(none) != 0 || null == nil || len(null) != 0 {
|
||||
t.Fatalf("nothing kept read as %v and %v, not an empty history", none, null)
|
||||
}
|
||||
if refused == nil || !strings.Contains(refused.Error(), "issued no state elsewhere.issues") {
|
||||
t.Fatalf("a state not issued answered %v", refused)
|
||||
}
|
||||
}
|
||||
+33
-5
@@ -39,7 +39,8 @@ type Tool struct {
|
||||
Description string
|
||||
// Input is the JSON schema of the arguments; nil is a tool that takes nothing.
|
||||
Input map[string]any
|
||||
Run func(args map[string]any) (any, error)
|
||||
// Run answers one call. Who called is CallerOf(args): the bus user the runtime proved, or Unproven.
|
||||
Run func(args map[string]any) (any, error)
|
||||
}
|
||||
|
||||
type message struct {
|
||||
@@ -65,6 +66,8 @@ type channel struct {
|
||||
subscribed int
|
||||
// watches are this bundle's state watches by the id each was asked with (ADR 0201).
|
||||
watches map[string]func(StateChange) error
|
||||
// provisioned are this bundle's watches of its values, by the id each was asked with (provision.go).
|
||||
provisioned map[string]func(ProvisionChange) error
|
||||
// takers and provers are this bundle's seat work and proof handlers, by worker and subject (ADR 0259).
|
||||
takers map[string]func(Work) error
|
||||
provers map[string]func(subject string, body json.RawMessage) (any, error)
|
||||
@@ -168,7 +171,8 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
|
||||
byName[t.Name] = t
|
||||
}
|
||||
c := &channel{enc: json.NewEncoder(out), waiting: map[string]chan message{}, watches: map[string]func(StateChange) error{},
|
||||
takers: map[string]func(Work) error{}, provers: map[string]func(string, json.RawMessage) (any, error){}}
|
||||
provisioned: map[string]func(ProvisionChange) error{},
|
||||
takers: map[string]func(Work) error{}, provers: map[string]func(string, json.RawMessage) (any, error){}}
|
||||
currentMu.Lock()
|
||||
current = c
|
||||
currentMu.Unlock()
|
||||
@@ -223,6 +227,8 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
|
||||
var p struct {
|
||||
Name string `json:"name"`
|
||||
Arguments map[string]any `json:"arguments"`
|
||||
// Meta is the runtime's, never the caller's: who called, taken from the subject (CallerOf).
|
||||
Meta map[string]any `json:"_meta"`
|
||||
}
|
||||
_ = json.Unmarshal(m.Params, &p)
|
||||
t, ok := byName[p.Name]
|
||||
@@ -232,18 +238,18 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
|
||||
}
|
||||
// Beside the loop, never in it: a tool that emits waits for the runtime's answer, which
|
||||
// arrives on the line this loop reads next — and calls do not queue behind a slow one.
|
||||
go func(id any, t Tool, args map[string]any) {
|
||||
go func(id any, t Tool, args map[string]any, caller string) {
|
||||
if args == nil {
|
||||
args = map[string]any{}
|
||||
}
|
||||
result, err := t.Run(args)
|
||||
result, err := runAs(t, caller, args)
|
||||
if err != nil {
|
||||
c.say(map[string]any{"id": id, "result": map[string]any{"content": []map[string]any{{"type": "text", "text": err.Error()}}, "isError": true}})
|
||||
return
|
||||
}
|
||||
text, _ := json.Marshal(result)
|
||||
c.say(map[string]any{"id": id, "result": map[string]any{"content": []map[string]any{{"type": "text", "text": string(text)}}}})
|
||||
}(id, t, p.Arguments)
|
||||
}(id, t, p.Arguments, callerIn(p.Meta))
|
||||
case "mesh/event":
|
||||
// One event the runtime read from this module's consumer (ADR 0198), answered once every
|
||||
// handler it is for has taken it; an error is answered as one, so the runtime does not
|
||||
@@ -294,6 +300,28 @@ func ServeOn(name string, tools []Tool, in io.Reader, out io.Writer) error {
|
||||
}
|
||||
c.say(map[string]any{"id": id, "result": map[string]any{}})
|
||||
}(id, *p.Change)
|
||||
case "mesh/provision":
|
||||
// One change to a value of this module's that this bundle watches (provision.go), named by the
|
||||
// watch's id, answered once its handler has taken it. Beside the loop, like a tool call.
|
||||
var p struct {
|
||||
Watch string `json:"watch"`
|
||||
Change *ProvisionChange `json:"change"`
|
||||
}
|
||||
_ = json.Unmarshal(m.Params, &p)
|
||||
c.mu.Lock()
|
||||
handle := c.provisioned[p.Watch]
|
||||
c.mu.Unlock()
|
||||
if handle == nil || p.Change == nil {
|
||||
c.say(map[string]any{"id": id, "error": map[string]any{"code": -32602, "message": "mesh/provision names no watch of this bundle"}})
|
||||
continue
|
||||
}
|
||||
go func(id any, change ProvisionChange) {
|
||||
if err := handle(change); err != nil {
|
||||
c.say(map[string]any{"id": id, "error": map[string]any{"code": -32000, "message": err.Error()}})
|
||||
return
|
||||
}
|
||||
c.say(map[string]any{"id": id, "result": map[string]any{}})
|
||||
}(id, *p.Change)
|
||||
case "mesh/work":
|
||||
// One piece of a seat's work this bundle takes (novox/hq ADR 0259): taken by an answer that is
|
||||
// not an error, offered again after one. Beside the loop, like a tool call.
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@novox/mesh-sdk",
|
||||
"version": "0.1.13",
|
||||
"version": "0.1.14",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@novox/mesh-sdk",
|
||||
"version": "0.1.13",
|
||||
"version": "0.1.14",
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.0.0",
|
||||
"typescript": "^5.6.0"
|
||||
|
||||
+5
-3
@@ -1,18 +1,20 @@
|
||||
{
|
||||
"name": "@novox/mesh-sdk",
|
||||
"version": "0.1.13",
|
||||
"version": "0.1.14",
|
||||
"description": "The stable spine a Novox Mesh module's own code builds against.",
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": "./dist/index.js",
|
||||
"./contracts": "./dist/contracts/index.js",
|
||||
"./provisioner": "./dist/provisioner/index.js",
|
||||
"./provider-loop": "./dist/provider-loop/index.js",
|
||||
"./provisioner": "./dist/provider-loop/index.js",
|
||||
"./tools": "./dist/tools/index.js",
|
||||
"./messaging": "./dist/messaging/index.js",
|
||||
"./events": "./dist/events/index.js",
|
||||
"./primitives": "./dist/primitives/index.js",
|
||||
"./stdio": "./dist/stdio/index.js",
|
||||
"./state": "./dist/state/index.js"
|
||||
"./state": "./dist/state/index.js",
|
||||
"./seal": "./dist/seal/index.js"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
|
||||
+5
-2
@@ -1,13 +1,16 @@
|
||||
// @novox/mesh-sdk — the stable spine a module's own code builds against (novox/hq ADR 0039).
|
||||
//
|
||||
// Import the area you need directly (`@novox/mesh-sdk/tools`, `/provisioner`, …) or the whole
|
||||
// Import the area you need directly (`@novox/mesh-sdk/tools`, `/provider-loop`, …) or the whole
|
||||
// surface from here. Everything specific to one module — its API client, its tool implementations,
|
||||
// its create-a-resource adapter — lives in the module, never here.
|
||||
|
||||
export * from "./contracts/index.js";
|
||||
export * as tools from "./tools/index.js";
|
||||
export * as provisioner from "./provisioner/index.js";
|
||||
export * as providerLoop from "./provider-loop/index.js";
|
||||
/** @deprecated The provider loop's name before design 57 (novox/hq issue 678): `providerLoop`. */
|
||||
export * as provisioner from "./provider-loop/index.js";
|
||||
export * as messaging from "./messaging/index.js";
|
||||
export * as events from "./events/index.js";
|
||||
export * as primitives from "./primitives/index.js";
|
||||
export * as state from "./state/index.js";
|
||||
export * as seal from "./seal/index.js";
|
||||
|
||||
@@ -1,3 +1,13 @@
|
||||
// The provider loop: the reconcile harness every provider on the controller's grant files shares.
|
||||
// Called `provisioner` until design 57 (novox/hq issue 678): that word is mesh-provisioner's, which
|
||||
// places every value a module needs (ADR 0349), so the loop is named for what it is. The old path
|
||||
// `@novox/mesh-sdk/provisioner` and the names `runProvisioner`, `provisionerTools`, `Provisioner` and
|
||||
// `ProvisionerOptions` stay as aliases of these until the providers that import them move to the Go
|
||||
// helper that answers consumers over the bus (mesh-sdk go/provider, design 57 B2 and B3) and this loop
|
||||
// retires with the grant files. The events (`provisioner.failing`, `.recovered`, `.retirement`) and the
|
||||
// tool names (`provisioner_retirement`, …) are what the controller and the operator read on the bus,
|
||||
// and keep their names.
|
||||
//
|
||||
// The reconcile harness every provider shares. The loop below is identical for postgres, redis,
|
||||
// minio and umami: read the contributions the mesh delivered, bring each consumer's resource into
|
||||
// being through the provider's adapter under the login and password the mesh minted, and retire
|
||||
@@ -10,10 +20,20 @@
|
||||
// does not hand one back — the consumer already receives its copy through the mesh's own channel.
|
||||
// This is why the old $MESH_SEAL_KEY, the symmetric seal, and the *.grant.json / *.credential files
|
||||
// are gone: they described a second, contradictory credential path the mesh does not have.
|
||||
//
|
||||
// **A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224).** A consumer
|
||||
// whose create, check or secret has failed without one success in between for `failingAfterMs` is
|
||||
// announced as `provisioner.failing` — naming the consumer, its machine and the class of error — and
|
||||
// again every `sayAgainEveryMs` while it lasts; the first success after that is
|
||||
// `provisioner.recovered`, as is a consumer the mesh stopped asking for, and the first success for
|
||||
// each consumer since this process started. The controller keeps the newest per provider and consumer
|
||||
// and `status` names it. Until novox/hq issue 272 only the Go copy of this loop (postgres, keycloak)
|
||||
// said it; every provider on this one failed its consumers in its journal alone.
|
||||
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import type { ToolDefinition } from "../contracts/index.js";
|
||||
import { emit } from "../events/index.js";
|
||||
import { registerModuleTools } from "../tools/index.js";
|
||||
|
||||
/** One consumer's resource to bring into being — everything the mesh derived and delivered. */
|
||||
@@ -86,6 +106,9 @@ export interface Adapter {
|
||||
/** Optional: every consumer the backend holds ACTIVE that the mesh made, read at start, so one the
|
||||
* mesh stopped asking for while this process was down is still retired (by the same rules). */
|
||||
held?(): Promise<string[]>;
|
||||
/** Optional: the class of one of the adapter's own errors (novox/hq ADR 0224), when its code knows
|
||||
* better than the error's words; "" or undefined falls back to `classOf` of the text. */
|
||||
classify?(err: unknown): string | undefined;
|
||||
}
|
||||
|
||||
/** One retired consumer, as `provisioner_retirement` answers it. */
|
||||
@@ -127,9 +150,9 @@ export interface Act {
|
||||
readonly via?: string;
|
||||
}
|
||||
|
||||
/** The running provisioner: a stop function, as before, that also answers and takes the person's
|
||||
/** The running provider loop: a stop function, as before, that also answers and takes the person's
|
||||
* acts on retirement (novox/hq ADR 0230). */
|
||||
export type Provisioner = (() => void) & {
|
||||
export type ProviderLoop = (() => void) & {
|
||||
retirement(): Promise<Retirement>;
|
||||
approve(consumers: readonly string[], act: Act): Promise<{ retired: string[] }>;
|
||||
reject(consumers: readonly string[], act: Act): Promise<{ kept: string[] }>;
|
||||
@@ -139,7 +162,22 @@ export type Provisioner = (() => void) & {
|
||||
/** The one event a provider says about retirement (novox/hq ADR 0230). */
|
||||
export const RETIREMENT_EVENT = "provisioner.retirement";
|
||||
|
||||
export interface ProvisionerOptions {
|
||||
/** The events a provider's standing toward one consumer is announced as (novox/hq ADR 0224). The
|
||||
* controller derives the permission to emit them for every module that receives contributions. */
|
||||
export const FAILING_EVENT = "provisioner.failing";
|
||||
export const RECOVERED_EVENT = "provisioner.recovered";
|
||||
|
||||
/** The classes a failing consumer is announced with: what a person reading `status` needs to know
|
||||
* before reading the journal (novox/hq ADR 0224). An adapter may say better (`classify`). */
|
||||
export const CLASS_CREDENTIALS = "credentials-rejected";
|
||||
export const CLASS_UNREACHABLE = "unreachable";
|
||||
export const CLASS_SECRET = "secret-unreadable";
|
||||
export const CLASS_REFUSED = "refused";
|
||||
|
||||
/** How long a consumer fails without one success before it is announced failing (ADR 0224). */
|
||||
export const FAILING_AFTER_MS = 5 * 60_000;
|
||||
|
||||
export interface ProviderLoopOptions {
|
||||
/** The contributions file the mesh writes for this resource (the module's `receives` path).
|
||||
* Defaults to $MESH_RECEIVES. */
|
||||
receives?: string;
|
||||
@@ -165,10 +203,16 @@ export interface ProvisionerOptions {
|
||||
/** Nor a set larger than this share of the consumers held, where more than one is held. Defaults
|
||||
* to 0.5. */
|
||||
retireFraction?: number;
|
||||
/** How often a waiting retirement is said again, in ms. Defaults to 15 minutes. */
|
||||
/** How often a waiting retirement, and a consumer still failing, is said again, in ms. Defaults to
|
||||
* 15 minutes. */
|
||||
sayAgainEveryMs?: number;
|
||||
/** Publishes one of the provider's events (`provisioner.retirement`); usually the SDK's `emit`.
|
||||
* Without it the provider's retirement is said in its journal only. */
|
||||
/** How long, in ms, a consumer fails without one success before it is announced
|
||||
* `provisioner.failing` (novox/hq ADR 0224). Defaults to FAILING_AFTER_MS, five minutes. */
|
||||
failingAfterMs?: number;
|
||||
/** Publishes one of the provider's events (`provisioner.failing`, `provisioner.recovered`,
|
||||
* `provisioner.retirement`). Defaults to the SDK's `emit`, so every provider says them on the bus
|
||||
* without asking (novox/hq issue 272); a failed publish is said in the journal and never fails
|
||||
* the act it reports. */
|
||||
announce?: (event: string, body: Record<string, unknown>) => Promise<void> | void;
|
||||
/** The machine this provider runs on, said in each event. Defaults to $MESH_NODE. */
|
||||
node?: string;
|
||||
@@ -216,7 +260,7 @@ interface Contribution {
|
||||
* `retireFraction` of those held where more than one is held, is not retired without a person: the
|
||||
* provider waits for approve or reject. RETIRED → DELETED only through `provisioner_delete`.
|
||||
*/
|
||||
export function runProvisioner(resource: string, adapter: Adapter, opts: ProvisionerOptions = {}): Provisioner {
|
||||
export function runProviderLoop(resource: string, adapter: Adapter, opts: ProviderLoopOptions = {}): ProviderLoop {
|
||||
const receives = opts.receives ?? envOrThrow("MESH_RECEIVES");
|
||||
const everyMs = opts.everyMs ?? 5000;
|
||||
const verifyEveryMs = opts.verifyEveryMs ?? 60_000;
|
||||
@@ -227,6 +271,8 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
const retireAtOnce = opts.retireAtOnce ?? 3;
|
||||
const retireFraction = opts.retireFraction ?? 0.5;
|
||||
const sayAgainEveryMs = opts.sayAgainEveryMs ?? 15 * 60_000;
|
||||
const failingAfterMs = opts.failingAfterMs ?? FAILING_AFTER_MS;
|
||||
const announce = opts.announce ?? ((event: string, body: Record<string, unknown>) => emit(event, body));
|
||||
const node = opts.node ?? process.env.MESH_NODE ?? "";
|
||||
const say = (line: string): void => console.error(`[provisioner:${resource}] ${line}`);
|
||||
// Retirement (novox/hq ADR 0230). The unasked set last seen and in how many consecutive readable
|
||||
@@ -245,6 +291,13 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
// itself. Each repeat doubles the wait before asking again, so such a bug costs one re-apply an
|
||||
// hour rather than one a minute, and it is said loudly rather than quietly repeated.
|
||||
const lost = new Map<string, { times: number; nextAt: number }>();
|
||||
// Each consumer's unbroken run of failures — since when, how often, the last error, and when it was
|
||||
// last announced (0: not yet) — and every consumer that has succeeded since this process started
|
||||
// (novox/hq ADR 0224).
|
||||
const trouble = new Map<string, {
|
||||
node?: string; since: number; attempts: number; cls: string; text: string; saidAt: number;
|
||||
}>();
|
||||
const cleared = new Set<string>();
|
||||
|
||||
// login (`as`) -> what was last applied: the hash that says whether it changed, and the
|
||||
// derivation to hand back on withdrawal, which by then is no longer in the file.
|
||||
@@ -311,6 +364,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
`file and who this process runs as (novox/hq issue 225)`,
|
||||
);
|
||||
}
|
||||
await failed(g.as, g.node, CLASS_SECRET, `secret not readable (${g.secret}): ${err}`);
|
||||
continue;
|
||||
}
|
||||
waiting.delete(g.as);
|
||||
@@ -328,11 +382,15 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
try {
|
||||
if (await withTimeout(adapter.holds!(p), holdsTimeoutMs)) {
|
||||
lost.delete(g.as);
|
||||
await succeeded(g.as);
|
||||
continue;
|
||||
}
|
||||
} catch (err) {
|
||||
// Unable to ask is not evidence of loss. Kept as applied, asked again next time.
|
||||
console.error(`[provisioner:${resource}] ${g.as}: could not check the backend, will ask again: ${scrub(err, password)}`);
|
||||
// Unable to ask is not evidence of loss. Kept as applied, asked again next time — but a
|
||||
// check that keeps failing is a consumer failed all the same (ADR 0224, issue 179).
|
||||
const text = scrub(err, password);
|
||||
console.error(`[provisioner:${resource}] ${g.as}: could not check the backend, will ask again: ${text}`);
|
||||
await failed(g.as, g.node, classed(err, text), text);
|
||||
// A backend that did not answer in time will not answer for the next consumer either, and
|
||||
// each would cost a timeout, so the rest of this pass is not asked. Any other failure may
|
||||
// be this consumer's alone, and the others are still asked.
|
||||
@@ -356,6 +414,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
await adapter.reenable({ as: g.as, derived: was.derived ?? p.derived });
|
||||
}
|
||||
await adapter.create(p);
|
||||
await succeeded(g.as);
|
||||
applied.set(g.as, { hash: h, derived: p.derived, node: g.node });
|
||||
if (retired.has(g.as)) {
|
||||
// Asked for again: the ordinary create re-enabled it and cleared its mark (ADR 0230).
|
||||
@@ -376,7 +435,9 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`[provisioner:${resource}] ${g.as}: create failed, will retry: ${scrub(err, password)}`);
|
||||
const text = scrub(err, password);
|
||||
console.error(`[provisioner:${resource}] ${g.as}: create failed, will retry: ${text}`);
|
||||
await failed(g.as, g.node, classed(err, text), text);
|
||||
// A failed create is not a disagreement: asked again at the next check, the count unchanged.
|
||||
if (reapplying !== undefined) lost.set(g.as, { times: reapplying - 1, nextAt: 0 });
|
||||
}
|
||||
@@ -386,6 +447,12 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
// `stablePasses` passes, and within the bound, or with a person's word (novox/hq ADR 0230).
|
||||
// Only consumers this harness applied, or the backend says the mesh made, are touched.
|
||||
const unasked = [...applied.keys()].filter((as) => !wantByAs.has(as)).sort();
|
||||
// A consumer the mesh stopped asking for that this provider holds nothing for is no longer failed
|
||||
// by anyone: said, so the standing the controller keeps for it is cleared rather than left naming a
|
||||
// consumer that is gone. One still held is said recovered when it is retired (ADR 0224, 0230).
|
||||
for (const as of [...trouble.keys()]) {
|
||||
if (!wantByAs.has(as) && !applied.has(as)) await recovered(as, "no longer asked for");
|
||||
}
|
||||
const key = unasked.join(",");
|
||||
if (pending && pending.consumers.join(",") !== key) {
|
||||
say(`the retirement waiting for a person (${pending.consumers.join(", ")}) is no longer what the mesh says; settled, nothing retired`);
|
||||
@@ -464,6 +531,7 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
}
|
||||
applied.delete(as);
|
||||
lost.delete(as);
|
||||
await recovered(as, "retired");
|
||||
retired.set(as, { node: was.node, at: at.getTime(), why, derived: was.derived, access: adapter.retire ? "disabled" : "kept" });
|
||||
done.push(as);
|
||||
} catch (err) {
|
||||
@@ -485,16 +553,86 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
|
||||
// Said on the bus, never failing the act it reports (the journal still has it).
|
||||
async function tell(change: string, body: Record<string, unknown>): Promise<void> {
|
||||
if (!opts.announce) return;
|
||||
await said(RETIREMENT_EVENT, {
|
||||
provider: resource, "provider-node": node, change, bound: bound(applied.size), at: iso(Date.now()), ...body,
|
||||
}, change);
|
||||
}
|
||||
|
||||
async function said(event: string, body: Record<string, unknown>, what = ""): Promise<void> {
|
||||
try {
|
||||
await opts.announce(RETIREMENT_EVENT, {
|
||||
provider: resource, "provider-node": node, change, bound: bound(applied.size), at: iso(Date.now()), ...body,
|
||||
});
|
||||
await announce(event, body);
|
||||
} catch (err) {
|
||||
say(`could not announce ${RETIREMENT_EVENT} ${change}: ${err}`);
|
||||
say(`could not announce ${event}${what ? ` ${what}` : ""}: ${err}`);
|
||||
}
|
||||
}
|
||||
|
||||
// An error's class: the adapter's word when it has one, else read from the text (ADR 0224).
|
||||
function classed(err: unknown, text: string): string {
|
||||
try {
|
||||
const own = adapter.classify?.(err);
|
||||
if (own) return own;
|
||||
} catch {
|
||||
// An adapter that cannot class its own error is read by the words.
|
||||
}
|
||||
// A check that did not answer in time is a backend out of reach, whatever its words.
|
||||
if (err instanceof HoldsTimeout) return CLASS_UNREACHABLE;
|
||||
return classOf(text);
|
||||
}
|
||||
|
||||
// One more failure in a consumer's unbroken run, announced once the run has lasted failingAfterMs,
|
||||
// then again every sayAgainEveryMs while it lasts (novox/hq ADR 0224).
|
||||
async function failed(as: string, consumerNode: string | undefined, cls: string, text: string): Promise<void> {
|
||||
const now = Date.now();
|
||||
let s = trouble.get(as);
|
||||
if (!s) {
|
||||
s = { since: now, attempts: 0, cls, text, saidAt: 0 };
|
||||
trouble.set(as, s);
|
||||
}
|
||||
s.node = consumerNode;
|
||||
s.cls = cls;
|
||||
s.text = text;
|
||||
s.attempts++;
|
||||
if (now - s.since < failingAfterMs) return;
|
||||
if (s.saidAt !== 0 && now - s.saidAt < sayAgainEveryMs) return;
|
||||
if (s.saidAt === 0) {
|
||||
say(
|
||||
`${as}: FAILING for ${Math.round((now - s.since) / 1000)}s (${s.attempts} attempts, ${cls}): ${text}. ` +
|
||||
`Announced as ${FAILING_EVENT}; \`status\` names it until it succeeds (novox/hq ADR 0224)`,
|
||||
);
|
||||
}
|
||||
s.saidAt = now;
|
||||
await said(FAILING_EVENT, {
|
||||
provider: resource, "provider-node": node, consumer: as, node: consumerNode ?? "",
|
||||
class: cls, error: clip(text), since: iso(s.since), attempts: s.attempts,
|
||||
});
|
||||
}
|
||||
|
||||
// A success ends a consumer's run of failures; one that was announced is announced recovered. And
|
||||
// the first success for a consumer since this process started is announced too, failing or not: a
|
||||
// provider that announced a failure and was restarted has forgotten it, and without this the
|
||||
// controller would name the consumer failing for ever after it recovered unheard.
|
||||
async function succeeded(as: string): Promise<void> {
|
||||
if (!trouble.has(as) && !cleared.has(as)) {
|
||||
cleared.add(as);
|
||||
await said(RECOVERED_EVENT, { provider: resource, "provider-node": node, consumer: as, why: "first-success" });
|
||||
return;
|
||||
}
|
||||
cleared.add(as);
|
||||
await recovered(as, "");
|
||||
}
|
||||
|
||||
async function recovered(as: string, why: string): Promise<void> {
|
||||
const s = trouble.get(as);
|
||||
if (!s) return;
|
||||
trouble.delete(as);
|
||||
if (s.saidAt === 0) return; // never announced, so there is nothing to take back
|
||||
if (!why) say(`${as}: recovered after ${Math.round((Date.now() - s.since) / 1000)}s and ${s.attempts} failed attempt(s)`);
|
||||
await said(RECOVERED_EVENT, {
|
||||
provider: resource, "provider-node": node, consumer: as, node: s.node ?? "",
|
||||
since: iso(s.since), attempts: s.attempts, ...(why ? { why } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
// What the backend holds that the mesh made: active ones become held, so one the mesh stopped asking
|
||||
// for while this process was down is still retired, by the same rules; retired ones are known.
|
||||
async function seed(): Promise<void> {
|
||||
@@ -629,8 +767,8 @@ export function runProvisioner(resource: string, adapter: Adapter, opts: Provisi
|
||||
const stop = () => {
|
||||
stopped = true;
|
||||
};
|
||||
const handle: Provisioner = Object.assign(stop, { retirement, approve, reject, deleteRetired });
|
||||
if (opts.module) registerModuleTools(opts.module, () => provisionerTools(handle));
|
||||
const handle: ProviderLoop = Object.assign(stop, { retirement, approve, reject, deleteRetired });
|
||||
if (opts.module) registerModuleTools(opts.module, () => providerLoopTools(handle));
|
||||
return handle;
|
||||
}
|
||||
|
||||
@@ -639,7 +777,7 @@ const text = (v: unknown): string => (typeof v === "string" ? v : "");
|
||||
|
||||
/** The four tools every provider serves for retirement (novox/hq ADR 0230), the controller's `retire`
|
||||
* and `cleanup` verbs asking them per machine. */
|
||||
export function provisionerTools(p: Provisioner): ToolDefinition[] {
|
||||
export function providerLoopTools(p: ProviderLoop): ToolDefinition[] {
|
||||
const act = { type: "string" };
|
||||
return [
|
||||
{
|
||||
@@ -713,6 +851,27 @@ async function readContributions(
|
||||
return (doc.given as Contribution[]).filter((g) => g.as && g.secret);
|
||||
}
|
||||
|
||||
/** An error's class read from its text — the words the backends the mesh runs use (novox/hq ADR
|
||||
* 0224): the same words, in the same order, as the Go copy of this loop reads. */
|
||||
export function classOf(text: string): string {
|
||||
const t = text.toLowerCase();
|
||||
const credentials = ["invalid_grant", "invalid user credentials", "password authentication failed",
|
||||
"authentication failed", "unauthorized", " 401"];
|
||||
if (credentials.some((w) => t.includes(w))) return CLASS_CREDENTIALS;
|
||||
const unreachable = ["connection refused", "no such host", "i/o timeout", "deadline exceeded",
|
||||
"connection reset", "network is unreachable", "no route to host", "eof",
|
||||
// Node's own words for the same faults.
|
||||
"econnrefused", "enotfound", "etimedout", "econnreset", "ehostunreach", "enetunreach", "eai_again"];
|
||||
if (unreachable.some((w) => t.includes(w))) return CLASS_UNREACHABLE;
|
||||
return CLASS_REFUSED;
|
||||
}
|
||||
|
||||
/** An announced error kept to what belongs in a status line. */
|
||||
function clip(text: string): string {
|
||||
const most = 300;
|
||||
return text.length <= most ? text : `${text.slice(0, most)}…`;
|
||||
}
|
||||
|
||||
/** A check that did not answer in time: the backend, not the consumer, is the likely cause. */
|
||||
class HoldsTimeout extends Error {}
|
||||
|
||||
@@ -751,3 +910,12 @@ function envOrThrow(name: string): string {
|
||||
if (!v) throw new Error(`${name} is not set — the provisioner cannot run without it`);
|
||||
return v;
|
||||
}
|
||||
|
||||
/** @deprecated The provider loop's name before design 57: {@link runProviderLoop}. */
|
||||
export const runProvisioner = runProviderLoop;
|
||||
/** @deprecated {@link providerLoopTools}. */
|
||||
export const provisionerTools = providerLoopTools;
|
||||
/** @deprecated {@link ProviderLoop}. */
|
||||
export type Provisioner = ProviderLoop;
|
||||
/** @deprecated {@link ProviderLoopOptions}. */
|
||||
export type ProvisionerOptions = ProviderLoopOptions;
|
||||
@@ -0,0 +1,90 @@
|
||||
// The sealed box itself (novox/hq ADR 0183, ADR 0206), with its two sources of randomness as arguments so
|
||||
// the tests can prove it byte for byte against the Go package's vectors. Modules import `./index.js`.
|
||||
|
||||
import {
|
||||
createCipheriv,
|
||||
createDecipheriv,
|
||||
createPrivateKey,
|
||||
createPublicKey,
|
||||
diffieHellman,
|
||||
hkdfSync,
|
||||
type KeyObject,
|
||||
} from "node:crypto";
|
||||
|
||||
/** A value sealed to one recipient, as it travels in JSON; every field but `v` is standard base64. */
|
||||
export interface SealedBox {
|
||||
readonly v: number;
|
||||
/** The one-time public key, SPKI DER. */
|
||||
readonly eph: string;
|
||||
readonly iv: string;
|
||||
readonly tag: string;
|
||||
readonly ct: string;
|
||||
}
|
||||
|
||||
/** The only box version `open` opens. */
|
||||
export const VERSION = 1;
|
||||
|
||||
const INFO = "novox-mesh sealed box v1";
|
||||
const IV_BYTES = 12;
|
||||
const TAG_BYTES = 16;
|
||||
|
||||
function x25519(key: KeyObject, what: string): KeyObject {
|
||||
if (key.asymmetricKeyType !== "x25519") throw new Error(`not an X25519 ${what} key`);
|
||||
return key;
|
||||
}
|
||||
|
||||
export function publicFromPem(pem: string): KeyObject {
|
||||
return x25519(createPublicKey({ key: pem, format: "pem" }), "public");
|
||||
}
|
||||
|
||||
export function privateFromPem(pem: string): KeyObject {
|
||||
return x25519(createPrivateKey({ key: pem, format: "pem" }), "private");
|
||||
}
|
||||
|
||||
/** The recipient's raw 32-byte public key: the last 32 bytes of its SPKI DER. */
|
||||
function raw(pub: KeyObject): Buffer {
|
||||
const der = pub.export({ format: "der", type: "spki" });
|
||||
return der.subarray(der.length - 32);
|
||||
}
|
||||
|
||||
function boxKey(secret: Buffer, ephDer: Buffer, recipientRaw: Buffer): Buffer {
|
||||
return Buffer.from(hkdfSync("sha256", secret, Buffer.concat([ephDer, recipientRaw]), INFO, 32));
|
||||
}
|
||||
|
||||
/** `seal` with the one-time private key and the iv given. */
|
||||
export function sealWith(plaintext: string, recipientPem: string, eph: KeyObject, iv: Buffer): SealedBox {
|
||||
const recipient = publicFromPem(recipientPem);
|
||||
x25519(eph, "one-time");
|
||||
if (iv.length !== IV_BYTES) throw new Error(`an iv of ${iv.length} bytes, not ${IV_BYTES}`);
|
||||
const secret = diffieHellman({ privateKey: eph, publicKey: recipient });
|
||||
const ephDer = createPublicKey(eph).export({ format: "der", type: "spki" });
|
||||
const cipher = createCipheriv("aes-256-gcm", boxKey(secret, ephDer, raw(recipient)), iv);
|
||||
const ct = Buffer.concat([cipher.update(plaintext, "utf8"), cipher.final()]);
|
||||
return {
|
||||
v: VERSION,
|
||||
eph: ephDer.toString("base64"),
|
||||
iv: iv.toString("base64"),
|
||||
tag: cipher.getAuthTag().toString("base64"),
|
||||
ct: ct.toString("base64"),
|
||||
};
|
||||
}
|
||||
|
||||
/** Open a box with the recipient's PEM private key; throws for a box to another key, changed, or of another version. */
|
||||
export function open(box: SealedBox, privatePem: string): string {
|
||||
if (box.v !== VERSION) throw new Error(`a sealed box of version ${box.v}, not ${VERSION}`);
|
||||
const priv = privateFromPem(privatePem);
|
||||
const ephDer = Buffer.from(box.eph, "base64");
|
||||
const eph = x25519(createPublicKey({ key: ephDer, format: "der", type: "spki" }), "one-time");
|
||||
const iv = Buffer.from(box.iv, "base64");
|
||||
const tag = Buffer.from(box.tag, "base64");
|
||||
// node accepts a shorter tag than 16 bytes; the box never has one, so a short one is refused here.
|
||||
if (iv.length !== IV_BYTES || tag.length !== TAG_BYTES) throw new Error("the box does not open with this key");
|
||||
const secret = diffieHellman({ privateKey: priv, publicKey: eph });
|
||||
const decipher = createDecipheriv("aes-256-gcm", boxKey(secret, ephDer, raw(createPublicKey(priv))), iv);
|
||||
decipher.setAuthTag(tag);
|
||||
try {
|
||||
return Buffer.concat([decipher.update(Buffer.from(box.ct, "base64")), decipher.final()]).toString("utf8");
|
||||
} catch {
|
||||
throw new Error("the box does not open with this key");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
// Sealing to one recipient (novox/hq ADR 0183, ADR 0206): X25519 for the agreement, HKDF-SHA256 for the
|
||||
// key, AES-256-GCM for the box. The TypeScript twin of the Go package `go/seal`, byte for byte: the same
|
||||
// fixtures prove both. A box is `{ v: 1, eph, iv, tag, ct }`; keys travel as PEM (SPKI public, PKCS #8
|
||||
// private).
|
||||
|
||||
import { generateKeyPairSync, randomBytes } from "node:crypto";
|
||||
import { sealWith, open, VERSION, type SealedBox } from "./box.js";
|
||||
|
||||
export { open, VERSION, type SealedBox };
|
||||
|
||||
/** A recipient's keypair as the two PEM strings it is kept and sent as. */
|
||||
export interface KeyPair {
|
||||
readonly publicKey: string;
|
||||
readonly privateKey: string;
|
||||
}
|
||||
|
||||
/** A fresh X25519 keypair, PEM-encoded. */
|
||||
export function generateKeyPair(): KeyPair {
|
||||
const { publicKey, privateKey } = generateKeyPairSync("x25519");
|
||||
return {
|
||||
publicKey: publicKey.export({ format: "pem", type: "spki" }) as string,
|
||||
privateKey: privateKey.export({ format: "pem", type: "pkcs8" }) as string,
|
||||
};
|
||||
}
|
||||
|
||||
/** Seal plaintext to the recipient's PEM public key, with a fresh one-time key and iv. */
|
||||
export function seal(plaintext: string, recipientPem: string): SealedBox {
|
||||
return sealWith(plaintext, recipientPem, generateKeyPairSync("x25519").privateKey, randomBytes(12));
|
||||
}
|
||||
+8
-7
@@ -8,7 +8,7 @@ import { mkdtemp, writeFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { runProvisioner, provisionerTools, RETIREMENT_EVENT, type Provisioner } from "../dist/provisioner/index.js";
|
||||
import { runProviderLoop, providerLoopTools, RETIREMENT_EVENT, type ProviderLoop } from "../dist/provider-loop/index.js";
|
||||
import { resetTools, collectTools } from "../dist/tools/index.js";
|
||||
|
||||
const EVERY = 10;
|
||||
@@ -32,7 +32,7 @@ interface World {
|
||||
reenabled: string[];
|
||||
events: Record<string, unknown>[];
|
||||
said: string[];
|
||||
p: Provisioner;
|
||||
p: ProviderLoop;
|
||||
done(): void;
|
||||
}
|
||||
|
||||
@@ -62,9 +62,10 @@ async function world(
|
||||
...(opts.retire === false ? {} : { async retire(p: { as: string; why: string }) { w.retired.push({ as: p.as, why: p.why }); } }),
|
||||
...(opts.held ? { async held() { return opts.held!; } } : {}),
|
||||
};
|
||||
w.p = runProvisioner("postgres-database", adapter, {
|
||||
w.p = runProviderLoop("postgres-database", adapter, {
|
||||
receives, everyMs: EVERY, node: "anchor", stableForMs: opts.stableForMs ?? 0,
|
||||
announce: (event, body) => { assert.equal(event, RETIREMENT_EVENT); w.events.push(body); },
|
||||
// Only retirement is these tests'; the standing (provisioner.failing, .recovered) is standing.test.ts's.
|
||||
announce: (event, body) => { if (event === RETIREMENT_EVENT) w.events.push(body); },
|
||||
});
|
||||
// A pass is counted by the file being read: wait wall time for n passes and some slack.
|
||||
w.done = () => { w.p(); console.error = was; };
|
||||
@@ -167,7 +168,7 @@ test("five passes are not enough: the same set must also hold for stableForMs",
|
||||
});
|
||||
|
||||
test("the default hold is ten minutes", async () => {
|
||||
const { STABLE_FOR_MS } = await import("../dist/provisioner/index.js");
|
||||
const { STABLE_FOR_MS } = await import("../dist/provider-loop/index.js");
|
||||
assert.equal(STABLE_FOR_MS, 10 * 60_000);
|
||||
});
|
||||
|
||||
@@ -326,7 +327,7 @@ test("the four tools are registered for the module and answer", async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), "retire-tools-"));
|
||||
const receives = join(dir, "r.json");
|
||||
await writeFile(receives, JSON.stringify({ requirement: "r", given: [] }));
|
||||
const p = runProvisioner("r", { async create() {}, async remove() {} }, { receives, everyMs: EVERY, module: "postgres" });
|
||||
const p = runProviderLoop("r", { async create() {}, async remove() {} }, { receives, everyMs: EVERY, module: "postgres" });
|
||||
try {
|
||||
const mine = collectTools({}).find((c) => c.module === "postgres");
|
||||
assert.deepEqual(mine?.tools.map((t) => t.name), [
|
||||
@@ -335,7 +336,7 @@ test("the four tools are registered for the module and answer", async () => {
|
||||
const answer = (await mine!.tools[0].run({})) as { resource: string; waiting: unknown };
|
||||
assert.equal(answer.resource, "r");
|
||||
assert.equal(answer.waiting, null);
|
||||
assert.equal(provisionerTools(p).length, 4);
|
||||
assert.equal(providerLoopTools(p).length, 4);
|
||||
} finally {
|
||||
p();
|
||||
resetTools();
|
||||
|
||||
+26
-13
@@ -6,7 +6,7 @@ import { join } from "node:path";
|
||||
|
||||
import { compareVersions } from "../dist/primitives/index.js";
|
||||
import { registerModuleTools, collectTools, collectToolsEach, resetTools, serveTools, listTools, invokeTool } from "../dist/tools/index.js";
|
||||
import { runProvisioner } from "../dist/provisioner/index.js";
|
||||
import { runProviderLoop } from "../dist/provider-loop/index.js";
|
||||
import { emit, on, type Event } from "../dist/events/index.js";
|
||||
import { useBroker } from "../dist/messaging/index.js";
|
||||
|
||||
@@ -47,7 +47,7 @@ test("provisioner creates each consumer with the mesh's login and password; remo
|
||||
doc([{ from: "webapp", node: "anchor", as: "webapp-anchor", secret: join(dir, "webapp.secret"), values: { name: "webapp" } }]),
|
||||
);
|
||||
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"analytics",
|
||||
{
|
||||
async create(p) {
|
||||
@@ -101,7 +101,7 @@ test("provisioner hands the adapter what the mesh derived, and still has it at w
|
||||
});
|
||||
await writeFile(receives, doc([entry("mesh-anchor-ncloud")]));
|
||||
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"s3-bucket",
|
||||
{
|
||||
async create(p) {
|
||||
@@ -153,7 +153,7 @@ test("provisioner stops calling an unreadable secret a race, and says what to lo
|
||||
console.error = (line: unknown) => { said.push(String(line)); };
|
||||
let stop: (() => void) | undefined;
|
||||
try {
|
||||
stop = runProvisioner("db", { async create() {}, async remove() {} }, { receives, everyMs: 5 });
|
||||
stop = runProviderLoop("db", { async create() {}, async remove() {} }, { receives, everyMs: 5 });
|
||||
await waitFor(() => said.some((l) => l.includes("CANNOT READ")), 3000);
|
||||
} finally {
|
||||
stop?.();
|
||||
@@ -185,7 +185,7 @@ test("provisioner removes nothing while its contributions file cannot be used (n
|
||||
]);
|
||||
await writeFile(receives, both);
|
||||
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"postgres-database",
|
||||
{
|
||||
async create(p) {
|
||||
@@ -239,7 +239,7 @@ test("provisioner applies again what the backend no longer holds, and trusts mem
|
||||
const backend = new Set<string>();
|
||||
let creates = 0;
|
||||
let asked = 0;
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create(p) {
|
||||
@@ -271,7 +271,7 @@ test("provisioner applies again what the backend no longer holds, and trusts mem
|
||||
// An adapter without holds is trusted from memory, as before: a forgotten backend stays forgotten.
|
||||
const forgetful = new Set<string>();
|
||||
let plainCreates = 0;
|
||||
const stopPlain = runProvisioner(
|
||||
const stopPlain = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create(p) {
|
||||
@@ -299,7 +299,7 @@ test("provisioner keeps a consumer applied when the backend cannot be asked", as
|
||||
);
|
||||
let creates = 0;
|
||||
let asked = 0;
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create() {
|
||||
@@ -330,7 +330,7 @@ test("provisioner backs off when create does not satisfy holds, and says so", as
|
||||
const original = console.error;
|
||||
console.error = (...a: unknown[]) => logged.push(a.join(" "));
|
||||
try {
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create() {
|
||||
@@ -371,7 +371,7 @@ test("provisioner treats a failed or hung holds as could-not-ask, and never logs
|
||||
const original = console.error;
|
||||
console.error = (...a: unknown[]) => logged.push(a.join(" "));
|
||||
try {
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create() {
|
||||
@@ -404,7 +404,7 @@ test("provisioner treats a failed or hung holds as could-not-ask, and never logs
|
||||
const hungLogged: string[] = [];
|
||||
console.error = (...a: unknown[]) => hungLogged.push(a.join(" "));
|
||||
try {
|
||||
const stopHung = runProvisioner(
|
||||
const stopHung = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create() {
|
||||
@@ -438,7 +438,7 @@ test("provisioner stops a verify pass at a timeout, and retries a failed re-appl
|
||||
try {
|
||||
// A timeout on "a" means "b" is not asked in that pass.
|
||||
const askedFor: string[] = [];
|
||||
const stopHung = runProvisioner(
|
||||
const stopHung = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create() {},
|
||||
@@ -460,7 +460,7 @@ test("provisioner stops a verify pass at a timeout, and retries a failed re-appl
|
||||
let createCalls = 0;
|
||||
const logged: string[] = [];
|
||||
console.error = (...a: unknown[]) => logged.push(a.join(" "));
|
||||
const stop = runProvisioner(
|
||||
const stop = runProviderLoop(
|
||||
"cache",
|
||||
{
|
||||
async create(p) {
|
||||
@@ -649,3 +649,16 @@ test("each registration is given the environment asked for it, and sees no other
|
||||
assert.deepEqual(seen.beta, { BETA_URL: "http://127.0.0.1:3000" });
|
||||
resetTools();
|
||||
});
|
||||
|
||||
test("the provider loop is named for what it is, and its old name stays an alias until its providers move (issue 678)", async () => {
|
||||
const loop = await import("../dist/provider-loop/index.js");
|
||||
const old = await import("../dist/index.js");
|
||||
assert.equal(old.provisioner.runProvisioner, loop.runProviderLoop);
|
||||
assert.equal(old.provisioner.provisionerTools, loop.providerLoopTools);
|
||||
assert.equal(old.providerLoop.runProviderLoop, loop.runProviderLoop);
|
||||
const pkg = JSON.parse(await (await import("node:fs/promises")).readFile(new URL("../package.json", import.meta.url), "utf8"));
|
||||
assert.equal(pkg.exports["./provider-loop"], "./dist/provider-loop/index.js");
|
||||
assert.equal(pkg.exports["./provisioner"], pkg.exports["./provider-loop"]);
|
||||
// The events and tools are what the controller and the operator read on the bus: their names stay.
|
||||
assert.equal(loop.FAILING_EVENT, "provisioner.failing");
|
||||
});
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { createPrivateKey } from "node:crypto";
|
||||
|
||||
import { seal, open, generateKeyPair, type SealedBox } from "../dist/seal/index.js";
|
||||
import { sealWith } from "../dist/seal/box.js";
|
||||
|
||||
// The fixtures the Go package proves itself against: boxes the licence manager's own seal.go made, and
|
||||
// the one the agent module's TypeScript made. One format, three implementations.
|
||||
const fixtures = new URL("../go/seal/testdata/", import.meta.url);
|
||||
const lm = JSON.parse(readFileSync(new URL("sealed-by-licence-manager.json", fixtures), "utf8")) as {
|
||||
recipient: { publicKey: string; privateKey: string };
|
||||
sealed: { plaintext: string; box: SealedBox }[];
|
||||
vectors: { plaintext: string; ephPrivate: string; iv: string; box: SealedBox }[];
|
||||
};
|
||||
|
||||
// PKCS #8 for an X25519 private key is this prefix and the 32 raw bytes.
|
||||
const pkcs8 = (raw: Buffer) =>
|
||||
createPrivateKey({ key: Buffer.concat([Buffer.from("302e020100300506032b656e04220420", "hex"), raw]), format: "der", type: "pkcs8" });
|
||||
|
||||
test("seal: every box the licence manager sealed opens to its plaintext", () => {
|
||||
assert.ok(lm.sealed.length > 0);
|
||||
for (const s of lm.sealed) assert.equal(open(s.box, lm.recipient.privateKey), s.plaintext);
|
||||
});
|
||||
|
||||
test("seal: given the same one-time key and iv, the box is byte for byte the licence manager's", () => {
|
||||
assert.ok(lm.vectors.length > 0);
|
||||
for (const v of lm.vectors) {
|
||||
const box = sealWith(v.plaintext, lm.recipient.publicKey, pkcs8(Buffer.from(v.ephPrivate, "base64")), Buffer.from(v.iv, "base64"));
|
||||
assert.deepEqual(box, v.box);
|
||||
assert.equal(open(box, lm.recipient.privateKey), v.plaintext);
|
||||
}
|
||||
});
|
||||
|
||||
test("seal: the agent module's TypeScript box opens", () => {
|
||||
const f = JSON.parse(readFileSync(new URL("sealed-by-typescript.json", fixtures), "utf8"));
|
||||
assert.equal(open(f.box, f.privateKey), f.plaintext);
|
||||
});
|
||||
|
||||
test("seal: a box opens for its recipient only, and two boxes share no one-time key or iv", () => {
|
||||
const a = generateKeyPair();
|
||||
const b = generateKeyPair();
|
||||
const box = seal("a token", a.publicKey);
|
||||
assert.equal(open(box, a.privateKey), "a token");
|
||||
assert.throws(() => open(box, b.privateKey));
|
||||
const again = seal("a token", a.publicKey);
|
||||
assert.notEqual(again.eph, box.eph);
|
||||
assert.notEqual(again.iv, box.iv);
|
||||
});
|
||||
|
||||
test("seal: a box changed in any field, or of another version, does not open", () => {
|
||||
const good = lm.sealed[1].box;
|
||||
const flip = (s: string) => {
|
||||
const b = Buffer.from(s, "base64");
|
||||
b[b.length - 1] ^= 1;
|
||||
return b.toString("base64");
|
||||
};
|
||||
for (const box of [
|
||||
{ ...good, ct: flip(good.ct) },
|
||||
{ ...good, tag: flip(good.tag) },
|
||||
{ ...good, iv: flip(good.iv) },
|
||||
{ ...good, eph: flip(good.eph) },
|
||||
{ ...good, v: 2 },
|
||||
]) assert.throws(() => open(box as SealedBox, lm.recipient.privateKey));
|
||||
});
|
||||
|
||||
test("seal: a key that is not X25519 PEM is refused", () => {
|
||||
assert.throws(() => seal("x", "not a key"));
|
||||
assert.throws(() => open(lm.sealed[0].box, lm.recipient.publicKey));
|
||||
});
|
||||
@@ -0,0 +1,272 @@
|
||||
// A provider that keeps failing a consumer says so on the bus (novox/hq ADR 0224, issue 272): not on
|
||||
// the first failure, which may be a restart; after failingAfterMs of failures with no success between;
|
||||
// again every sayAgainEveryMs while it lasts; and recovered on the first success, or when the consumer
|
||||
// goes. The same behaviour as the Go copy of the loop's standing_test.go, with the clock shortened.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, writeFile, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
runProviderLoop, classOf, FAILING_EVENT, RECOVERED_EVENT, RETIREMENT_EVENT,
|
||||
CLASS_CREDENTIALS, CLASS_UNREACHABLE, CLASS_SECRET, CLASS_REFUSED, type Adapter,
|
||||
} from "../dist/provider-loop/index.js";
|
||||
import { useBroker } from "../dist/messaging/index.js";
|
||||
|
||||
const EVERY = 10;
|
||||
const FAILING_AFTER = 150;
|
||||
const SAY_AGAIN = 400;
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
async function until(cond: () => boolean, ms = 4000): Promise<void> {
|
||||
const start = Date.now();
|
||||
while (!cond()) {
|
||||
if (Date.now() - start > ms) throw new Error("timed out waiting");
|
||||
await sleep(5);
|
||||
}
|
||||
}
|
||||
|
||||
interface Said { event: string; body: Record<string, unknown> }
|
||||
|
||||
interface World {
|
||||
dir: string;
|
||||
ask(list: { as: string; node?: string; values?: Record<string, unknown> }[]): Promise<void>;
|
||||
failing: Error | null;
|
||||
holdsErr: Error | null;
|
||||
created: string[];
|
||||
/** Every event but the first success since start, which has its own test. */
|
||||
said: Said[];
|
||||
all: Said[];
|
||||
stop(): void;
|
||||
}
|
||||
|
||||
async function world(opts: { classify?: boolean; announce?: false; stableForMs?: number } = {}): Promise<World> {
|
||||
const dir = await mkdtemp(join(tmpdir(), "standing-"));
|
||||
const receives = join(dir, "postgres-database.json");
|
||||
const w = { dir, failing: null, holdsErr: null, created: [], said: [], all: [] } as unknown as World;
|
||||
w.ask = async (list) => {
|
||||
const given = [];
|
||||
for (const g of list) {
|
||||
await writeFile(join(dir, `${g.as}.secret`), `pw-${g.as}\n`);
|
||||
given.push({ from: g.as, node: g.node ?? "anchor", as: g.as, secret: join(dir, `${g.as}.secret`), values: g.values });
|
||||
}
|
||||
await writeFile(receives, JSON.stringify({ contributions: 1, requirement: "postgres-database", given }));
|
||||
};
|
||||
const adapter: Adapter = {
|
||||
async create(p) {
|
||||
if (w.failing) throw w.failing;
|
||||
w.created.push(p.as);
|
||||
},
|
||||
async remove() {},
|
||||
async retire() {},
|
||||
async holds() {
|
||||
if (w.holdsErr) throw w.holdsErr;
|
||||
return true;
|
||||
},
|
||||
...(opts.classify ? { classify: () => "its-own" } : {}),
|
||||
};
|
||||
const was = console.error;
|
||||
console.error = () => {};
|
||||
const stop = runProviderLoop("postgres-database", adapter, {
|
||||
receives, everyMs: EVERY, verifyEveryMs: 0, node: "home-server",
|
||||
failingAfterMs: FAILING_AFTER, sayAgainEveryMs: SAY_AGAIN, stableForMs: opts.stableForMs ?? 0,
|
||||
...(opts.announce === false ? {} : {
|
||||
announce: (event: string, body: Record<string, unknown>) => {
|
||||
w.all.push({ event, body });
|
||||
if (body.why !== "first-success") w.said.push({ event, body });
|
||||
},
|
||||
}),
|
||||
});
|
||||
w.stop = () => {
|
||||
stop();
|
||||
console.error = was;
|
||||
void rm(dir, { recursive: true, force: true });
|
||||
};
|
||||
return w;
|
||||
}
|
||||
|
||||
const events = (said: Said[]) => said.filter((s) => s.event !== RETIREMENT_EVENT).map((s) => s.event).join(",");
|
||||
|
||||
test("a consumer failed for minutes is announced, naming it, its machine and the class, again while it lasts, and recovered", async () => {
|
||||
const w = await world();
|
||||
try {
|
||||
w.failing = new Error(`token request failed: 401 {"error":"invalid_grant"} for pw-a`);
|
||||
await w.ask([{ as: "a", node: "anchor" }]);
|
||||
await sleep(FAILING_AFTER / 2);
|
||||
assert.equal(events(w.said), "", "announced before failingAfterMs");
|
||||
await until(() => w.said.length === 1);
|
||||
const b = w.said[0].body;
|
||||
assert.equal(w.said[0].event, FAILING_EVENT);
|
||||
assert.equal(b.consumer, "a");
|
||||
assert.equal(b.node, "anchor");
|
||||
assert.equal(b.class, CLASS_CREDENTIALS);
|
||||
assert.equal(b.provider, "postgres-database");
|
||||
assert.equal(b["provider-node"], "home-server");
|
||||
assert.ok((b.attempts as number) > 3, `attempts ${b.attempts}`);
|
||||
assert.match(String(b.since), /^\d{4}-\d{2}-\d{2}T/);
|
||||
assert.doesNotMatch(String(b.error), /pw-a/, "the consumer's password travelled");
|
||||
|
||||
// Said again while it lasts, not every pass.
|
||||
await sleep(SAY_AGAIN / 2);
|
||||
assert.equal(events(w.said), FAILING_EVENT, "repeated too soon");
|
||||
await until(() => w.said.length === 2);
|
||||
assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`);
|
||||
|
||||
// The first success takes it back.
|
||||
w.failing = null;
|
||||
await until(() => w.said.length === 3);
|
||||
assert.equal(w.said[2].event, RECOVERED_EVENT);
|
||||
assert.equal(w.said[2].body.consumer, "a");
|
||||
assert.equal(w.said[2].body.node, "anchor");
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("one success between failures starts the run again", async () => {
|
||||
const w = await world();
|
||||
try {
|
||||
w.failing = new Error("connection refused");
|
||||
await w.ask([{ as: "a" }]);
|
||||
await sleep(FAILING_AFTER * 0.6);
|
||||
w.failing = null;
|
||||
await until(() => w.created.length === 1);
|
||||
w.failing = new Error("connection refused");
|
||||
await w.ask([{ as: "a", values: { name: "changed" } }]);
|
||||
await sleep(FAILING_AFTER * 0.6);
|
||||
assert.equal(events(w.said), "", "two short runs are not one long one");
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// The check that failed for a day on 2026-10-05 (issue 179): consumers already made, every check
|
||||
// refused. A check that cannot be asked is a failure too.
|
||||
test("a check that keeps failing is a failure too", async () => {
|
||||
const w = await world();
|
||||
try {
|
||||
await w.ask([{ as: "a" }]);
|
||||
await until(() => w.created.length === 1);
|
||||
w.holdsErr = new Error("401 invalid_grant");
|
||||
await until(() => w.said.length === 1);
|
||||
assert.equal(w.said[0].event, FAILING_EVENT);
|
||||
assert.equal(w.said[0].body.class, CLASS_CREDENTIALS);
|
||||
w.holdsErr = null;
|
||||
await until(() => w.said.length === 2);
|
||||
assert.equal(events(w.said), `${FAILING_EVENT},${RECOVERED_EVENT}`);
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("an unreadable secret is announced as such", async () => {
|
||||
const w = await world();
|
||||
try {
|
||||
await w.ask([{ as: "a" }]);
|
||||
await rm(join(w.dir, "a.secret"));
|
||||
await until(() => w.said.length === 1);
|
||||
assert.equal(w.said[0].event, FAILING_EVENT);
|
||||
assert.equal(w.said[0].body.class, CLASS_SECRET);
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a consumer no longer asked for is no longer failing", async () => {
|
||||
const w = await world();
|
||||
try {
|
||||
w.failing = new Error("boom");
|
||||
await w.ask([{ as: "a" }, { as: "b" }]);
|
||||
await until(() => w.said.length === 2);
|
||||
assert.equal(events(w.said), `${FAILING_EVENT},${FAILING_EVENT}`);
|
||||
await w.ask([{ as: "a" }]);
|
||||
await until(() => w.said.some((s) => s.event === RECOVERED_EVENT));
|
||||
const last = w.said.find((s) => s.event === RECOVERED_EVENT)!;
|
||||
assert.equal(last.body.consumer, "b");
|
||||
assert.equal(last.body.why, "no longer asked for");
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("a failing consumer the mesh stopped asking for is said recovered when it is retired", async () => {
|
||||
const w = await world();
|
||||
try {
|
||||
await w.ask([{ as: "a" }, { as: "b" }, { as: "c" }]);
|
||||
await until(() => w.created.length === 3);
|
||||
w.holdsErr = new Error("boom");
|
||||
await until(() => w.said.filter((s) => s.event === FAILING_EVENT).length >= 3);
|
||||
// c is no longer asked for while it is still failing; the others then recover.
|
||||
await w.ask([{ as: "a" }, { as: "b" }]);
|
||||
w.holdsErr = null;
|
||||
await until(() => w.said.some((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c"));
|
||||
const c = w.said.find((s) => s.event === RECOVERED_EVENT && s.body.consumer === "c")!;
|
||||
assert.equal(c.body.why, "retired");
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
test("an error is classed by its words", () => {
|
||||
const table: Record<string, string> = {
|
||||
'Keycloak token request failed: 401 {"error":"invalid_grant"}': CLASS_CREDENTIALS,
|
||||
'FATAL: password authentication failed for user "postgres"': CLASS_CREDENTIALS,
|
||||
"dial tcp 127.0.0.1:5432: connect: connection refused": CLASS_UNREACHABLE,
|
||||
"context deadline exceeded": CLASS_UNREACHABLE,
|
||||
"Error: connect ECONNREFUSED 127.0.0.1:9000": CLASS_UNREACHABLE,
|
||||
"getaddrinfo ENOTFOUND minio": CLASS_UNREACHABLE,
|
||||
'extension "nope" is not available': CLASS_REFUSED,
|
||||
};
|
||||
for (const [text, want] of Object.entries(table)) assert.equal(classOf(text), want, text);
|
||||
});
|
||||
|
||||
test("an adapter that classes its own errors is believed", async () => {
|
||||
const w = await world({ classify: true });
|
||||
try {
|
||||
w.failing = new Error("anything");
|
||||
await w.ask([{ as: "a" }]);
|
||||
await until(() => w.said.length === 1);
|
||||
assert.equal(w.said[0].body.class, "its-own");
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// A provider restarted after announcing a failure has forgotten it; its first success for each
|
||||
// consumer is announced, so the controller clears what it kept rather than naming it for ever.
|
||||
test("the first success since start is announced once", async () => {
|
||||
const w = await world();
|
||||
try {
|
||||
await w.ask([{ as: "a" }]);
|
||||
await until(() => w.all.length >= 1);
|
||||
await sleep(EVERY * 10);
|
||||
assert.equal(w.all.length, 1, JSON.stringify(w.all));
|
||||
assert.equal(w.all[0].event, RECOVERED_EVENT);
|
||||
assert.equal(w.all[0].body.why, "first-success");
|
||||
assert.equal(w.all[0].body.consumer, "a");
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
|
||||
// No provider on main passed `announce`, so none said anything (issue 272): the loop says it through
|
||||
// the SDK's own emit unless it is given another way.
|
||||
test("without announce the standing is said through the SDK's emit", async () => {
|
||||
const published: { key: string; body: Record<string, unknown> }[] = [];
|
||||
useBroker(() => ({
|
||||
async publish(env: { key: string; body: Record<string, unknown> }) { published.push({ key: env.key, body: env.body }); },
|
||||
async subscribe() { return () => {}; },
|
||||
}) as never);
|
||||
const w = await world({ announce: false });
|
||||
try {
|
||||
w.failing = new Error("connection refused");
|
||||
await w.ask([{ as: "a" }]);
|
||||
await until(() => published.some((p) => p.key === FAILING_EVENT));
|
||||
const p = published.find((x) => x.key === FAILING_EVENT)!;
|
||||
assert.equal(p.body.consumer, "a");
|
||||
assert.equal(p.body.class, CLASS_UNREACHABLE);
|
||||
} finally {
|
||||
w.stop();
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user