Compare commits

..
Author SHA1 Message Date
mesh-tools 37e4219e28 Restore the final blank line of seven golden files the patch push dropped
mesh/delivery-group group 57-config-and-provisioning rejected: a member's own check failed
mesh/merge-gate head 37e4219e28fd merged into trunk fcdecb436117 — pass: [release feat/57-config-and-provisioning, judged by its controller e2ea694217a7] t…
mesh/repo-check head 37e4219e28fd merged into trunk fcdecb436117 — fail: [release feat/57-config-and-provisioning, judged by its controller e2ea694217a7] T…
mesh/delivery rejected: the gate failed or could not run, or the repository's own check failed
The forge applied the feature's patch without the last, empty, added line of
these files, so the commit's tree differed from the reviewed head (issue 866).
2026-10-15 21:39:32 +00:00
mesh-tools 97a2d841c7 Land feature 57's controller work on its release branch
The controller plans config, templates and moved providers' provisions,
hands bus accounts to the vault and judges the new manifest words, on the
feature's SDK and node-engine validator (feature 57, config and provisioning).
2026-10-15 21:27:33 +00:00
mesh-admin fcdecb4361 Merge pull request '57/K2: The controller reads, judges against and keeps the catalogue's list of predefined keys (issue 661)' (#323) from pkg/57-k2-predefined-keys into feat/57-config-and-provisioning 2026-10-14 06:52:24 +00:00
mesh-adminandClaude Opus 5.5 197e60c97b List the K2 store tests again, apart from the integration branch's new lines
mesh/merge-gate head 197e60c97bcc merged into trunk 9932adfa00fe — pass: the change touches no module of the mesh's graph
mesh/repo-check head 197e60c97bcc merged into trunk 9932adfa00fe — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group pkg/57-k2-predefined-keys delivered: every member is delivered
Placed above the lines both branches added at the end of the file, so the pull request merges into feat/57-config-and-provisioning without a conflict and the merged list is the union of both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 06:42:51 +00:00
mesh-adminandClaude Opus 5.5 e75c376a7d Take the integration branch's store-test list before merging it
mesh/delivery-group group pkg/57-k2-predefined-keys checking: 1 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head e75c376a7dd9 merged into trunk 9932adfa00fe — pass: the change touches no module of the mesh's graph
mesh/repo-check head e75c376a7dd9 merged into trunk 9932adfa00fe — pass: its merge-check.sh passed
The two lists added lines at the same place; taking the integration branch's here lets it merge cleanly, and the K2 store tests are listed again right after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 06:41:52 +00:00
mesh-admin 9932adfa00 Merge pull request 'Release 57: bring main into the integration branch (conflicts resolved)' (#324) from pkg/57-sync-main-1 into feat/57-config-and-provisioning 2026-10-14 06:31:43 +00:00
mesh-adminandClaude Opus 5.5 4b2eac702a Bring main into release 57's branch, conflicts resolved
mesh/merge-gate head 4b2eac702ab1 merged into trunk 221fe5a60d1f — pass: the change touches no module of the mesh's graph
mesh/repo-check head 4b2eac702ab1 merged into trunk 221fe5a60d1f — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): feat/57-config-and-provisioning'…
mesh/delivery delivered
So the release branch carries what main gained, and the store-tests drop check passes again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 06:22:29 +00:00
mesh-adminandClaude Opus 5.5 c88474a127 Bring main into release 57's branch, part 2 of 3: the rest at once and the try verb
So the release branch carries what main gained; part 3 follows on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 06:20:08 +00:00
mesh-adminandClaude Opus 5.5 3d3ca6e500 Bring main into release 57's branch, part 1 of 3: the builder's try outcome and the release walk
So the release branch carries what main gained; parts 2 and 3 follow on this branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 06:16:47 +00:00
mesh-adminandClaude Opus 5.5 fbcf44702c Prove the list is read strictly and judges module check, the gate and registration
mesh/delivery-group group pkg/57-k2-predefined-keys rejected: a member's own check failed
mesh/merge-gate head fbcf44702cd6 merged into trunk 9932adfa00fe — fail: head fbcf44702cd6 does not merge cleanly into trunk 9932adfa00fe: in conflict test…
mesh/repo-check head fbcf44702cd6 merged into trunk 9932adfa00fe — fail: head fbcf44702cd6 does not merge cleanly into trunk 9932adfa00fe: in conflict test…
mesh/delivery superseded: a newer head of the same pull request
Part 4: the tests of the list's reading and of the controller's commands on it (hq ADR 0348 clause 7, issue 661).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:48:49 +00:00
mesh-adminandClaude Opus 5.5 2941ef3695 Read the list of predefined keys strictly, and prove the build, check, gate and registration on it
Part 3: the list's own reading (catalogue/predefined.go) and the tests of the build, module check, merge gate and registration (hq ADR 0348 clause 7, issue 661).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:47:44 +00:00
mesh-adminandClaude Opus 5.5 9f2e16ff70 Keep the newest trunk build's list of predefined keys, and prove each refusal
The list itself, read strictly (catalogue/predefined.go); the controller keeps a catalogue trunk build's list in its store when its commit's generation is larger than the one kept (migration 0098), so an older build's list heard later never replaces a newer one, and loads it with its store (hq ADR 0348 clause 7, issue 661). Part 2 of 2: the new files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:46:49 +00:00
mesh-adminandClaude Opus 5.5 c0bee5d042 Judge every manifest against the list of predefined keys of its tree, and carry the list from a build
Mesh and node config may hold values only for keys on a fixed, reviewed list (hq ADR 0348 clause 7, design 57 package K2, issue 661). A build reads predefined.json at the root of the repository at the commit it builds and carries it with the commit's generation; module check and the merge gate read it from the tree. (Part 1 of 2: the changes to existing files.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:45:48 +00:00
mesh-admin 221fe5a60d Merge pull request '57/K1: A module declares its config with type, description and validation (issue 660)' (#320) from pkg/57-k1-config-declaration into feat/57-config-and-provisioning 2026-10-14 05:25:35 +00:00
mesh-adminandClaude Opus 5.5 9aa0332694 Refuse an undeclared takes-config key, and say an unset required value in the config form's words
mesh/merge-gate head 9aa033269495 merged into trunk 60d390adeddc — pass: the change touches no module of the mesh's graph
mesh/repo-check head 9aa033269495 merged into trunk 60d390adeddc — pass: its merge-check.sh passed
mesh/delivery delivered
A key a mergeable file took had no type, validation or trusted mark, and the config verb could never set it (ADR 0348 §4). A required config value nothing sets was refused naming ${setting:} and settings set, words the module does not use. Review of mesh-controller#320 (issue 660).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:17:26 +00:00
mesh-admin 60d390aded Merge pull request '57/F2b: A settings layer no longer replaces what a provider serves (issue 696)' (#321) from pkg/57-f2b-served-keys into feat/57-config-and-provisioning 2026-10-14 05:16:03 +00:00
mesh-adminandClaude Opus 5.5 c5fe7d4795 Stop a settings layer replacing what a provider serves (issue 696)
mesh/merge-gate head c5fe7d4795d7 merged into trunk dc36536e199e — pass: the change touches no module of the mesh's graph
mesh/repo-check head c5fe7d4795d7 merged into trunk dc36536e199e — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group pkg/57-f2b-served-keys delivered: every member is delivered
What a provider serves is its definition's (ADR 0348 clause 11): a layer
fills only the keys it asks for as ${setting:<key>}. Settle replaced any
served key a layer named, so a setting for one of a provider's files could
change what every consumer was told. influxdb on ace was the one live layer
relying on it and now serves its settings (mesh-catalog, merged first), so
grafana's data source composes byte for byte as before. UnusedSettings no
longer counts a literal served key as a place a setting lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:03:08 +00:00
mesh-adminandClaude Opus 5.5 ae071e2b59 Prove each config refusal and the byte-identical composition of a module moved to config (issue 660)
mesh/merge-gate out of date: judged against trunk dc36536e199e, the trunk is now 60d390adeddc; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk dc36536e199e, the trunk is now 60d390adeddc; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
One registration refusal test per rule of design 57 package K1, the forms kept apart and written back as read, and dunst rewritten from settings to config composing to the same bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:39:13 +00:00
mesh-adminandClaude Opus 5.5 233956ef09 Let a module declare its config with type, description and validation (issue 660)
The config form's declaration and its registration refusals, each by name with what to do (ADR 0348, design 57 package K1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:38:04 +00:00
mesh-adminandClaude Opus 5.5 2f40ae5f76 Read the config form beside the settings form in a manifest (part of issue 660)
The manifest's hooks for the config form: read into the fields the settings form fills, written back as read, and judged by ConfigProblems; the rules are in config.go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:36:45 +00:00
mesh-admin dc36536e19 Merge pull request '57/V1: The vault's seat promises get (issue 674)' (#317) from pkg/57-v1-vault-in-go into feat/57-config-and-provisioning 2026-10-14 01:19:23 +00:00
mesh-admin 71362c196d Merge pull request '57/F2: Placement leaves the settings layers, held beside the assignment (issue 651)' (#316) from pkg/57-f2-placement-facts into feat/57-config-and-provisioning 2026-10-14 01:09:55 +00:00
mesh-admin 617881c309 Merge pull request '57/T1: A file resource may name a template file in its module's repository (issue 654)' (#313) from pkg/57-t1-template-files into feat/57-config-and-provisioning 2026-10-14 01:00:47 +00:00
mesh-adminandClaude Opus 5.5 661918f4e4 Promise get on the vault's seat, so the Go vault can serve it (issue 674)
mesh/merge-gate head 661918f4e4e0 merged into trunk 71362c196dbc — pass: the change touches no module of the mesh's graph
mesh/repo-check head 661918f4e4e0 merged into trunk 71362c196dbc — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group pkg/57-v1-vault-in-go delivered: every member is delivered
The vault hands a secret only on request and reply, sealed to the asker's key
(hq ADR 0349 §11). The seat set is the controller's, so the verb is promised
here; optional until the vault that runs serves it (design 33 §7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:51:59 +00:00
mesh-adminandClaude Opus 5.5 def3ccefe3 Refuse templates read through a symlink anywhere under files/
mesh/delivery delivered
mesh/merge-gate head def3ccefe3c8 merged into trunk bfc79c64d989 — pass: the change touches no module of the mesh's graph
mesh/repo-check head def3ccefe3c8 merged into trunk bfc79c64d989 — pass: its merge-check.sh passed
A symlinked files/ or a symlinked directory under it let module check, the merge
gate and module add read a template from outside the repository tree, and a
symlink no template names passed them while the build's packing refuses it
(review of mesh-controller#313). Every entry under files/ must now be a plain
directory or file, and templates are read through an os.Root of the module's
directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:51:48 +00:00
mesh-adminandClaude Opus 5.5 912d76df25 Move every layer's placement out with migration 0097, and prove no machine is told anything new (issue 651)
mesh/delivery delivered
mesh/merge-gate head 912d76df25f8 merged into trunk 617881c3093d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 912d76df25f8 merged into trunk 617881c3093d — pass: its merge-check.sh passed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:45:58 +00:00
mesh-adminandClaude Opus 5.5 3048dccd0e Keep the declarations the controller composed before placement moved (issue 651)
Written by the integration branch's controller at bfc79c6 from the seed of
TestADeclarationIsTheSameAfterPlacementMoves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:44:51 +00:00
mesh-adminandClaude Opus 5.5 f0af09cfc7 Test placement read only from the placement, and raise a store before a migration (issue 651)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:44:20 +00:00
mesh-adminandClaude Opus 5.5 85b85a3fa9 Say a layer whole where the mesh reports it, and give tests their placement (issue 651)
The facts snapshot, the take preview and the settings history read a layer as it
was set; tests that place give the layer through LayerOf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:43:40 +00:00
mesh-adminandClaude Opus 5.5 02b542a059 Keep a layer's placement beside it, and say the layer whole (issue 651)
The settings verb's path splits a layer it is given into its config values and
its placement, so what a person types still works; a layer reads back and goes
into the history whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:42:37 +00:00
mesh-adminandClaude Opus 5.5 7530c2aeac Read placement from a layer's Placement, never from its config values (issue 651)
ADR 0348 clause 11: places, accesses and kept networks are placement, not config.
Today's mergeable files still take them as said, so nothing composes differently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:41:06 +00:00
mesh-adminandClaude Opus 5.5 f45e0a1de5 Let a file resource name a template file in its module's repository: composition, check and gate tests
mesh/merge-gate head f45e0a1de508 merged into trunk bfc79c64d989 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f45e0a1de508 merged into trunk bfc79c64d989 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:59:22 +00:00
mesh-adminandClaude Opus 5.5 276f6755ab Let a file resource name a template file in its module's repository: the catalogue's reading of templates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:58:16 +00:00
mesh-adminandClaude Opus 5.5 1ce4ab8c2a Let a file resource name a template file in its module's repository: the templates and their tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:57:33 +00:00
mesh-adminandClaude Opus 5.5 d43fbc202c Let a file resource name a template file in its module's repository
A file's text may live in files/<name> beside the manifest instead of inline,
so modules can move their files out of module.json (ADR 0349 clause 2, issue
654). The build packs files/ into the module's declared archive and reads each
template back from the packed bytes, proven by the digest, into content: the
manifest the mesh holds, composition and the node-engine stay as they are.
module check, the merge gate and module add read templates from the tree beside
the manifest, so a change to a template alone is judged. A file naming both
template and content, a template outside files/, a missing one, and a held
manifest that still names one are refused by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:56:43 +00:00
mesh-admin bfc79c64d9 Open the integration branch of release 57 (config and provisioning)
So the packages of hq design 57 meet here, reviewed one by one, and reach main as one delivery group.
2026-10-13 23:39:22 +00:00
259 changed files with 34856 additions and 351 deletions
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+5
View File
@@ -427,6 +427,11 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
for _, s := range built.Sources {
result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
}
// The list of predefined keys the manifest was judged against, for the controller to publish
// (novox/hq ADR 0348 §7).
if p := built.Predefined; p != nil {
result.Predefined = &link.PredefinedList{Keys: p.Keys, Generation: p.Generation}
}
say("built", built.Manifest.Module+" from "+short(built.Commit))
}
}
+3 -2
View File
@@ -424,8 +424,9 @@ func comparisonOf(module string, c comparison, opts takeOptions) (preview string
}
// And its settings on this machine, composed against its definition (rule 1, rule 6).
for _, layer := range c.layers {
keys := make([]string, 0, len(layer.Values))
for k := range layer.Values {
said := layer.Whole()
keys := make([]string, 0, len(said))
for k := range said {
keys = append(keys, k)
}
sort.Strings(keys)
+46
View File
@@ -38,15 +38,61 @@ func watchApplyFailing(f *signalFacts) []conditions.Observation {
}
}
var out []conditions.Observation
said := map[string]bool{}
for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
continue
}
out = append(out, applyFailing(d, f.now))
said[d.Node] = true
}
// **A module's files the provisioner failed to place are a failed apply too** (novox/hq ADR 0362 §3): its client's
// newest word, judged by the same bound — the client says a failure once, and fills again on every change —
// on a machine whose node-engine applied what it was sent.
failed := map[string][]inventory.ProvReport{}
for _, r := range f.provFailed {
if r.Status != inventory.ProvFailed || f.now.Sub(r.At) <= applyEvery || !heard[r.Node] || said[r.Node] {
continue
}
failed[r.Node] = append(failed[r.Node], r)
}
for _, node := range sortedNames(failed) {
out = append(out, provisioningFailing(node, failed[node], f.now))
}
return out
}
// provisioningFailing is one machine's S21 condition for the modules whose files its provisioner client failed to
// place.
func provisioningFailing(m string, failed []inventory.ProvReport, now time.Time) conditions.Observation {
sort.Slice(failed, func(a, b int) bool { return failed[a].Module < failed[b].Module })
var names, words []string
since := failed[0].At
for _, r := range failed {
names = append(names, r.Module)
words = append(words, r.Module+": "+oneLine(orNoReason(r.Reason)))
if r.At.Before(since) {
since = r.At
}
}
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
Severity: conditions.Warning, Said: strings.Join(words, "; "),
Resolved: m + " applies what it is sent again"}
o.Summary = fmt.Sprintf("%s's provisioner has failed to place the files of %s since %s (bound %s)", m,
strings.Join(names, ", "), since.UTC().Format("2006-01-02 15:04 MST"), ago(applyEvery))
o.Headline = m + " is not applying its update"
o.Explanation = fmt.Sprintf("The provisioner on %s has not been able to write the files of %s for %s. The "+
"details say why.", m, strings.Join(names, ", "), humanDuration(now.Sub(since)))
return o
}
func orNoReason(s string) string {
if s == "" {
return "no reason given"
}
return s
}
// applyFailing is one machine's S21 condition.
func applyFailing(d inventory.Doing, now time.Time) conditions.Observation {
m := d.Node
+35 -3
View File
@@ -183,8 +183,10 @@ func buildFrom(result link.BuildResult) inventory.Build {
kept.Manifest = recordedManifest(result.Manifest, announced)
// The module name comes from the manifest, which only exists when the build got that far.
if len(kept.Manifest) > 0 {
if m, err := catalogue.ParseManifest(kept.Manifest); err == nil {
kept.Module = m.Module
if list, err := judgedWith(result); err == nil {
if m, err := catalogue.ParseManifestWith(kept.Manifest, list); err == nil {
kept.Module = m.Module
}
}
}
return kept
@@ -603,6 +605,11 @@ func saysWhenThePolicyActs(ctx context.Context, inv *inventory.Inventory, module
// written with the same values.
func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResult) (
catalogue.Manifest, inventory.Build, error) {
// The list of predefined keys as the store keeps it now: another controller process may have kept a newer one.
if err := loadPredefined(ctx, inv); err != nil {
return catalogue.Manifest{}, inventory.Build{}, fmt.Errorf("the list of predefined keys kept cannot be "+
"read: %w", err)
}
kept := buildFrom(result)
if err := inv.RecordBuild(ctx, kept); err != nil {
return catalogue.Manifest{}, kept, err
@@ -613,7 +620,15 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return catalogue.Manifest{}, kept, fmt.Errorf("%s could not build %s:\n%s",
result.On, result.Repository, result.Failed)
}
manifest, err := catalogue.ParseManifest(kept.Manifest)
// Judged first against the list the build was judged against: the catalogue's at the commit built (novox/hq
// ADR 0348 §7), so a module naming a key its own commit adds is read; and again below against the list the
// mesh publishes, before it is registered.
list, err := judgedWith(result)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
result.Repository, short(result.Commit), err)
}
manifest, err := catalogue.ParseManifestWith(kept.Manifest, list)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s and what came back is not a manifest: %w",
result.On, result.Repository, err)
@@ -687,6 +702,23 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
"back: it is recorded and not registered again — a newer build is", result.On, manifest.Module,
short(result.Commit))
}
// **The newest trunk build's list of predefined keys is published** (novox/hq ADR 0348 §7): a build of the
// catalogue that may be registered keeps its list when its commit is newer than the one kept. Then the module
// is judged against the list the mesh publishes, as every manifest read again is: one naming a key that list
// does not hold, or of another type, is refused here and never enters the catalogue.
if carried, err := carriedList(result); err != nil {
return manifest, kept, err
} else if carried != nil {
if err := publishPredefined(ctx, inv, result, carried); err != nil {
fmt.Printf("%s: the list of predefined keys it carried is not published: %v\n", result.ID, err)
}
}
if manifest.NamesPredefined() {
if _, err := catalogue.ParseManifest(kept.Manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: judged against the "+
"list of predefined keys the mesh publishes, %w", result.On, manifest.Module, short(result.Commit), err)
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): a rebuild made from
// what the build the mesh stands on was made from registers that build's artifacts at the new
// commit, so no machine is sent a new digest for a source nobody changed — an image is not
+36 -1
View File
@@ -47,6 +47,24 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
shelf := catalogue.Shelf{}
faulted := map[string]bool{}
failed := 0
// The list of predefined keys at the root of the repository above each manifest (novox/hq ADR 0348 §7), read
// once per repository: a manifest naming a key is judged against the list its own tree holds.
lists := map[string]catalogue.PredefinedList{}
listOf := func(path string) (catalogue.PredefinedList, error) {
root := catalogue.RepositoryRoot(filepath.Dir(path))
if root == "" {
return nil, nil
}
if list, read := lists[root]; read {
return list, nil
}
list, err := catalogue.ReadPredefined(root)
if err != nil {
return nil, err
}
lists[root] = list
return list, nil
}
for _, path := range paths {
raw, err := os.ReadFile(path)
if err != nil {
@@ -54,7 +72,18 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed++
continue
}
m, err := catalogue.ParseManifest(raw)
list, err := listOf(path)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
continue
}
m, err := catalogue.ParseManifestWith(raw, list)
if err == nil {
// Each template's text read from the tree beside the manifest, and judged as inline content is
// (novox/hq ADR 0349 §2): a missing template is refused by name.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
@@ -215,6 +244,12 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
checks = append(checks, fmt.Sprintf("%v by %s", r["id"], catalogue.HealthWords(h)))
}
}
// And its own runtime's, where it runs as its own account (novox/hq ADR 0366).
if own, has := catalogue.OwnRuntimeHealth(m); has {
if h, _, _ := catalogue.ReadHealth(own); h.Kind != "" {
checks = append(checks, fmt.Sprintf("%v by %s", own["id"], catalogue.HealthWords(h)))
}
}
if len(checks) > 0 {
fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; "))
}
+616
View File
@@ -0,0 +1,616 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"reflect"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **The move of config** (novox/hq ADR 0348 §9 and its consequences, design 57 K5, issue 664): every config value a
// settings layer holds is set in mesh-config, one of the moves of data a release runs once after its delivery, after
// the move of placement. A module's layer on a node becomes its own value on that node (`module.<node>.<module>.<key>`).
// A whole-mesh layer has no module value for the whole mesh (ADR 0348 §3): its value becomes mesh config for the
// predefined key it falls back to where that module alone falls back to the key (claude-code's managed settings
// become `agent-managed-settings`), and otherwise the module's own value on each node it is assigned to, laid under
// that node's layer as composition lays it. A key a layer still says by its file's name (nodered's `timeZone`) is
// set under its config key (`time-zone`, issue 751).
//
// **Every node composes byte for byte as before.** The move sets the values, reads every module's resolved values
// back through mesh-config and compares them with what the layers lay today; only when they agree does it empty the
// layers it moved (each kept whole in its history first) and turn on the switch (inventory.FinishConfigMove), after
// which the controller reads mesh-config where it read the layers. A module still in the settings form keeps its
// layers and is said so. Anything that cannot move as it composes refuses the whole move, by name, before anything
// is set. Asked again, it sets nothing that is set already.
// configLevel is where mesh-config keeps a value: mesh config, node config, or a module's own value on a node.
type configLevel struct {
Level string `json:"level"`
Node string `json:"node,omitempty"`
Module string `json:"module,omitempty"`
}
func (l configLevel) valueKey(key string) string {
switch l.Level {
case "node":
return "node." + l.Node + "." + key
case "module":
return "module." + l.Node + "." + l.Module + "." + key
}
return "mesh." + key
}
func (l configLevel) String() string {
switch l.Level {
case "node":
return "node config on " + l.Node
case "module":
return l.Module + "'s own value on " + l.Node
}
return "mesh config"
}
// configSet is one value the move sets in mesh-config, and the layers it comes from.
type configSet struct {
configLevel
Key string `json:"key"`
Value any `json:"value"`
Trusted bool `json:"trusted,omitempty"`
// From is the layers it comes from, as `<module> on <where>: <key as the layer says it>`.
From []string `json:"from"`
// Already is true when mesh-config holds this value at this level already: nothing is set.
Already bool `json:"already,omitempty"`
}
// configMoveAnswer is what the move of config says it did, or would do.
type configMoveAnswer struct {
// DryRun is true when nothing was set: asked without --confirm.
DryRun bool `json:"dry-run"`
Sets []configSet `json:"sets"`
// Moved is each layer whose values the move sets, emptied once they read back.
Moved []string `json:"moved"`
// Stays is each layer of a module still in the settings form: kept, and composed from as before.
Stays []string `json:"stays,omitempty"`
// Unread is each layer that composes nothing (its module is not assigned where it says): kept in the history,
// emptied, and set nowhere.
Unread []string `json:"unread,omitempty"`
// Problems is why the move refuses, each by name; nothing is set while there is one.
Problems []string `json:"problems,omitempty"`
// Switched is true once the controller reads mesh-config where it read the layers.
Switched bool `json:"switched"`
}
// configMovePlan is the move worked out from the layers, before anything is asked of mesh-config.
type configMovePlan struct {
answer configMoveAnswer
// moved is the layers the move empties once the values read back.
moved []inventory.LayerToMove
// expected is, for each module the move reaches, keyed `<node>.<module>`, every value composition lays over
// its files today: what mesh-config must resolve for it from mesh config, node config or its own value.
expected map[string]map[string]any
// modules is each module in the config form, by the nodes it is assigned to: whose resolved values are read back.
modules map[string][]string
}
func whereOf(node string) string {
if node == "" {
return "the whole mesh"
}
return node
}
// planConfigMove works out the move from every layer that holds a value, the catalogue, and where each module is
// assigned.
func planConfigMove(layers []inventory.LayerToMove, shelf map[string]catalogue.Manifest,
on map[string][]string) configMovePlan {
p := configMovePlan{answer: configMoveAnswer{Sets: []configSet{}, Moved: []string{}},
expected: map[string]map[string]any{}, modules: map[string][]string{}}
problem := func(format string, a ...any) {
p.answer.Problems = append(p.answer.Problems, fmt.Sprintf(format, a...))
}
for name, m := range shelf {
if m.DeclaresConfig() {
p.modules[name] = on[name]
}
}
byModule := map[string]*layersOfModule{}
var names []string
for _, l := range layers {
said := fmt.Sprintf("%s on %s", l.Module, whereOf(l.Node))
m, known := shelf[l.Module]
if !known || !m.DeclaresConfig() {
p.answer.Stays = append(p.answer.Stays, said+" (its module is in the settings form)")
continue
}
if _, placement := catalogue.SplitPlacement(l.Values); len(placement) > 0 {
problem("%s still holds placement (%s): run move-placement first", said, strings.Join(movedKeys(placement), ", "))
continue
}
assigned := on[l.Module]
if (l.Node == "" && len(assigned) == 0) || (l.Node != "" && !containsString(assigned, l.Node)) {
p.answer.Unread = append(p.answer.Unread, said+" (the module is not assigned there, so it composes nothing)")
p.moved = append(p.moved, l)
continue
}
// The layer's keys as config keys: a key the file names otherwise is set under its config key.
as := catalogue.TakenAs(m)
values := map[string]any{}
from := map[string]string{}
for _, k := range movedKeys(l.Values) {
key := k
if _, declared := m.Settings[k]; !declared {
if renamed, ok := as[k]; ok {
key = renamed
} else {
problem("%s says %q, which %s does not declare: declare it, or clear it from the layer", said, k, l.Module)
continue
}
}
if _, both := values[key]; both {
// The layer says the key and the file's name for it: the config key wins, as composition has it (issue 751).
if key != k {
continue
}
}
values[key] = l.Values[k]
from[key] = said + ": " + k
}
om := byModule[l.Module]
if om == nil {
om = &layersOfModule{nodes: map[string]map[string]any{}, saidAt: map[string]map[string]string{}}
byModule[l.Module] = om
names = append(names, l.Module)
}
om.saidAt[l.Node] = from
if l.Node == "" {
om.mesh = values
} else {
om.nodes[l.Node] = values
}
p.moved = append(p.moved, l)
p.answer.Moved = append(p.answer.Moved, said)
}
sort.Strings(names)
// Which predefined key each module falls back to, so a whole-mesh value goes to mesh config only where no other
// module's value moves with it.
fallingBack := map[string][]string{}
for name, m := range shelf {
for _, d := range m.Settings {
if d.Predefined != "" && !containsString(fallingBack[d.Predefined], name) {
fallingBack[d.Predefined] = append(fallingBack[d.Predefined], name)
}
}
}
for _, name := range names {
m, om := shelf[name], byModule[name]
asked := catalogue.AskedKeys(m)
keys := map[string]bool{}
for k := range om.mesh {
keys[k] = true
}
for _, values := range om.nodes {
for k := range values {
keys[k] = true
}
}
for _, key := range movedKeys(keys) {
d := m.Settings[key]
meshValue, meshSays := om.mesh[key]
alone := d.Predefined != "" && len(fallingBack[d.Predefined]) == 1
if meshSays && alone {
p.set(configLevel{Level: "mesh"}, d.Predefined, meshValue, d, om.said("", key))
}
for _, node := range on[name] {
nodeValue, nodeSays := om.nodes[node][key]
value := meshValue
switch {
case nodeSays && meshSays:
_, under := meshValue.(map[string]any)
_, over := nodeValue.(map[string]any)
if under && over && asked[key] {
problem("%s on %s lays an object for %q over the whole mesh's, and a placeholder takes it whole "+
"where its file merges the two: say the value on %s whole, then move", name, node, key, node)
continue
}
value = catalogue.LaidOver(meshValue, nodeValue)
case nodeSays:
value = nodeValue
case !meshSays:
continue
}
p.expect(node, name, key, value)
if nodeSays || !alone {
saidBy := om.said(node, key)
if meshSays {
saidBy = append(om.said("", key), saidBy...)
}
p.set(configLevel{Level: "module", Node: node, Module: name}, key, value, d, saidBy)
}
}
}
}
sort.Strings(p.answer.Problems)
return p
}
// layersOfModule is one module's layers as the move reads them: its whole-mesh layer's config values, each node's,
// and where each key was said, by node ("" for the whole mesh).
type layersOfModule struct {
mesh map[string]any
nodes map[string]map[string]any
saidAt map[string]map[string]string
}
func (om *layersOfModule) said(node, key string) []string {
if s, ok := om.saidAt[node][key]; ok {
return []string{s}
}
return nil
}
// set is one value the move sets, judged against its declaration as mesh-config will judge it.
func (p *configMovePlan) set(level configLevel, key string, value any, d catalogue.SettingDeclaration, from []string) {
p.answer.Sets = append(p.answer.Sets, configSet{configLevel: level, Key: key, Value: value, Trusted: d.Trusted,
From: from})
for _, wrong := range catalogue.ValueProblems(d, value) {
p.answer.Problems = append(p.answer.Problems, fmt.Sprintf("%s at %s: %s", key, level, wrong))
}
}
func (p *configMovePlan) expect(node, module, key string, value any) {
k := node + "." + module
if p.expected[k] == nil {
p.expected[k] = map[string]any{}
}
p.expected[k][key] = value
}
func movedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// configService is mesh-config as the move asks it: what it holds at each level, what it resolved, and a value set
// as the controller carries it from its terminal.
type configService interface {
// Values is every value set, by its key in mesh-config_values (`mesh.<key>`, `node.<node>.<key>`,
// `module.<node>.<module>.<key>`).
Values(ctx context.Context) (map[string]json.RawMessage, error)
// Resolved is every resolved value.
Resolved(ctx context.Context) ([]resolvedEntry, error)
// Set sets one value at its level, by whom and why.
Set(ctx context.Context, level configLevel, key string, value any, by, why string) error
}
// runConfigMove is the move against a store and mesh-config: a dry run says what it would set.
func runConfigMove(ctx context.Context, inv *inventory.Inventory, config configService, apply bool, by,
why string) (configMoveAnswer, error) {
layers, holding, err := inv.LayersToMove(ctx)
if err != nil {
return configMoveAnswer{}, err
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return configMoveAnswer{}, err
}
shelf := map[string]catalogue.Manifest{}
on := map[string][]string{}
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
on[e.Manifest.Module] = e.On
}
p := planConfigMove(layers, shelf, on)
if len(holding) > 0 && len(p.answer.Problems) == 0 {
p.answer.Problems = append(p.answer.Problems, "a layer still holds placement ("+strings.Join(holding, "; ")+
"): run move-placement first")
}
moved, err := inv.ConfigMoved(ctx)
if err != nil {
return configMoveAnswer{}, err
}
p.answer.Switched = moved.Moved
if config == nil {
return p.answer, errors.New("mesh-config cannot be asked, so nothing was moved")
}
held, err := config.Values(ctx)
if err != nil {
return p.answer, fmt.Errorf("what mesh-config holds cannot be read, so nothing was moved: %w", err)
}
for i := range p.answer.Sets {
s := &p.answer.Sets[i]
if was, ok := held[s.valueKey(s.Key)]; ok && sameJSON(was, s.Value) {
s.Already = true
}
}
p.answer.DryRun = !apply
if !apply || len(p.answer.Problems) > 0 {
if apply {
return p.answer, fmt.Errorf("the move of config refuses, and nothing was set: %s",
strings.Join(p.answer.Problems, "; "))
}
return p.answer, nil
}
if len(p.moved) == 0 && moved.Moved {
return p.answer, nil
}
for _, s := range p.answer.Sets {
if s.Already {
continue
}
said := "moved from the settings layer of " + strings.Join(s.From, " and ") + " (issue 664): " + why
if err := config.Set(ctx, s.configLevel, s.Key, s.Value, by, said); err != nil {
return p.answer, fmt.Errorf("%s at %s was not set, and no layer was emptied; ask the move again: %w",
s.Key, s.configLevel, err)
}
}
// Read back through mesh-config before anything points at it (design 57): every module the move reaches, and
// every other in the config form, must resolve exactly what its layers lay today.
if differs, err := readsBack(ctx, config, p); err != nil {
return p.answer, err
} else if len(differs) > 0 {
return p.answer, fmt.Errorf("mesh-config does not resolve what the layers lay today, so no layer was emptied and "+
"the controller still composes from the layers: %s", strings.Join(differs, "; "))
}
if err := inv.FinishConfigMove(ctx, p.moved, by, why); err != nil {
return p.answer, err
}
p.answer.Switched = true
return p.answer, nil
}
// readsBack compares what mesh-config resolved with what the layers lay, for every module in the config form on
// every node it is assigned to, and says each difference. A value the layers do not lay and mesh-config resolves from
// a level would be laid once the switch is on: a difference too.
func readsBack(ctx context.Context, config configService, p configMovePlan) ([]string, error) {
var resolved []resolvedEntry
var err error
// mesh-config writes the resolved values with each set; a pass that missed one is put right by its next
// resolution, so a difference is read again for a little while before it is said.
for wait := 0; ; wait++ {
if resolved, err = config.Resolved(ctx); err != nil {
return nil, fmt.Errorf("mesh-config's resolved values cannot be read back, so no layer was emptied: %w", err)
}
differs, err := differences(resolved, p)
if err != nil || len(differs) == 0 || wait >= readBackTries {
return differs, err
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(readBackEvery):
}
}
}
// How often, and how many times, the move reads mesh-config's resolved values back before it says they differ.
var (
readBackEvery = 2 * time.Second
readBackTries = 15
)
func differences(resolved []resolvedEntry, p configMovePlan) ([]string, error) {
var differs []string
for _, module := range movedKeys(p.modules) {
for _, node := range p.modules[module] {
laid, err := configLaid(resolved, node, module)
if err != nil {
return nil, err
}
want := p.expected[node+"."+module]
if want == nil {
want = map[string]any{}
}
for _, k := range movedKeys(mergedKeys(laid, want)) {
got, has := laid[k]
w, wants := want[k]
switch {
case !wants:
differs = append(differs, fmt.Sprintf("%s on %s resolves %s from a level no layer says; clear it "+
"there, or set it in a layer first", module, node, k))
case !has:
differs = append(differs, fmt.Sprintf("%s on %s does not resolve %s from the value moved", module, node, k))
case !reflect.DeepEqual(normalJSON(got), normalJSON(w)):
differs = append(differs, fmt.Sprintf("%s on %s resolves %s as %s, and its layers lay %s", module,
node, k, compactJSON(got), compactJSON(w)))
}
}
}
}
return differs, nil
}
func mergedKeys(a, b map[string]any) map[string]bool {
out := map[string]bool{}
for k := range a {
out[k] = true
}
for k := range b {
out[k] = true
}
return out
}
func normalJSON(v any) any {
raw, _ := json.Marshal(v)
var out any
_ = json.Unmarshal(raw, &out)
return out
}
func compactJSON(v any) string {
raw, _ := json.Marshal(v)
return string(raw)
}
func sameJSON(raw json.RawMessage, v any) bool {
var held any
if err := json.Unmarshal(raw, &held); err != nil {
return false
}
return reflect.DeepEqual(held, normalJSON(v))
}
// busConfig is mesh-config on this controller's bus: its buckets read directly, a value set through its verb in
// the controller's own name, so mesh-config takes a trusted value as given at the controller's terminal (ADR 0348 §8).
type busConfig struct{}
func (busConfig) Values(ctx context.Context) (map[string]json.RawMessage, error) {
out := map[string]json.RawMessage{}
err := onTheBus(func(conn *nats.Conn) error {
js, err := jetstream.New(conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
kv, err := js.KeyValue(ctx, broker.BucketName("mesh-config", "values"))
if err != nil {
return err
}
w, err := kv.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
for e := range w.Updates() {
if e == nil {
return nil
}
var v struct {
Value json.RawMessage `json:"value"`
}
if err := json.Unmarshal(e.Value(), &v); err != nil {
return fmt.Errorf("mesh-config holds %q, which is not a value: %w", e.Key(), err)
}
out[e.Key()] = v.Value
}
return errors.New("mesh-config's values stopped answering")
})
return out, err
}
func (busConfig) Resolved(ctx context.Context) ([]resolvedEntry, error) { return readResolved(ctx) }
func (busConfig) Set(ctx context.Context, level configLevel, key string, value any, by, why string) error {
args := map[string]any{"action": "set", "key": key, "value": value, "why": why, "by": by}
if level.Node != "" {
args["node"] = level.Node
}
if level.Module != "" {
args["module"] = level.Module
}
return onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatToolAs(ctx, conn, broker.ControllerName, "mesh-config", "config", args, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
return nil
})
}
// moveConfigCommand is `move-config`: the move of config (issue 664).
func moveConfigCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("move-config", flag.ContinueOnError)
asJSON := set.Bool("json", false, "answer as JSON")
confirm := set.Bool("confirm", false, "set the values and switch, rather than only say what would move")
f := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 0 {
return errors.New("move-config [--json] [--confirm --why <text>]")
}
if !*confirm && f.given() {
return errors.New("move-config takes --why only with --confirm: without it nothing is moved. Nothing was done")
}
if *confirm {
if err := f.require("move-config"); err != nil {
return err
}
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if *confirm {
f.record(ctx, "move-config", nil)
}
why := ""
if *confirm {
why = strings.TrimSpace(*f.why)
}
answer, err := runConfigMove(ctx, open.inventory, busConfig{}, *confirm, "the move of config, run by "+link.Caller(), why)
if *asJSON {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if encErr := encoder.Encode(answer); encErr != nil {
return encErr
}
return err
}
fmt.Print(describeConfigMove(answer))
return err
}
// describeConfigMove says the answer as a person reads it.
func describeConfigMove(a configMoveAnswer) string {
var b strings.Builder
verb := "set"
if a.DryRun {
verb = "would set"
}
for _, s := range a.Sets {
state := verb
if s.Already {
state = "holds already"
}
fmt.Fprintf(&b, "%s %s = %s at %s (from %s)\n", state, s.Key, compactJSON(s.Value), s.configLevel,
strings.Join(s.From, " and "))
}
for _, s := range a.Unread {
fmt.Fprintf(&b, "emptied, set nowhere: %s\n", s)
}
for _, s := range a.Stays {
fmt.Fprintf(&b, "stays: %s\n", s)
}
for _, s := range a.Problems {
fmt.Fprintf(&b, "refused: %s\n", s)
}
switch {
case len(a.Problems) > 0:
b.WriteString("nothing was set: the move refuses until each is put right\n")
case a.DryRun:
fmt.Fprintf(&b, "%d layer(s), %d value(s); nothing was set — with --confirm --why <text> they are\n",
len(a.Moved), len(a.Sets))
case a.Switched:
fmt.Fprintf(&b, "%d layer(s) moved into mesh-config, each kept whole in its history first; the controller "+
"reads mesh-config where it read them\n", len(a.Moved))
}
return b.String()
}
+785
View File
@@ -0,0 +1,785 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
)
// configuredMesh is every live settings layer (issue 664) — read with the settings verb on 2026-10-15, on the four
// machines as anchor, laptop, workstation and desktop, the whole mesh's with no node; personal values stood in for,
// and the placement words left out, which move-placement moves first — with each module in the config form shaped
// as its definition on the feature branch: its config, its files (templates inlined) and directories, placeholders
// of kinds this test's mesh does not hold stood in for. A module still in the settings form (the media catalogue's,
// and kernel) stands in as a mergeable file that takes every key. `on` is where each is assigned, as live.
type configuredMesh struct {
Modules []json.RawMessage `json:"modules"`
Layers []configuredLayer `json:"layers"`
On map[string][]string `json:"on"`
// Predefined is the catalogue's list of predefined keys, which a module falls back to.
Predefined json.RawMessage `json:"predefined"`
}
type configuredLayer struct {
Node string `json:"node"`
Module string `json:"module"`
Values map[string]any `json:"values"`
}
var fourMachines = []string{"anchor", "laptop", "workstation", "desktop"}
// aConfiguredMesh is aMesh with two machines more, every module of the configured mesh registered and assigned, and
// its layers held as the store holds them once move-placement has run.
func aConfiguredMesh(t *testing.T) (*stores, configuredMesh, map[string]map[string]any) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for i, name := range []string{"workstation", "desktop"} {
record, err := inv.AddNode(ctx, name)
if err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(ctx, name, name+".example:51820", "here", false, fmt.Sprintf("10.77.0.%d", i+3)); err != nil {
t.Fatal(err)
}
if err := inv.RecordProfile(ctx, record.ID, map[string]any{"capabilities": []any{
map[string]any{"name": "container-runtime", "present": true},
map[string]any{"name": "systemd", "present": true}}}); err != nil {
t.Fatal(err)
}
if err := inv.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, name, overlay.Name); err != nil {
t.Fatal(err)
}
}
raw, err := os.ReadFile(filepath.Join("testdata", "config-moved", "mesh.json"))
if err != nil {
t.Fatal(err)
}
var w configuredMesh
if err := json.Unmarshal(raw, &w); err != nil {
t.Fatal(err)
}
list, err := catalogue.ParsePredefined(w.Predefined)
if err != nil {
t.Fatal(err)
}
catalogue.UsePredefined(list)
served := map[string]map[string]any{}
for _, text := range w.Modules {
var shape map[string]json.RawMessage
if err := json.Unmarshal(text, &shape); err != nil {
t.Fatal(err)
}
var name string
_ = json.Unmarshal(shape["module"], &name)
if s, ok := shape["serves-checked"]; ok {
var serves map[string]any
if err := json.Unmarshal(s, &serves); err != nil {
t.Fatal(err)
}
served[name] = serves
delete(shape, "serves-checked")
}
text, err := json.Marshal(shape)
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(text)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
register(t, open, m)
}
for _, l := range w.Layers {
values, err := json.Marshal(l.Values)
if err != nil {
t.Fatal(err)
}
if l.Node == "" {
inv.ExecForTest(t, `insert into settings (node, module, values) values (null, $1, $2)`, l.Module, string(values))
continue
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.Node, l.Module, string(values))
}
for _, module := range movedKeys(w.On) {
for _, node := range w.On[module] {
if _, err := assign(ctx, open, node, module); err != nil {
t.Fatalf("assign %s %s: %v", node, module, err)
}
}
}
return open, w, served
}
// standInMeshConfig is mesh-config as this test holds it: each value at its level, and resolved from the catalogue
// the store holds as mesh-config resolves (novox/hq ADR 0348 §2): the module's own value on the node, else node
// config then mesh config for the predefined key it falls back to, else its default; a required value nothing sets is
// unset.
type standInMeshConfig struct {
open *stores
values map[string]any
sets []string
// asked is each call of its verb: the action and the level's key.
asked []string
}
func (c *standInMeshConfig) Values(context.Context) (map[string]json.RawMessage, error) {
out := map[string]json.RawMessage{}
for k, v := range c.values {
raw, err := json.Marshal(v)
if err != nil {
return nil, err
}
out[k] = raw
}
return out, nil
}
func (c *standInMeshConfig) Set(_ context.Context, level configLevel, key string, value any, by, why string) error {
if by == "" || why == "" {
return fmt.Errorf("%s at %s is set without who or why", key, level)
}
c.values[level.valueKey(key)] = normalJSON(value)
c.sets = append(c.sets, level.valueKey(key))
return nil
}
func (c *standInMeshConfig) Resolved(ctx context.Context) ([]resolvedEntry, error) {
entries, err := c.open.inventory.Catalogued(ctx)
if err != nil {
return nil, err
}
var out []resolvedEntry
for _, e := range entries {
m := e.Manifest
if !m.DeclaresConfig() {
continue
}
for _, node := range e.On {
for _, key := range movedKeys(m.Settings) {
d := m.Settings[key]
r := resolvedEntry{Node: node, Module: m.Module, Key: key, Source: "default"}
value, has := c.values["module."+node+"."+m.Module+"."+key]
switch {
case has:
r.Source = "module"
case d.Predefined != "" && c.values["node."+node+"."+d.Predefined] != nil:
value, r.Source = c.values["node."+node+"."+d.Predefined], "node"
case d.Predefined != "" && c.values["mesh."+d.Predefined] != nil:
value, r.Source = c.values["mesh."+d.Predefined], "mesh"
case d.Required:
r.Source = "unset"
default:
value = d.Default
}
if r.Source != "unset" {
if r.Value, err = json.Marshal(value); err != nil {
return nil, err
}
}
out = append(out, r)
}
}
}
return out, nil
}
// laid is the stand-in as composition reads it once the switch is on (inventory.ConfigValues).
func (c *standInMeshConfig) laid(ctx context.Context, node, module string) (map[string]any, error) {
resolved, err := c.Resolved(ctx)
if err != nil {
return nil, err
}
return configLaid(resolved, node, module)
}
// snapshotFacts is the facts snapshot as gatherFacts gives it.
type snapshotFacts = snapshot.Facts
// toldEveryMachine is what each of the four machines is told — every resource and the filter's rules — and what
// each module that serves something serves, settled from its layers, as JSON a person can diff.
func toldEveryMachine(t *testing.T, open *stores, served map[string]map[string]any, on map[string][]string) map[string]string {
t.Helper()
ctx := t.Context()
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
told := map[string]string{}
for _, node := range fourMachines {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
rules, err := plan.Rules(with)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
settles := map[string]any{}
for module, serves := range served {
if !containsString(on[module], node) {
continue
}
layers, err := open.inventory.SettingsFor(ctx, node, module)
if err != nil {
t.Fatal(err)
}
for name, s := range serves {
fact, _ := s.(map[string]any)
settled, err := catalogue.Settle(fact, catalogue.WithDefaults(shelf[module], layers))
if err != nil {
settles[module+" serves "+name] = "error: " + err.Error()
continue
}
settles[module+" serves "+name] = settled
}
}
got, err := json.MarshalIndent(map[string]any{"resources": composed.Resources, "left-out": composed.LeftOut,
"rules": rules, "served": settles}, "", " ")
if err != nil {
t.Fatal(err)
}
told[node] = string(got)
}
return told
}
// effectiveEverywhere is each module's every value on each machine it is assigned to, as `settings show` and
// `settings preferences` say them (catalogue.Effective): the value, never where it came from, which the move
// changes by design; a key a layer says by its file's name is read as its config key.
func effectiveEverywhere(t *testing.T, open *stores, w configuredMesh) map[string]any {
t.Helper()
ctx := t.Context()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
out := map[string]any{}
for _, module := range movedKeys(w.On) {
if !shelf[module].DeclaresConfig() {
continue
}
for _, node := range w.On[module] {
layers, err := open.inventory.SettingsFor(ctx, node, module)
if err != nil {
t.Fatal(err)
}
// A layer's old spelling of a key (nodered's timeZone) is listed beside the key it is the file's name
// for, and reaches the file as that key does (issue 751): read as that key, which the move sets.
as := catalogue.TakenAs(shelf[module])
for _, v := range catalogue.Effective(shelf[module], layers) {
if key, old := as[v.Key]; old {
out[node+"."+module+"."+key] = normalJSON(v.Value)
continue
}
if _, said := out[node+"."+module+"."+v.Key]; said && v.FromDefault {
continue
}
out[node+"."+module+"."+v.Key] = normalJSON(v.Value)
}
}
}
return out
}
func setAt(a configMoveAnswer, key string) []configSet {
var out []configSet
for _, s := range a.Sets {
if s.valueKey(s.Key) == key {
out = append(out, s)
}
}
return out
}
// **The settings layers move into mesh-config and no machine is told anything new** (novox/hq ADR 0348 §9, design
// 57 K5, issue 664). Every live layer is seeded as the store holds it once move-placement has run, on modules shaped
// as their definitions, and every machine composed. The move, asked without confirming, says each value it would set
// and where from, and sets nothing; confirmed, it sets them in mesh-config, reads them back, empties the layers it
// moved and switches. Each machine is then told byte for byte what it was told before, every module serves the same,
// and `settings` lists the same values `config` resolves. Asked again, it sets nothing.
func TestEveryMachineIsToldTheSameAfterTheMoveOfConfig(t *testing.T) {
open, w, served := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
before := toldEveryMachine(t, open, served, w.On)
listedBefore := effectiveEverywhere(t, open, w)
factsBefore, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
asked, err := runConfigMove(ctx, inv, config, false, "the move of config, run by a test", "")
if err != nil || !asked.DryRun || asked.Switched || len(asked.Problems) > 0 || len(config.sets) > 0 {
t.Fatalf("the move, not confirmed, says %+v (%v) and set %v", asked, err, config.sets)
}
// nodered's live layer says the file's name for its time zone: set under the config key (issue 751).
if s := setAt(asked, "module.anchor.nodered.time-zone"); len(s) != 1 || s[0].Value != "Europe/Brussels" ||
!strings.HasSuffix(s[0].From[0], ": timeZone") {
t.Errorf("nodered's time zone moves as %+v", s)
}
for _, s := range asked.Sets {
if s.Key == "timeZone" {
t.Errorf("a key is set under the file's name: %+v", s)
}
}
// claude-code's whole-mesh managed settings become mesh config for agent-managed-settings (ADR 0348's
// consequences), and its whole-mesh layer sets no module value.
if s := setAt(asked, "mesh.agent-managed-settings"); len(s) != 1 || s[0].From[0] != "claude-code on the whole mesh: managed_settings" {
t.Errorf("claude-code's managed settings move as %+v", s)
}
for _, node := range fourMachines {
if s := setAt(asked, "module."+node+".claude-code.managed_settings"); len(s) > 0 {
t.Errorf("claude-code's managed settings are also set on %s: %+v", node, s)
}
}
// A whole-mesh layer of a value no predefined key holds goes to each machine it is assigned to, under that
// machine's layer: power's on the workstation is the workstation's, its own elsewhere.
if s := setAt(asked, "module.workstation.power.handle-power-key"); len(s) != 1 || s[0].Value != "suspend" {
t.Errorf("power's key on the workstation moves as %+v", s)
}
if s := setAt(asked, "module.anchor.power.handle-power-key"); len(s) != 1 || s[0].Value != "poweroff" {
t.Errorf("power's key on the anchor moves as %+v", s)
}
for _, s := range asked.Sets {
if s.Level == "node" {
t.Errorf("a machine's layer became node config: %+v", s)
}
}
var stays []string
for _, s := range asked.Stays {
stays = append(stays, strings.Fields(s)[0])
}
sort.Strings(stays)
if want := []string{"bazarr", "bazarr", "bookshelf", "jackett", "kernel", "kometa", "lidarr", "nzbget", "ombi",
"plex", "qbittorrent", "radarr", "sonarr", "tautulli"}; len(stays) < 10 || !containsString(stays, "kernel") {
t.Errorf("the layers that stay are %v, want those of %v", stays, want)
}
if after := toldEveryMachine(t, open, served, w.On); !reflect.DeepEqual(after, before) {
t.Fatal("a dry run changed what a machine is told")
}
moved, err := runConfigMove(ctx, inv, config, true, "the move of config, run by a test", "release 57 is delivered")
if err != nil || moved.DryRun || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if len(config.sets) != len(asked.Sets) {
t.Fatalf("the move set %d values, and said it would set %d", len(config.sets), len(asked.Sets))
}
if held := inv.QueryForTest(t, `select module from settings s where values <> '{}'
and module in (select name from module where manifest ? 'config')`); len(held) != 0 {
t.Fatalf("a layer of a module in the config form still holds values: %v", held)
}
after := toldEveryMachine(t, open, served, w.On)
for _, node := range fourMachines {
if after[node] != before[node] {
t.Errorf("%s is told something else after the move:\n%s\nwas:\n%s", node, after[node], before[node])
}
}
if listedAfter := effectiveEverywhere(t, open, w); !reflect.DeepEqual(listedAfter, listedBefore) {
t.Errorf("settings lists other values after the move:\n%v\nwas:\n%v", listedAfter, listedBefore)
}
// And config lists what settings listed: each module's resolved value on each machine is the value settings
// said before the move.
resolved, err := config.Resolved(ctx)
if err != nil {
t.Fatal(err)
}
for _, r := range resolved {
said, listed := listedBefore[r.Node+"."+r.Module+"."+r.Key]
if r.Source == "unset" {
if listed && said != nil {
t.Errorf("%s of %s on %s is unset in config, and settings said %v", r.Key, r.Module, r.Node, said)
}
continue
}
var v any
_ = json.Unmarshal(r.Value, &v)
if !listed || !reflect.DeepEqual(v, said) {
t.Errorf("config resolves %s of %s on %s as %v, and settings said %v", r.Key, r.Module, r.Node, v, said)
}
}
// The facts snapshot carries mesh-config's values for the merge gate, as each machine's layer of each module, and
// each machine's declaration composes to the digest it did before the move.
factsAfter, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
// Machines are named in a snapshot by stand-ins of its own; each is found by what it composes.
digests := func(f snapshotFacts) ([]string, string) {
var out []string
zone := ""
for _, m := range f.Machines {
out = append(out, m.Declaration.Digest)
for _, l := range m.Settings {
if l.Module == "nodered" {
zone = compactJSON(l.Values["time-zone"])
}
}
}
sort.Strings(out)
return out, zone
}
was, _ := digests(factsBefore)
now, zone := digests(factsAfter)
if zone != `"Europe/Brussels"` {
t.Errorf("the facts snapshot does not carry nodered's time zone from mesh-config: %s", zone)
}
if len(now) != len(fourMachines) || now[0] == "" || !reflect.DeepEqual(now, was) {
t.Errorf("the declarations in the facts snapshot are %v after the move, %v before", now, was)
}
for _, s := range factsAfter.Settings {
if _, isConfig := map[string]bool{"betterbird": true, "claude-code": true, "power": true, "xdg": true}[s.Module]; isConfig &&
len(s.Values) > 0 {
t.Errorf("the facts snapshot still carries the whole mesh's layer of %s: %v", s.Module, s.Values)
}
}
config.sets = nil
again, err := runConfigMove(ctx, inv, config, true, "the move of config, run by a test", "again")
if err != nil || len(config.sets) != 0 || len(again.Moved) != 0 {
t.Fatalf("asked again, the move set %v and says %+v (%v)", config.sets, again, err)
}
if after := toldEveryMachine(t, open, served, w.On); !reflect.DeepEqual(after, before) {
t.Fatal("the move asked again changed what a machine is told")
}
}
// **The move refuses what it cannot move as it composes, and sets nothing** (issue 664): a layer that still holds
// placement (move-placement has not run), a key its module does not declare, and a value its declaration refuses —
// each said by name; and a value mesh-config resolves from a level no layer says (it would be laid once the switch
// is on) leaves every layer and the switch as they were.
func TestTheMoveOfConfigRefusesWhatWouldChangeAMachine(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
inv.ExecForTest(t, `update settings set values = values || '{"endpoints": {"web": {"reach": "public"}}}'
where module = 'nodered'`)
inv.ExecForTest(t, `update settings set values = values || '{"colour": "blue"}' where module = 'unifi'`)
inv.ExecForTest(t, `update settings set values = '{"font-size": 900}' where module = 'dunst'`)
answer, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered")
if err == nil || answer.Switched || len(config.sets) > 0 {
t.Fatalf("the move went ahead: %+v (%v), set %v", answer, err, config.sets)
}
for _, want := range []string{"nodered on anchor still holds placement (endpoints): run move-placement first",
`unifi on anchor says "colour", which unifi does not declare`, "font-size at dunst's own value on workstation"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
inv.ExecForTest(t, `update settings set values = values - 'endpoints' where module = 'nodered'`)
inv.ExecForTest(t, `update settings set values = values - 'colour' where module = 'unifi'`)
inv.ExecForTest(t, `update settings set values = '{"font-size": 13, "width": 500}' where module = 'dunst'`)
// Set in mesh-config before the move, on a module whose layers say nothing of it: the switch would lay it.
config.values["module.anchor.grafana."+movedKeys(func() map[string]catalogue.SettingDeclaration {
shelf, _ := inv.Catalogue(ctx)
return shelf["grafana"].Settings
}())[0]] = "set before the move"
answer, err = runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered")
if err == nil || !strings.Contains(err.Error(), "grafana on anchor resolves") || answer.Switched {
t.Fatalf("a value no layer says was laid by the switch: %+v (%v)", answer, err)
}
if moved, _ := inv.ConfigMoved(ctx); moved.Moved {
t.Fatal("the switch is on after a refused move")
}
if held := inv.QueryForTest(t, `select count(*) from settings where values <> '{}'`); held[0][0] == "0" {
t.Fatal("a refused move emptied the layers")
}
}
// The verb's command line: a dry run unless confirmed, and a confirmed move says why, recorded as a person's step of
// the release, never a repair (issue 664).
func TestTheMoveOfConfigIsADryRunUnlessConfirmed(t *testing.T) {
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "move-config --json"},
{map[string]any{"confirm": "true", "why": "release 57 is delivered"}, "move-config --json --confirm --why release 57 is delivered"},
} {
argv, err := argvFor("move-config", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%v: %q %v, want %q", c.args, argv, err, c.want)
}
}
for _, args := range []map[string]any{{"confirm": "true"}, {"why": "tidy"}, {"node": "anchor"}} {
if argv, err := argvFor("move-config", args); err == nil {
t.Errorf("%v was composed as %q", args, argv)
}
}
if repairingCommand([]string{"move-config", "--json", "--confirm", "--why", "x"}) != "move-config" {
t.Error("a confirmed move through the generic verb would go unrecorded")
}
if !personsDecision(link.HandAct{Verb: "move-config"}) {
t.Error("a move of the release would count toward a healer the mesh lacks")
}
}
// ask is mesh-config's verb as the stand-in answers it: list, set, clear and propose.
func (c *standInMeshConfig) ask(ctx context.Context, args map[string]any) (json.RawMessage, error) {
str := func(k string) string { s, _ := args[k].(string); return s }
level := configLevel{Level: "mesh"}
switch {
case str("module") != "" && str("node") != "":
level = configLevel{Level: "module", Node: str("node"), Module: str("module")}
case str("node") != "":
level = configLevel{Level: "node", Node: str("node")}
}
c.asked = append(c.asked, str("action")+" "+level.valueKey(str("key")))
switch str("action") {
case "list":
resolved, err := c.Resolved(ctx)
if err != nil {
return nil, err
}
out := map[string]any{"resolved": []resolvedEntry{}, "set": []any{}}
var listed []resolvedEntry
for _, r := range resolved {
if r.Module == str("module") && (str("node") == "" || r.Node == str("node")) {
listed = append(listed, r)
}
}
out["resolved"] = listed
var set []any
for _, k := range movedKeys(c.values) {
parts := strings.SplitN(k, ".", 4)
if len(parts) == 4 && parts[0] == "module" && parts[2] == str("module") {
set = append(set, map[string]any{"level": configLevel{Level: "module", Node: parts[1], Module: parts[2]},
"key": parts[3], "value": c.values[k], "by": "a test"})
}
}
out["set"] = set
return json.Marshal(out)
case "set":
if str("why") == "" || str("by") == "" {
return nil, fmt.Errorf("a set without why or by: %v", args)
}
return nil, c.Set(ctx, level, str("key"), args["value"], str("by"), str("why"))
case "clear":
if _, had := c.values[level.valueKey(str("key"))]; !had {
return nil, fmt.Errorf("nothing is set for %s", str("key"))
}
delete(c.values, level.valueKey(str("key")))
return json.RawMessage(`{}`), nil
case "propose":
return json.RawMessage(`{"proposal":"p-1"}`), nil
}
return nil, fmt.Errorf("the stand-in does not %s", str("action"))
}
// **`settings` answers by forwarding to `config`** (novox/hq ADR 0348 §1, design 57 K5, issue 664): once the move
// has run, a module in the config form is shown from mesh-config, its values set and cleared there as its own on the
// machine (placement still beside the assignment), and proposed through it; a trusted value is refused through a
// verb, a module value for the whole mesh is refused, and a module still in the settings form keeps today's verb.
func TestTheSettingsVerbForwardsToConfigOnceTheValuesMoved(t *testing.T) {
open, w, served := aConfiguredMesh(t)
ctx := t.Context()
inv := open.inventory
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
savedRead, savedAsk := readConfigValues, configAsk
readConfigValues, configAsk = config.laid, config.ask
t.Cleanup(func() { readConfigValues, configAsk = savedRead, savedAsk })
if _, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil {
t.Fatal(err)
}
shown := stdoutOf(t, func() error { return settingsCommand(ctx, []string{"show", "nodered", "--node", "anchor"}) })
for _, want := range []string{`"time-zone": "Europe/Brussels"`, `time-zone = "Europe/Brussels" (module)`,
"mesh-config.config", `"mqtt": {`} {
if !strings.Contains(shown, want) {
t.Errorf("settings show does not say %q:\n%s", want, shown)
}
}
if strings.Contains(shown, "timeZone") {
t.Errorf("settings show still says the file's name:\n%s", shown)
}
config.asked = nil
said := stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12, "width": 500}`, "--node", "workstation"})
})
if config.values["module.workstation.dunst.font-size"] != float64(12) || !strings.Contains(said, "~ font-size") ||
containsString(config.asked, "set module.workstation.dunst.width") {
t.Fatalf("settings set reached mesh-config as %v (%v):\n%s", config.values, config.asked, said)
}
told := toldEveryMachine(t, open, served, w.On)["workstation"]
if !strings.Contains(told, "12") {
t.Error("the workstation is not told the font size set through settings")
}
if err := settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12}`, "--node", "workstation"}); err == nil ||
!strings.Contains(err.Error(), "would no longer set width") {
t.Fatalf("a set dropping a value without --replace: %v", err)
}
if err := settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12}`}); err == nil ||
!strings.Contains(err.Error(), "no own value for the whole mesh") {
t.Fatalf("a module value for the whole mesh: %v", err)
}
t.Setenv(verbVar, "settings")
for _, c := range []struct{ module, node, values, key, held, why string }{
// A file root reads: refused as before the move (novox/hq issue 339).
{"hostname", "anchor", `{"hostname": "elsewhere"}`, "module.anchor.hostname.hostname", "Ace", "issue 339"},
{"messenger", "laptop", `{"desktop-machines": ["g14", "shanks"], "time-zone": "UTC"}`,
"module.laptop.messenger.time-zone", "Europe/Brussels", "issue 339"},
} {
err := settingsCommand(ctx, []string{"set", c.module, c.values, "--node", c.node})
if err == nil || !strings.Contains(err.Error(), c.why) || config.values[c.key] != c.held {
t.Fatalf("a trusted value of %s through a verb: %v, holds %v", c.module, err, config.values[c.key])
}
}
// And a trusted value no file makes the terminal's is refused as trusted, since mesh-config takes the controller's
// word for it (ADR 0348 §8).
if err := refuseTrustedThroughAVerb(ctx, inv, "messenger", "laptop", []string{"time-zone"}); err == nil ||
!strings.Contains(err.Error(), "ADR 0348 §8") {
t.Fatalf("a trusted value through a verb: %v", err)
}
if err := refuseTrustedThroughAVerb(ctx, inv, "dunst", "workstation", []string{"width"}); err != nil {
t.Fatalf("a value nobody trusts through a verb: %v", err)
}
stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12, "width": 600}`, "--node", "workstation"})
})
if config.values["module.workstation.dunst.width"] != float64(600) {
t.Fatalf("a value nobody trusts was not set through a verb: %v", config.values["module.workstation.dunst.width"])
}
t.Setenv(verbVar, "")
stdoutOf(t, func() error { return settingsCommand(ctx, []string{"clear", "dunst", "--node", "workstation"}) })
for _, k := range []string{"font-size", "width"} {
if _, held := config.values["module.workstation.dunst."+k]; held {
t.Errorf("settings clear left dunst's %s on the workstation", k)
}
}
config.asked = nil
proposed := stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"propose", "hostname", `{"hostname": "elsewhere"}`, "--node", "anchor"})
})
if !containsString(config.asked, "propose module.anchor.hostname.hostname") || !strings.Contains(proposed, "p-1") {
t.Fatalf("settings propose reached mesh-config as %v:\n%s", config.asked, proposed)
}
config.asked = nil
kept := stdoutOf(t, func() error { return settingsCommand(ctx, []string{"show", "kernel", "--node", "laptop"}) })
if !strings.Contains(kept, "transparent-huge-pages") || len(config.asked) > 0 {
t.Fatalf("a module in the settings form is shown as %q, asking mesh-config %v", kept, config.asked)
}
}
// **An approval for a module whose config has moved is refused, and writes nothing** (issue 830, novox/hq ADR 0348):
// a proposal raised while grafana read its layer (it had none, so the layer is as the operator was shown it) and
// approved after the move would put its values back into the layer, ReadsConfig would turn false, and the module
// would compose from emptied layers everywhere.
func TestAnApprovalRaisedBeforeTheMoveOfConfigIsRefusedAfterIt(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
key := movedKeys(func() map[string]catalogue.SettingDeclaration { return shelf["grafana"].Settings }())[0]
values := map[string]any{key: "raised before the move"}
p := settingsProposal{Module: "grafana", Node: "anchor", Values: values, From: "g14/claude-code", At: time.Now(),
Digest: layerDigest(values), BeforeDigest: layerDigest(nil)}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has {
t.Fatalf("grafana has a layer on anchor before the test: %v %v", has, err)
}
if moved, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Fatalf("grafana does not read config after the move: %v %v", reads, err)
}
_, err = setLayerIn(open)(ctx, p, "s30", "approved by the operator")
if err == nil || !strings.Contains(err.Error(), "moved to mesh-config since the operator was shown the change") ||
!strings.Contains(err.Error(), "propose it again") {
t.Fatalf("the stale approval: %v", err)
}
if layer, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has && len(layer) > 0 {
t.Errorf("the refused approval wrote the layer: %v %v %v", layer, has, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Errorf("the refused approval turned ReadsConfig off: %v %v", reads, err)
}
}
// **A proposal of placement only is set after the move, as before** (issue 830): forwardPropose sends it down the
// old path on purpose, so refusing it would loop the operator (refused, proposed again, refused).
func TestAPlacementOnlyApprovalRaisedBeforeTheMoveIsSetAfterIt(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
values := map[string]any{catalogue.PlacesSetting: map[string]any{"data": "/srv/grafana-data"}}
p := settingsProposal{Module: "grafana", Node: "anchor", Values: values, From: "g14/claude-code", At: time.Now(),
Digest: layerDigest(values), BeforeDigest: layerDigest(nil)}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has {
t.Fatalf("grafana has a layer on anchor before the test: %v %v", has, err)
}
if moved, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if _, err := setLayerIn(open)(ctx, p, "s31", "approved by the operator"); err != nil {
t.Fatalf("the placement-only approval after the move: %v", err)
}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || !has {
t.Errorf("the approval did not set the layer: %v %v", has, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Errorf("the approval turned ReadsConfig off: %v %v", reads, err)
}
}
+128
View File
@@ -0,0 +1,128 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// **Where the controller reads config values once the move has run** (novox/hq ADR 0348 §9, design 57 K5, issue 664):
// mesh-config_resolved, the bucket mesh-config alone writes, keyed `<node>.<module>.<key>`, each value with the
// source it came from. Read only for a module the switch reaches (inventory.ReadsConfig), so a controller before the
// move never asks the bus for config. The exception to ADR 0348 §9 ("the controller holds no config value") that
// ADR 0349 clause 17's old way needs, until design 57's R1 hands every module to the provisioner.
// resolvedEntry is one key of mesh-config_resolved as the controller reads it: the value and where it came from.
type resolvedEntry struct {
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
Source string `json:"source"`
}
// The sources a resolved value is laid over a module's files from: mesh config, node config and the module's own
// value. Its manifest's default is composition's already, and a taken key's default is never laid over its file;
// a required value nothing sets (`unset`) is refused by composition by name, as it is today.
var laidSources = map[string]bool{"mesh": true, "node": true, "module": true}
// configLaid is the values of one module on one node that are laid over its files, from the resolved entries.
func configLaid(entries []resolvedEntry, node, module string) (map[string]any, error) {
out := map[string]any{}
for _, e := range entries {
if e.Node != node || e.Module != module || !laidSources[e.Source] {
continue
}
var v any
if err := json.Unmarshal(e.Value, &v); err != nil {
return nil, fmt.Errorf("mesh-config's value %s of %s on %s cannot be read: %w", e.Key, module, node, err)
}
out[e.Key] = v
}
return out, nil
}
// resolvedRead is what one read of the whole bucket held, and when it was read.
type resolvedRead struct {
mu sync.Mutex
at time.Time
entries []resolvedEntry
}
// resolvedFresh is how long one read of the bucket answers: a plan reads every module of a machine, and one read
// serves them all; a value set since is read by the next composition.
const resolvedFresh = 5 * time.Second
var theResolved resolvedRead
// readConfigValues is where an opened inventory reads config values once the switch is on; a test replaces it.
var readConfigValues inventory.ConfigValues = busConfigValues
// busConfigValues reads one module's config values on one node from mesh-config_resolved (inventory.ConfigValues).
func busConfigValues(ctx context.Context, node, module string) (map[string]any, error) {
theResolved.mu.Lock()
defer theResolved.mu.Unlock()
if theResolved.entries == nil || time.Since(theResolved.at) > resolvedFresh {
entries, err := readResolved(ctx)
if err != nil {
return nil, err
}
theResolved.entries, theResolved.at = entries, time.Now()
}
return configLaid(theResolved.entries, node, module)
}
// readResolved is every entry of mesh-config_resolved, read on the bus.
func readResolved(ctx context.Context) ([]resolvedEntry, error) {
entries := []resolvedEntry{}
err := onTheBus(func(conn *nats.Conn) error {
js, err := jetstream.New(conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
kv, err := js.KeyValue(ctx, broker.ConfigResolvedBucket)
if err != nil {
return fmt.Errorf("%s cannot be opened: %w", broker.ConfigResolvedBucket, err)
}
w, err := kv.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
return fmt.Errorf("%s was not read whole: %w", broker.ConfigResolvedBucket, ctx.Err())
case e, open := <-w.Updates():
if !open {
return errors.New(broker.ConfigResolvedBucket + " stopped answering")
}
if e == nil {
return nil
}
var r resolvedEntry
if err := json.Unmarshal(e.Value(), &r); err != nil {
return fmt.Errorf("%s holds %q, which is not a resolved value: %w", broker.ConfigResolvedBucket,
e.Key(), err)
}
if r.Node == "" || r.Module == "" || r.Key == "" || !strings.HasPrefix(e.Key(), r.Node+"."+r.Module+".") {
return fmt.Errorf("%s holds %q, whose value names another key", broker.ConfigResolvedBucket, e.Key())
}
entries = append(entries, r)
}
}
})
return entries, err
}
+297
View File
@@ -0,0 +1,297 @@
package main
import (
"context"
"crypto/ecdh"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The core's own credentials, sealed by the vault (novox/hq ADR 0349 §16, issue 755; the lead's decision on issue 755,
// notes 4 and 5).
//
// For a machine whose engine opens the vault's box (link.VaultBoxContract), each own secret of a module on the core's
// list (catalogue.CoreCredentialModules) travels as the vault sealed it to that machine's key, which the controller
// asks for on the send path (`get`, answered to the controller alone and sealed only to the key that machine's engine
// stated: mesh-vault's core.go) and carries without being able to open it. A machine below that contract is sent the
// controller's own sealed value, exactly as before.
//
// **The vault seals every answer afresh**, so the box last got is kept (inventory.CoreBox) and carried again while the
// vault's generation and the machine's key are the same: a declaration that changed with every answer would never be
// current. Every question but a send reads the kept box and asks nothing.
//
// **When the vault does not answer, the controller never falls back to its own seal.** A credential with a kept box
// keeps it — the machine keeps the file it has — and one with none leaves its module out of the declaration (the
// machine keeps what it holds of it, and a module not there yet is not placed). Either way the send raises a
// condition naming the vault and the machine, cleared by the first send the vault answers whole.
// engineOpensVaultBox says whether a machine's node-engine opens the vault's box, by its own newest statement.
func engineOpensVaultBox(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.VaultBoxContract, nil
}
// coreAnswer is the vault's `get` answer as the controller reads it: the name, the generation and the box, never
// opened.
type coreAnswer struct {
Name string `json:"name"`
Generation int `json:"generation"`
Sealed json.RawMessage `json:"sealed"`
}
// coreAsk asks the vault for one secret sealed to the given key.
type coreAsk func(name, publicKey string) (coreAnswer, error)
// coreBoxes is what the vault's boxes make of one machine's composition.
type coreBoxes struct {
// sealed is each core credential's box, by module and name (catalogue.Rendering.VaultSealed).
sealed map[string]map[string]json.RawMessage
// notPlaced is each core module left out, with why (catalogue.Rendering.NotPlaced).
notPlaced map[string]string
// keep is each box newly answered, to keep.
keep []inventory.CoreBox
// held is each credential the vault did not answer for whose kept box is carried again; silent is why the vault
// did not answer, per credential.
held, silent []string
}
// coreModulesOf is every module of the plan on the core's list whose own secrets this machine's declaration carries,
// with their plain own secret names, sorted: one handed to the provisioner has its secrets from the vault through the
// client, and one issued outside the mesh is never the vault's to seal here.
func coreModulesOf(modules []catalogue.Manifest, handed func(catalogue.Manifest) bool) map[string][]string {
out := map[string][]string{}
for _, m := range modules {
if !catalogue.CarriesVaultSealed(m.Module) || (handed != nil && handed(m)) {
continue
}
var names []string
for name, own := range m.OwnSecrets.Plain() {
if own.IssuedBy != catalogue.IssuedOutside {
names = append(names, name)
}
}
if len(names) > 0 {
sort.Strings(names)
out[m.Module] = names
}
}
return out
}
// sealingKeyPEM is a machine's sealing key, recorded raw in base64, as the SPKI PEM the vault seals to.
func sealingKeyPEM(raw string) (string, error) {
b, err := base64.StdEncoding.DecodeString(strings.TrimSpace(raw))
if err != nil || len(b) != 32 {
return "", errors.New("the machine's sealing key is not a raw X25519 key")
}
pub, err := ecdh.X25519().NewPublicKey(b)
if err != nil {
return "", err
}
der, err := x509.MarshalPKIXPublicKey(pub)
if err != nil {
return "", err
}
return string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: der})), nil
}
// coreBoxesOf decides each core credential of a machine: on a send, the vault's answer (the kept box while its
// generation and key are the same), the kept box where the vault did not answer, or the module left out where nothing
// was kept; on any other question, the kept box, or a stand-in the next send asks for. Pure: what to keep and what to
// say is answered, not done.
func coreBoxesOf(node, sealingKey string, core map[string][]string, kept map[string]map[string]inventory.CoreBox,
choosing Choosing, ask coreAsk) coreBoxes {
out := coreBoxes{sealed: map[string]map[string]json.RawMessage{}, notPlaced: map[string]string{}}
modules := make([]string, 0, len(core))
for m := range core {
modules = append(modules, m)
}
sort.Strings(modules)
for _, module := range modules {
if sealingKey == "" {
out.notPlaced[module] = fmt.Sprintf("%s has no sealing key, so the vault can seal %s's credentials to "+
"nothing there (novox/hq ADR 0349 §16)", node, module)
continue
}
key, err := sealingKeyPEM(sealingKey)
if err != nil {
out.notPlaced[module] = fmt.Sprintf("%s's credentials cannot be asked of the vault: %v", module, err)
continue
}
boxes := map[string]json.RawMessage{}
for _, name := range core[module] {
k, has := kept[module][name]
valid := has && k.SealedTo == sealingKey && len(k.Box) > 0
if choosing != Allocating {
if valid {
boxes[name] = k.Box
} else {
boxes[name] = catalogue.ForeseenVaultBox
}
continue
}
secret := "own/" + node + "/" + module + "/" + name
got, err := ask(secret, key)
if err == nil && (got.Generation <= 0 || canonicalBox(got.Sealed) == nil) {
err = errors.New("it answered no box")
}
if err != nil {
out.silent = append(out.silent, fmt.Sprintf("%s: %v", secret, err))
if valid {
boxes[name] = k.Box
out.held = append(out.held, module+"/"+name)
continue
}
out.notPlaced[module] = fmt.Sprintf("the vault did not give %s's credential %s for %s (%v), and the "+
"controller never seals it itself (novox/hq ADR 0349 §16): not placed until the vault answers",
module, name, node, err)
break
}
if valid && k.Generation == got.Generation {
boxes[name] = k.Box
continue
}
box := canonicalBox(got.Sealed)
boxes[name] = box
out.keep = append(out.keep, inventory.CoreBox{Module: module, Name: name, Generation: got.Generation,
SealedTo: sealingKey, Box: box})
}
if _, left := out.notPlaced[module]; !left {
out.sealed[module] = boxes
}
}
return out
}
// canonicalBox is the vault's box in the one form the declaration carries — the sealed box's fields, in their order,
// and nothing else — or nil when it is not one.
func canonicalBox(raw json.RawMessage) json.RawMessage {
var b struct {
V int `json:"v"`
Eph string `json:"eph"`
IV string `json:"iv"`
Tag string `json:"tag"`
Ct string `json:"ct"`
}
if err := json.Unmarshal(raw, &b); err != nil || b.V <= 0 || b.Eph == "" || b.Ct == "" {
return nil
}
out, _ := json.Marshal(b)
return out
}
// coreVault is how a send asks the vault for a core credential: its `get`, in the controller's name. A test stands
// in for it.
var coreVault = func(ctx context.Context) coreAsk {
vault := theVault(ctx)
return func(name, publicKey string) (coreAnswer, error) {
raw, err := vault("get", map[string]any{"name": name, "public_key": publicKey})
if err != nil {
return coreAnswer{}, err
}
var a coreAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return coreAnswer{}, fmt.Errorf("the vault's answer is not one: %w", err)
}
return a, nil
}
}
// coreBoxesFor is coreBoxesOf for one machine, read from and kept in the store, and said on a send.
func coreBoxesFor(ctx context.Context, inv *inventory.Inventory, node string, modules []catalogue.Manifest,
handed func(catalogue.Manifest) bool, choosing Choosing) (coreBoxes, error) {
core := coreModulesOf(modules, handed)
if len(core) == 0 {
if choosing == Allocating {
settleVaultSilent(ctx, node)
}
return coreBoxes{}, nil
}
key, err := inv.SealingKeyOf(ctx, node)
if err != nil {
return coreBoxes{}, err
}
kept, err := inv.KeptCoreBoxes(ctx, node)
if err != nil {
return coreBoxes{}, err
}
var ask coreAsk
if choosing == Allocating {
ask = coreVault(ctx)
}
out := coreBoxesOf(node, key, core, kept, choosing, ask)
for _, b := range out.keep {
if err := inv.KeepCoreBox(ctx, node, b); err != nil {
return coreBoxes{}, err
}
}
if choosing == Allocating {
sayVaultSilent(ctx, node, out)
}
return out, nil
}
// kindVaultSilent is a send that did not have the vault's box for a core credential of a machine.
const kindVaultSilent = "vault-silent"
// vaultSilentObservation is that, as a condition: `machine.<machine>.vault-silent`.
func vaultSilentObservation(node string, b coreBoxes) conditions.Observation {
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: node, Kind: kindVaultSilent, Token: kindVaultSilent,
Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource}
var said []string
if len(b.notPlaced) > 0 {
o.Severity = conditions.Urgent
var left []string
for m := range b.notPlaced {
left = append(left, m)
}
sort.Strings(left)
said = append(said, "not placed: "+strings.Join(left, ", "))
}
if len(b.held) > 0 {
said = append(said, "kept as last sent: "+strings.Join(b.held, ", "))
}
o.Summary = fmt.Sprintf("the vault did not answer for the core's credentials on %s; %s", node, strings.Join(said, "; "))
o.Said = o.Summary + " — " + strings.Join(b.silent, "; ")
return o
}
// sayVaultSilent raises the condition for a send the vault did not answer whole, and clears it for one it did.
func sayVaultSilent(ctx context.Context, node string, b coreBoxes) {
if len(b.silent) == 0 && len(b.notPlaced) == 0 {
settleVaultSilent(ctx, node)
return
}
fmt.Printf("HELD: %s\n", vaultSilentObservation(node, b).Said)
if conditionsFrom == nil {
return
}
if _, err := conditionsFrom.Observe(ctx, vaultSilentObservation(node, b)); err != nil {
fmt.Printf("the condition for the vault's silence on %s could not be raised: %v\n", node, err)
}
}
// settleVaultSilent clears the condition once a send had every core credential of the machine from the vault.
func settleVaultSilent(ctx context.Context, node string) {
if conditionsFrom == nil {
return
}
key := vaultSilentObservation(node, coreBoxes{}).Key()
if _, err := conditionsFrom.Clear(ctx, key, "the vault answered for every core credential of "+node); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
}
+220
View File
@@ -0,0 +1,220 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The core's own credentials travel sealed by the vault to their machine (novox/hq ADR 0349 §16, issue 755; the lead's
// decision, issue 755 notes 4 and 5): to an engine that opens the vault's box, the send carries the box the vault
// answered the controller and never the controller's own seal; when the vault does not answer, a kept box is carried
// again and a module with none is not placed, with a condition naming the vault and the machine. An engine below the
// contract composes exactly as before, and the vault is not asked.
// coreStandIn answers `get` with a box per name and generation, or its silence.
type coreStandIn struct {
asked []string
keys []string
generation int
silent error
}
func (v *coreStandIn) ask(name, key string) (coreAnswer, error) {
v.asked, v.keys = append(v.asked, name), append(v.keys, key)
if v.silent != nil {
return coreAnswer{}, v.silent
}
box := `{"v":1,"eph":"ZQ==","iv":"aQ==","tag":"dA==","ct":"` + strings.Repeat("c", len(v.asked)) + `"}`
return coreAnswer{Name: name, Generation: v.generation, Sealed: json.RawMessage(box)}, nil
}
func withVault(t *testing.T, v *coreStandIn) {
t.Helper()
before := coreVault
coreVault = func(context.Context) coreAsk { return v.ask }
t.Cleanup(func() { coreVault = before })
}
func aCoreMesh(t *testing.T, contract int) (*stores, *coreStandIn) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
client := aClientModule()
register(t, open, client)
if _, err := assign(ctx, open, "laptop", client.Module); err != nil {
t.Fatal(err)
}
if err := issueProvisionerClient(ctx, open.inventory, client, "laptop", broker.Broker{Fingerprint: "ab"},
"10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.RecordHealth(ctx, inventory.NodeHealth{Node: "laptop", Contract: contract,
SaidAt: time.Now()}); err != nil {
t.Fatal(err)
}
v := &coreStandIn{generation: 1}
withVault(t, v)
return open, v
}
func clientsBroker(resources []map[string]any) map[string]any {
for _, r := range resources {
if r["id"] == "mesh-provisioner-client.needs-"+broker.ProvisionerSecret {
return r
}
}
return nil
}
func vaultSilence(t *testing.T, node string) (conditions.Condition, bool) {
t.Helper()
c, found, err := conditionsFrom.Get(t.Context(), vaultSilentObservation(node, coreBoxes{}).Key())
if err != nil {
t.Fatal(err)
}
return c, found
}
func TestTheSendCarriesTheVaultsBoxForACoreCredentialAndKeepsIt(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
key, err := open.inventory.SealingKeyOf(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
f := clientsBroker(sentTo(t, open, "laptop"))
if f == nil || f["sealed"] != nil || f[catalogue.VaultSealedField] == nil {
t.Fatalf("the core credential is sent as %v: the vault's box, never the controller's seal", f)
}
if want := []string{"own/laptop/mesh-provisioner-client/" + broker.ProvisionerSecret}; !reflect.DeepEqual(vault.asked, want) {
t.Fatalf("the vault was asked %v", vault.asked)
}
if want, _ := sealingKeyPEM(key); vault.keys[0] != want {
t.Fatalf("the vault was asked to seal to %q, not the machine's key", vault.keys[0])
}
first, _ := json.Marshal(f[catalogue.VaultSealedField])
// The vault answers afresh with the same generation: the kept box is carried again, so the declaration is stable.
again, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
if string(again) != string(first) {
t.Fatalf("a fresh box of the same generation changed the declaration: %s, then %s", first, again)
}
// A rotation in the vault (a newer generation) is carried.
vault.generation = 2
rotated, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
if string(rotated) == string(first) {
t.Fatal("a newer generation was not carried")
}
if _, found := vaultSilence(t, "laptop"); found {
t.Fatal("a send the vault answered raised its silence")
}
}
func TestASilentVaultHoldsTheKeptBoxAndSaysSo(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
first, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
vault.silent = errors.New("no responders")
held := clientsBroker(sentTo(t, open, "laptop"))
got, _ := json.Marshal(held[catalogue.VaultSealedField])
if held == nil || held["sealed"] != nil || string(got) != string(first) {
t.Fatalf("a silent vault's credential is sent as %v: the box kept, never the controller's seal", held)
}
c, found := vaultSilence(t, "laptop")
if !found || !strings.Contains(c.Summary, "vault") || !strings.Contains(c.Summary, "laptop") {
t.Fatalf("no condition names the vault and the machine: %+v", c)
}
// The vault answers again: the condition clears.
vault.silent = nil
sentTo(t, open, "laptop")
if _, found := vaultSilence(t, "laptop"); found {
t.Fatal("the condition stayed after the vault answered")
}
}
func TestACoreModuleFirstPlacedWhileTheVaultIsSilentIsNotPlaced(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
vault.silent = errors.New("no responders")
plan, settings, err := planFor(t.Context(), open, "laptop")
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(t.Context(), open, "laptop", plan, settings)
if err != nil {
t.Fatal(err)
}
for _, r := range declared.Resources {
if strings.HasPrefix(r["id"].(string), "mesh-provisioner-client.") {
t.Fatalf("a core module was placed though the vault gave nothing for it: %v", r)
}
}
if why := declared.leftOutWhy["mesh-provisioner-client"]; !strings.Contains(why, "never seals it itself") {
t.Fatalf("left out %v", declared.leftOutWhy)
}
c, found := vaultSilence(t, "laptop")
if !found || c.Severity != conditions.Urgent || !strings.Contains(c.Summary, "not placed") {
t.Fatalf("the condition does not say why: %+v", c)
}
}
// An engine below VaultBoxContract is sent the controller's own seal, as before, and the vault is not asked.
func TestAnEngineBelowTheContractIsSentTheControllersSealAsBefore(t *testing.T) {
open, vault := aCoreMesh(t, link.HandedContract)
f := clientsBroker(sentTo(t, open, "laptop"))
if f == nil || f["sealed"] == nil || f[catalogue.VaultSealedField] != nil {
t.Fatalf("an older engine is sent %v", f)
}
if len(vault.asked) != 0 {
t.Fatalf("the vault was asked %v for an older engine", vault.asked)
}
}
// Every question but a send reads the kept box and asks nothing; with none kept, a stand-in the next send asks for.
func TestOnlyASendAsksTheVault(t *testing.T) {
core := map[string][]string{"nats": {"broker"}}
key := aPublicKey(t)
kept := map[string]map[string]inventory.CoreBox{"nats": {"broker": {Module: "nats", Name: "broker",
Generation: 1, SealedTo: key, Box: json.RawMessage(`{"v":1}`)}}}
never := func(string, string) (coreAnswer, error) {
t.Fatal("the vault was asked by a question that is not a send")
return coreAnswer{}, nil
}
for _, choosing := range []Choosing{Reading, Foreseeing} {
if got := coreBoxesOf("anchor", key, core, kept, choosing, never); string(got.sealed["nats"]["broker"]) != `{"v":1}` {
t.Errorf("read %s", got.sealed["nats"]["broker"])
}
if got := coreBoxesOf("anchor", key, core, nil, choosing, never); string(got.sealed["nats"]["broker"]) !=
string(catalogue.ForeseenVaultBox) {
t.Errorf("with none kept, read %s", got.sealed["nats"]["broker"])
}
}
// A box kept for another key of the machine is never carried.
other := coreBoxesOf("anchor", aPublicKey(t), core, kept, Allocating,
func(string, string) (coreAnswer, error) { return coreAnswer{}, errors.New("silent") })
if _, left := other.notPlaced["nats"]; !left {
t.Fatalf("a box sealed to the machine's old key was carried: %+v", other)
}
}
// The core's modules on a machine: on the list, not handed, with their own secrets but one issued outside.
func TestTheCoresModulesAreTheListsWithTheirOwnSecrets(t *testing.T) {
modules := []catalogue.Manifest{
{Module: "nats", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/a"}, "given": {Path: "/b",
IssuedBy: catalogue.IssuedOutside}}},
{Module: "postgres", OwnSecrets: catalogue.OwnSecrets{"superuser": {Path: "/c"}}},
{Module: "mesh-provisioner", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/d"}}},
}
got := coreModulesOf(modules, func(m catalogue.Manifest) bool { return m.Module == "mesh-provisioner" })
if want := map[string][]string{"nats": {"broker"}}; !reflect.DeepEqual(got, want) {
t.Fatalf("got %v, want %v", got, want)
}
}
+3 -1
View File
@@ -1051,7 +1051,9 @@ func appliedFrom(ctx context.Context, inv *inventory.Inventory) inventory.Applie
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, walkSent.Format(time.RFC3339), err)
return inventory.AppliedReport{}, false
}
return r, ok
// And, for each module the send handed to the provisioner, its client's report on that build (novox/hq ADR
// 0362): applied only once both said it, at the later; failed if the client failed.
return inventory.ProvisionedSend(node, send, r, ok, provLookup(ctx))
}
}
+15
View File
@@ -72,6 +72,9 @@ type deskGive struct {
// nil keeps no record (a test that does not look). end closes it with how it ended.
open func(node, module, name, desk string) (int64, error)
end func(id int64, outcome string) error
// vault, when set, is the path of a module whose secrets are filled the new way (vault_secrets.go): the vault
// records the ask, holds its bounds and opens what is typed; open, end and accept are not used.
vault vaultAsker
}
// askedByName is the caller as the prompt names it: the first clause of what the bus said, in the characters
@@ -138,6 +141,9 @@ func (d deskGive) give(node, module, name, desk string) (string, error) {
"bus, where an agent may answer first (novox/hq ADR 0259 §10)", module, name, node, module, name)
}
}
if d.vault != nil {
return d.giveToVault(node, module, name, desk)
}
// The bounds, read and kept before anybody is asked to type (novox/hq ADR 0277): one open prompt per secret,
// few an hour. How the ask ends is recorded whatever happens below.
outcome := "failed"
@@ -297,6 +303,15 @@ func askAtDesk(ctx context.Context, node, module, name, desk string) error {
return announceSecretGiven(ctx, node, module, name, how, askedBy)
},
}
// A module whose secrets are filled the new way is asked for by the vault (ADR 0349 clause 9); every other
// keeps today's path (clause 17).
ours, err := filledTheNewWay(module)
if err != nil {
return fmt.Errorf("nothing was asked: %w", err)
}
if ours {
d.vault = theVault(ctx)
}
words, err := d.give(node, module, name, desk)
if err != nil {
return err
+2 -2
View File
@@ -348,11 +348,11 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
if layer.From == catalogue.MeshWideLayer {
if !meshWide[module] {
meshWide[module] = true
f.Settings = append(f.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Values)})
f.Settings = append(f.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Whole())})
}
continue
}
m.Settings = append(m.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Values)})
m.Settings = append(m.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Whole())})
}
}
m.Declaration = declarationFacts(ctx, open, n.Name, gens, gensErr, scrub)
+37 -1
View File
@@ -235,6 +235,10 @@ type gateFacts struct {
sent map[string]inventory.SentDeclaration
sentBuilds map[string]map[string]string
commits map[string]string
// provisioned answers each machine's provisioner client's newest report of a module (novox/hq ADR 0349 §15):
// a module the gate's send handed to the provisioner is healthy only once it reported that build placed. Nil
// reads as nothing said.
provisioned inventory.ProvLookup
}
// heldReading is the provider a module's findings are held under, as "<module> on <machine>", and, when that
@@ -291,7 +295,7 @@ func (f gateFacts) supersededOn(module, machine string) (string, bool) {
var gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
inv := open.inventory
f := gateFacts{now: time.Now(), reports: map[string]inventory.Reported{}, engines: map[string]string{},
rolledBack: witnessed.all()}
rolledBack: witnessed.all(), provisioned: provLookup(ctx)}
reports, err := inv.LastReports(ctx)
if err != nil {
return f, err
@@ -368,6 +372,12 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
case r.Outcome != inventory.OutcomeApplied:
return healthNotYet, fmt.Sprintf("%s reported %q", machine, r.Outcome)
}
// **And its files placed** (novox/hq ADR 0349 §15): a module the send handed to the provisioner is judged only
// once that machine's client reported the build's templates placed, so a module whose files are not yet written
// is never judged; a failure there is the build's on that machine.
if h, why := judgeProvisioned(module, machine, r, f); h != healthGood {
return h, why
}
// **No new condition about it**: about the machine itself, or naming the module on that machine,
// raised since the judging began. The gate's own are not evidence about the build. A fault that was
// there at the send and reopened since is not new; one that had cleared before the send and came back
@@ -449,6 +459,32 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
return healthGood, ""
}
// judgeProvisioned is whether a module's files are placed on a machine, for a module the gate's send handed to the
// provisioner there: healthy for any other.
func judgeProvisioned(module, machine string, r inventory.Reported, f gateFacts) (health, string) {
sent, kept := f.sent[machine]
digest := sent.Provisioned[module]
if !kept || digest == "" {
return healthGood, ""
}
at := time.Time{}
if r.At != nil {
at = *r.At
}
one := inventory.SentDeclaration{Digest: sent.Digest, At: sent.At, Provisioned: map[string]string{module: digest}}
got, ok := inventory.ProvisionedSend(machine, one, inventory.AppliedReport{At: at,
Outcome: inventory.OutcomeApplied}, true, f.provisioned)
switch {
case !ok:
return healthNotYet, fmt.Sprintf("the provisioner on %s has not reported %s's files placed for the new "+
"build", machine, module)
case got.Outcome != inventory.OutcomeApplied:
return healthBroken, fmt.Sprintf("the provisioner on %s did not place %s's files: %s", machine, module,
got.Outcome)
}
return healthGood, ""
}
// machineWord is what one judging found wrong with a machine itself, apart from its modules: facts is
// the judging's facts without those conditions, on what each names among the modules the send moved,
// and whole what holds the machine back as a whole.
+20 -1
View File
@@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
@@ -60,12 +61,30 @@ type sentRecord struct {
toldGeneration int64
sender string
modules []string
// plans is each module the send handed to the provisioner, its plan, as JSON (design 57 P6); nil for none.
plans []byte
}
// sentRecordOf is a composed send's record: its sender is the caller this process acts for.
func sentRecordOf(ctx context.Context, d sendable) sentRecord {
return sentRecord{sequence: d.Sequence, epoch: d.actingEpoch, generation: d.composedFrom,
r := sentRecord{sequence: d.Sequence, epoch: d.actingEpoch, generation: d.composedFrom,
toldGeneration: d.Generation, sender: senderOf(callerOf(ctx)), modules: d.modules}
if len(d.plans) > 0 {
kept := map[string]keptPlan{}
for module, p := range d.plans {
kept[module] = keptPlan{Plan: p, RequiredBy: d.requiredBy[module]}
}
// A map of plain strings and slices: it marshals.
r.plans, _ = json.Marshal(kept)
}
return r
}
// keptPlan is a handed module's plan as a send keeps it (migration 0100): the plan, and for a module only pulled in
// by another, the module that requires it (issue 765, inventory.SentProvisioningPlans).
type keptPlan struct {
catalogue.Plan
RequiredBy string `json:"required-by,omitempty"`
}
// callerOf is who asked for what this process does: the seat call's caller when ctx belongs to one, the
+6 -3
View File
@@ -8,7 +8,8 @@ import (
)
// A rotation asked through the seat carries why to the command, which records it in the hand-act
// log (novox/hq ADR 0228); why with a provision is refused as passed over, not dropped.
// log (novox/hq ADR 0228); and with a provision too, since the vault asks why of every rotation of a
// credential it holds (issue 767).
func TestARotationThroughTheSeatCarriesWhy(t *testing.T) {
argv, err := argvFor("rotate", map[string]any{"node": "anchor", "module": "letta",
"secret": "server-password", "why": "leaked into logs", "cause": "leaked"})
@@ -19,8 +20,10 @@ func TestARotationThroughTheSeatCarriesWhy(t *testing.T) {
if err != nil || strings.Join(argv, " ") != "secret rotate anchor letta server-password" {
t.Fatalf("without why: %v %v", argv, err)
}
if argv, err := argvFor("rotate", map[string]any{"provision": "postgres-database", "why": "leaked"}); err == nil {
t.Fatalf("why beside a provision was passed over: %v", argv)
argv, err = argvFor("rotate", map[string]any{"provision": "postgres-database", "consumer": "ace", "why": "leaked",
"cause": "leaked"})
if err != nil || strings.Join(argv, " ") != "rotate postgres-database --consumer ace --why leaked --cause leaked" {
t.Fatalf("why beside a provision: %v %v", argv, err)
}
}
+9
View File
@@ -105,6 +105,12 @@ var handActVerbs = []handActVerb{
// (ADR 0257).
{Verb: "mirrors", Decision: "which copies in the store no record names are the mesh's is a person's " +
"word (ADR 0257)"},
// A move of data a release runs once after its delivery (design 57): a person's step of the release, never a
// repair.
{Verb: "move-placement", Decision: "a release's move of data runs once after its delivery, on a person's " +
"word (design 57, issue 736)"},
{Verb: "move-config", Decision: "a release's move of data runs once after its delivery, on a person's " +
"word (design 57, issue 664)"},
{Verb: "bus upgrade", Decision: "the bus is never rolled by the mesh: replacing it is a planned step a " +
"person starts (ADR 0236)"},
{Verb: "upgrade release-backlog", Decision: "after a release plan failed, the next opens only when a " +
@@ -118,6 +124,9 @@ var handActVerbs = []handActVerb{
{Verb: "secret accept", Decision: "a value an outside party issued is given by a person, at their desk"},
{Verb: "secret rotate", Decision: "a value a person judged disclosed is replaced on their word",
DecidedFor: []string{causeLeakedInLogs}},
// A provider's credential the vault holds, rotated there on a person's word (issue 767): the same judgement.
{Verb: "rotate", Decision: "a value a person judged disclosed is replaced on their word",
DecidedFor: []string{causeLeakedInLogs}},
}
// personsDecision is whether an act in the log is a person's decision by design, by the verb that
+60 -5
View File
@@ -43,6 +43,32 @@ const issuedAsk = 20 * time.Second
// issuedProvider is one assignment of a module whose offer says its software issues the provision's credential.
type issuedProvider struct {
Node, Module, Tool string
// Holders are the provision's holders from this assignment's machine, named to the vault, whose tool may answer
// a value for each (issue 749).
Holders []issuedHolder
}
// issuedHolder is one holder of a provision from a provider: a consuming module on one machine (issue 749).
type issuedHolder struct {
Node string `json:"node"`
Module string `json:"module"`
}
func (h issuedHolder) String() string { return h.Module + " on " + h.Node }
// issuedHoldersFrom are the provision's holders from provider, each consuming module on each machine once.
func issuedHoldersFrom(holders []inventory.Holder, provider string) []issuedHolder {
seen := map[issuedHolder]bool{}
var out []issuedHolder
for _, h := range holders {
one := issuedHolder{Node: h.Consumer, Module: h.ConsumerModule}
if h.Provider != provider || seen[one] {
continue
}
seen[one] = true
out = append(out, one)
}
return out
}
// issuedTaker is the line's work, with what it reaches handed in: the store, the bus and the sends.
@@ -53,6 +79,10 @@ type issuedTaker struct {
keep func(provision, provider, value string) (inventory.IssuedKept, error)
record func(act link.HandAct) error
send func(machines []string) error
// ofTheVault says whether a provider module's secrets are filled the new way; for one that is, the value goes
// into the vault (vault, vault_secrets.go) and nothing is kept, sealed or sent here. Nil is none.
ofTheVault vaultSwitch
vault vaultAsker
}
// take asks every provider of the provision for its software's value and keeps each; it answers what it did, in
@@ -90,6 +120,15 @@ func (t issuedTaker) take(provision string) (string, error) {
// takeFrom is one provider: asked, opened, kept, recorded and sent when it changed.
func (t issuedTaker) takeFrom(provision string, p issuedProvider) (string, error) {
if t.ofTheVault != nil {
ours, err := t.ofTheVault(p.Module)
if err != nil {
return "", err
}
if ours {
return t.takeIntoVault(provision, p)
}
}
public, private, err := secrets.Keypair()
if err != nil {
return "", fmt.Errorf("no key could be made to take the value: %w", err)
@@ -174,6 +213,16 @@ func issuedProvidersOf(ctx context.Context, inv *inventory.Inventory, provision
if err != nil {
return nil, true, err
}
out, err := issuedProvidersFrom(shelf, holders, func(node string) ([]string, error) {
return inv.Assigned(ctx, node)
}, provision)
return out, true, err
}
// issuedProvidersFrom are the assignments to ask, from the provision's holders: on every machine providing it, the
// assigned modules whose offer says their software issues it, each with that machine's holders (issue 749).
func issuedProvidersFrom(shelf map[string]catalogue.Manifest, holders []inventory.Holder,
assignedOn func(node string) ([]string, error), provision string) ([]issuedProvider, error) {
nodes := map[string]bool{}
for _, h := range holders {
nodes[h.Provider] = true
@@ -185,13 +234,17 @@ func issuedProvidersOf(ctx context.Context, inv *inventory.Inventory, provision
sort.Strings(names)
var out []issuedProvider
for _, node := range names {
assigned, err := inv.Assigned(ctx, node)
assigned, err := assignedOn(node)
if err != nil {
return nil, true, err
return nil, err
}
out = append(out, issuedOn(shelf, node, assigned, provision)...)
on := issuedOn(shelf, node, assigned, provision)
for i := range on {
on[i].Holders = issuedHoldersFrom(holders, node)
}
out = append(out, on...)
}
return out, true, nil
return out, nil
}
// offersIssued is whether any module offers the provision with a credential its software issues.
@@ -262,7 +315,9 @@ func secretIssued(ctx context.Context, args []string) error {
return err
})
},
send: func(machines []string) error { return sendTo(ctx, open, machines) },
send: func(machines []string) error { return sendTo(ctx, open, machines) },
ofTheVault: filledTheNewWay,
vault: theVault(ctx),
}
words, err := t.take(provision)
if words != "" {
+82 -1
View File
@@ -193,7 +193,7 @@ func TestTheProvidersAskedAreTheOffersThatSaySo(t *testing.T) {
"jellyfin": {Module: "jellyfin", Provides: []catalogue.Offer{{Name: "plex-api"}}},
}
got := issuedOn(shelf, "ace", []string{"jellyfin", "plex", "kometa"}, "plex-api")
if len(got) != 1 || got[0] != (issuedProvider{Node: "ace", Module: "plex", Tool: "plex_issued_credential"}) {
if len(got) != 1 || got[0].Node != "ace" || got[0].Module != "plex" || got[0].Tool != "plex_issued_credential" {
t.Fatalf("the providers asked: %+v", got)
}
if err := issuedRefusesRotation(shelf, "plex-api"); err == nil || !strings.Contains(err.Error(), "secret issued plex-api") {
@@ -216,3 +216,84 @@ func TestAnIssuedValueIsADecisionInTheHandActLog(t *testing.T) {
}
t.Fatal("secret issued is not in the hand-act table")
}
// **A need's credential is the vault's for its holder alone** (issue 757) exactly when its providing module is marked
// moved, so `secret issued` goes to the vault naming each holder, and its software issues the provision's credential.
func TestANeedIsIssuedToItsHolderOnlyWhereTheProviderIsMovedAndItsSoftwareIssuesIt(t *testing.T) {
forge := func(moved bool, issued bool) catalogue.Manifest {
offer := catalogue.Offer{Name: "gitea-read"}
if issued {
offer.Credential = &catalogue.OfferCredential{IssuedBy: catalogue.IssuedBySoftware, Tool: "gitea_agent_token"}
}
return catalogue.Manifest{Module: "gitea", Moved: moved, Provides: []catalogue.Offer{offer}}
}
shared := func(moved bool) catalogue.Manifest {
return catalogue.Manifest{Module: "gitea", Moved: moved, OwnSecrets: catalogue.OwnSecrets{"key": {Path: "/k"}},
Provides: []catalogue.Offer{{Name: "gitea-read", Credential: &catalogue.OfferCredential{Own: "key"}}}}
}
given := func() catalogue.Manifest {
return catalogue.Manifest{Module: "gitea", Moved: true, Provides: []catalogue.Offer{{Name: "gitea-read",
Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedOutside}}}}
}
need := catalogue.Needed{Name: "gitea-read", From: "novox", Module: "gitea", For: "claude-code"}
for why, c := range map[string]struct {
shelf map[string]catalogue.Manifest
need catalogue.Needed
want bool
}{
"moved and issued by its software": {map[string]catalogue.Manifest{"gitea": forge(true, true)}, need, true},
"not moved": {map[string]catalogue.Manifest{"gitea": forge(false, true)}, need, false},
"a login for each consumer": {map[string]catalogue.Manifest{"gitea": forge(true, false)}, need, true},
"a login for each, not moved": {map[string]catalogue.Manifest{"gitea": forge(false, false)}, need, false},
"a credential every holder shares": {map[string]catalogue.Manifest{"gitea": shared(true)}, need, false},
"a credential a person gives": {map[string]catalogue.Manifest{"gitea": given()}, need, false},
"a provider not on the shelf": {map[string]catalogue.Manifest{}, need, false},
"answered by a record": {map[string]catalogue.Manifest{"gitea": forge(true, true)},
catalogue.Needed{Name: "gitea-read", ByRecord: true, Module: "gitea", For: "claude-code"}, false},
} {
if got := issuedToHolder(c.shelf, c.need); got != c.want {
t.Errorf("%s: issued to its holder is %v, want %v", why, got, c.want)
}
}
}
// **The vault is a moved provider for its consumers' credentials** (issue 793): its pair rows are the vault's since
// V3, so a need of `secret` is issued to its holder and answered by a moved provider, whether or not the module is
// marked moved; another module is not.
func TestTheModuleHoldingTheVaultSeatCountsAsMovedForItsConsumers(t *testing.T) {
vault := catalogue.Manifest{Module: "mesh-vault", Claims: []catalogue.Claim{{Name: "mesh-vault"}},
Provides: []catalogue.Offer{{Name: "secret"}}}
other := catalogue.Manifest{Module: "mesh-vault", Provides: []catalogue.Offer{{Name: "secret"}}}
need := catalogue.Needed{Name: "secret", Local: "admin", From: "novox", Module: "mesh-vault", For: "gitea"}
if !issuedToHolder(map[string]catalogue.Manifest{"mesh-vault": vault}, need) ||
!providerMoved(map[string]catalogue.Manifest{"mesh-vault": vault}, need) {
t.Error("the vault's consumer is not answered by a moved provider")
}
if issuedToHolder(map[string]catalogue.Manifest{"mesh-vault": other}, need) ||
providerMoved(map[string]catalogue.Manifest{"mesh-vault": other}, need) {
t.Error("a module not holding the vault seat counts as moved")
}
}
// **A provision answered by the helper makes its provider a moved one for that provision alone** (issue 792): the
// provides entry says `"answered-by": "helper"`, the module need not be marked moved, and no credential is the vault's
// for it. Another provision of the module, or a module without the entry, does not count.
func TestAnOfferAnsweredByTheHelperCountsAsMovedForThatProvisionOnly(t *testing.T) {
proxy := catalogue.Manifest{Module: "route-proxy", Provides: []catalogue.Offer{
{Name: "route", AnsweredBy: catalogue.AnsweredByHelper}, {Name: "other"}}}
plain := catalogue.Manifest{Module: "route-proxy", Provides: []catalogue.Offer{{Name: "route"}}}
shelf := func(m catalogue.Manifest) map[string]catalogue.Manifest {
return map[string]catalogue.Manifest{m.Module: m}
}
route := catalogue.Needed{Name: "route", From: "novox", Module: "route-proxy", For: "gitea"}
other := catalogue.Needed{Name: "other", From: "novox", Module: "route-proxy", For: "gitea"}
if !providerMoved(shelf(proxy), route) {
t.Error("a provision answered by the helper is not answered by a moved provider")
}
if providerMoved(shelf(proxy), other) || providerMoved(shelf(plain), route) {
t.Error("a provision not answered by the helper counts as moved")
}
if issuedToHolder(shelf(proxy), route) {
t.Error("the flag made a credential the vault's")
}
}
+67
View File
@@ -0,0 +1,67 @@
package main
import (
"context"
"fmt"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A covered account is handed to the vault when it is minted (novox/hq ADR 0374, issue 787).
//
// **Covered** are the bus accounts `module issue` mints for a core module (catalogue.CoreCredentialModules) or for a
// module marked moved; no other account is. Until the vault makes bus accounts (ADR 0113 clause 6, issue 774) the
// controller mints such an account and writes its hash to the bus's user list, as before, and then hands the whole
// value at once to the vault's `accept` with origin `issued`: sealed here to the key the vault made for that call, so
// the vault alone opens it (ADR 0349 clause 8). The vault keeps it as the module's own secret on that machine and
// raises `secret-changed`; a re-issue is the same hand with a new value, which replaces the old in place. An account
// not covered is minted and kept as it was and handed to nobody.
// vaultAskerForIssue is the vault as `module issue` asks it. A test replaces it.
var vaultAskerForIssue = func(ctx context.Context) vaultAsker { return theVault(ctx) }
// accountIsCovered says whether the bus accounts minted for m are the vault's to hold (ADR 0374 clause 1).
func accountIsCovered(m catalogue.Manifest) bool {
return catalogue.CarriesVaultSealed(m.Module) || m.Moved
}
// vaultInstalled says whether the vault is assigned to any machine. Until it is, as in genesis (ADR 0113 §15 to §18,
// mesh-host's bootstrap, which issues the bus's and the provisioner's accounts before the vault step), there is no
// vault to hand to: the value is kept in the controller's store as before, and the vault takes it afterwards by the
// move (move-in takes a value of the controller's store with origin made, accepted or issued, as it is). Only a
// vault that is installed and does not answer fails the issue.
func vaultInstalled(ctx context.Context, inv *inventory.Inventory) (bool, error) {
entries, err := inv.Catalogued(ctx)
if err != nil {
return false, err
}
for _, e := range entries {
if e.Manifest.Module == "mesh-vault" && len(e.On) > 0 {
return true, nil
}
}
return false, nil
}
// handIssuedToVault hands the value just minted for module's own secret name on node to the vault, when the module's
// accounts are covered, and does nothing otherwise. Loud when the vault does not take it: the bus's user list
// already holds the new hash, so a vault that kept the old value would serve a password that no longer logs in.
func handIssuedToVault(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node, name, held string) error {
if !accountIsCovered(m) {
return nil
}
if installed, err := vaultInstalled(ctx, inv); err != nil {
return fmt.Errorf("whether the vault is installed cannot be read, so %s of %s was not handed to it: %w", name,
m.Module, err)
} else if !installed {
return nil
}
target := ownSecretArgs(node, m.Module, name)
target["origin"] = "issued"
if _, err := acceptIntoVault(vaultAskerForIssue(ctx), target, held, "module issue"); err != nil {
return fmt.Errorf("the credential is minted and kept here, and the bus's user list holds its hash, but the "+
"vault did not take %s of %s on %s: %w (`module issue` again hands a new one)", name, m.Module, node, err)
}
return nil
}
+157
View File
@@ -0,0 +1,157 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every other test mints without a vault to hand to: a stand-in that takes anything, replaced by the tests below.
func init() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
}
// A covered account is handed to the vault when it is minted (novox/hq ADR 0374, issue 787): the whole value, sealed
// to the key the vault made for the call, with origin issued. A module that is neither core nor moved hands nothing.
func TestModuleIssueHandsACoveredAccountToTheVaultAndOnlyThat(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
theVaultIsInstalled(t, open)
inv := open.inventory
known := broker.Broker{Fingerprint: "ab"}
core := aProvisioner()
core.Module = "nats"
moved := aBot("moved-bot", true)
moved.OwnSecrets["broker"] = catalogue.OwnSecret{Path: "/var/lib/moved-bot/broker"}
plain := aBot("kept-bot", false)
plain.OwnSecrets["broker"] = catalogue.OwnSecret{Path: "/var/lib/kept-bot/broker"}
vault := newStandInVault(t)
vaultAskerForIssue = func(context.Context) vaultAsker { return vault.ask }
t.Cleanup(func() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
})
for _, m := range []catalogue.Manifest{core, moved, plain} {
register(t, open, m)
if _, err := assign(ctx, open, "laptop", m.Module); err != nil {
t.Fatal(err)
}
}
issue := func(m catalogue.Manifest, password string) {
t.Helper()
if err := issueWith(ctx, inv, m, "laptop", "", known, "10.0.0.1:4222", "module."+m.Module+".laptop", password); err != nil {
t.Fatal(err)
}
}
issue(core, "first-core-password")
issue(moved, "first-moved-password")
issue(plain, "kept-password")
if err := issueProvisionerClient(ctx, inv, core, "laptop", known, "10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
said := vault.said()
for _, want := range []string{`"module":"nats"`, `"module":"moved-bot"`, `"name":"broker"`,
`"name":"` + broker.ProvisionerSecret + `"`, `"node":"laptop"`} {
if !strings.Contains(said, want) {
t.Errorf("the vault was told %s, without %s", said, want)
}
}
if n := strings.Count(said, `"origin":"issued"`); n != 3 {
t.Errorf("%d accounts were handed with origin issued, want 3 (the core's two and the moved module's): %s", n, said)
}
if strings.Contains(said, "kept-bot") {
t.Errorf("an account that is neither core nor moved was handed to the vault: %s", said)
}
if strings.Contains(said, "first-core-password") || strings.Contains(said, "first-moved-password") {
t.Errorf("a value crossed the bus in the clear: %s", said)
}
// A re-issue hands the new value, which the vault opens: the whole {url, fingerprint, node, user, password}.
issue(moved, "second-moved-password")
if got := vault.taken["accept"]; !strings.Contains(got, "second-moved-password") ||
!strings.Contains(got, `"user":"module.moved-bot.laptop"`) || !strings.Contains(got, `"node":"laptop"`) ||
!strings.Contains(got, `"url":"nats://10.0.0.1:4222"`) || !strings.Contains(got, `"fingerprint":"ab"`) {
t.Errorf("the vault was handed %q after a re-issue, not the new account whole", got)
}
}
// A vault that does not take the value fails the issue loudly: the bus's user list holds the new hash already.
func TestModuleIssueFailsWhenTheVaultDoesNotTakeACoveredAccount(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
moved := aBot("moved-bot", true)
moved.OwnSecrets["broker"] = catalogue.OwnSecret{Path: "/var/lib/moved-bot/broker"}
theVaultIsInstalled(t, open)
register(t, open, moved)
if _, err := assign(ctx, open, "laptop", moved.Module); err != nil {
t.Fatal(err)
}
vault := newStandInVault(t)
vault.refuse = errors.New("the vault is not there")
vaultAskerForIssue = func(context.Context) vaultAsker { return vault.ask }
t.Cleanup(func() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
})
err := issueWith(ctx, open.inventory, moved, "laptop", "", broker.Broker{Fingerprint: "ab"}, "10.0.0.1:4222",
"module.moved-bot.laptop", "pw")
if err == nil || !strings.Contains(err.Error(), "vault did not take") {
t.Fatalf("a vault that took nothing was reported as %v", err)
}
}
func theVaultIsInstalled(t *testing.T, open *stores) {
t.Helper()
vault := catalogue.Manifest{Module: "mesh-vault", Version: "1"}
register(t, open, vault)
if _, err := assign(t.Context(), open, "laptop", vault.Module); err != nil {
t.Fatal(err)
}
}
// In genesis the bus's and the provisioner's accounts are issued before the vault is installed (ADR 0113 §15 to §18):
// there is no vault to hand to, so nothing is asked and nothing fails, and the value stays in the controller's store
// as before, for the move to carry into the vault. Once the vault is installed, one that does not answer fails.
func TestModuleIssueBeforeTheVaultIsInstalledKeepsTheValueAndHandsNothing(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
known := broker.Broker{Fingerprint: "ab"}
core := aProvisioner()
core.Module = "nats"
register(t, open, core)
if _, err := assign(ctx, open, "laptop", core.Module); err != nil {
t.Fatal(err)
}
vault := newStandInVault(t)
vault.refuse = errors.New("there is no vault yet")
vaultAskerForIssue = func(context.Context) vaultAsker { return vault.ask }
t.Cleanup(func() {
vaultAskerForIssue = func(context.Context) vaultAsker { return newStandInVault(nil).ask }
})
if err := issueWith(ctx, inv, core, "laptop", "", known, "10.0.0.1:4222", "module.nats.laptop", "pw"); err != nil {
t.Fatalf("genesis's issue of the bus's account failed with no vault installed: %v", err)
}
if err := issueProvisionerClient(ctx, inv, core, "laptop", known, "10.0.0.1:4222"); err != nil {
t.Fatalf("genesis's issue of the client's account failed with no vault installed: %v", err)
}
if said := vault.said(); said != "" {
t.Errorf("the vault was asked before it was installed: %s", said)
}
if _, held, err := inv.ModuleSecretIfIssued(ctx, "laptop", "nats", "broker"); err != nil || !held {
t.Errorf("the value is not kept in the controller's store (held %v, %v)", held, err)
}
theVaultIsInstalled(t, open)
if err := issueWith(ctx, inv, core, "laptop", "", known, "10.0.0.1:4222", "module.nats.laptop", "pw2"); err == nil {
t.Error("a vault that is installed and does not answer did not fail the issue")
}
}
+56 -2
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"slices"
"strings"
"testing"
"time"
@@ -30,13 +31,13 @@ func TestAWalkEndsOnItsMachinesReports(t *testing.T) {
{Node: "novox", Outcome: inventory.OutcomeFailed, At: &at, Declared: "d-novox"},
{Node: "shanks", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-before"},
}
waiting, failed, silent := walkReports(p, reports, now)
waiting, failed, silent := walkReports(p, reports, now, nil)
if !slices.Equal(waiting, []string{"shanks"}) || !slices.Equal(failed, []string{"novox"}) ||
!slices.Equal(silent, []string{"laptop"}) {
t.Fatalf("waiting %v, failed %v, silent %v", waiting, failed, silent)
}
reports[2].Declared = "d-shanks"
if waiting, _, _ := walkReports(p, reports, now); len(waiting) != 0 {
if waiting, _, _ := walkReports(p, reports, now, nil); len(waiting) != 0 {
t.Fatalf("the walk still waits once every machine reported: %v", waiting)
}
}
@@ -186,3 +187,56 @@ func TestAMachineSilentPastItsBoundIsLeftAndSaidAtOnce(t *testing.T) {
t.Fatalf("the silence is not said: %+v", obs)
}
}
// Issue 840 (novox/hq ADR 0362 §1-5): a walk's end counts a machine's send of a provisioned module applied only once
// the engine and the machine's provisioner client both said so.
func TestAWalkEndWaitsForTheProvisioner(t *testing.T) {
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
at := now.Add(-10 * time.Second)
plan := func(sentAt time.Time) inventory.Plan {
return inventory.Plan{Modules: map[string]*inventory.PlanModule{"app": {Rest: map[string]inventory.SentDeclaration{
"ace": {Digest: "d-ace", At: sentAt, Provisioned: map[string]string{"app": "t1"}},
"g14": {Digest: "d-g14", At: sentAt, Provisioned: map[string]string{"app": "t1"}},
}}}}
}
reports := []inventory.Reported{
{Node: "ace", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-ace"},
{Node: "g14", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-g14"},
}
says := map[string]inventory.ProvReport{}
prov := func(node, module string) (inventory.ProvReport, bool) { r, ok := says[node+"/"+module]; return r, ok }
applied := inventory.ProvReport{Status: inventory.ProvApplied, Templates: "t1", At: at}
// (a) the engine applied, the client has not said: the end waits on both.
waiting, failed, silent := walkReports(plan(now.Add(-time.Minute)), reports, now, prov)
if !slices.Equal(waiting, []string{"ace", "g14"}) || len(failed)+len(silent) != 0 {
t.Fatalf("a: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// ace's client applied, g14's failed on that build (b): ace done, g14 a failed apply naming the reason.
says["ace/app"] = applied
says["g14/app"] = inventory.ProvReport{Status: inventory.ProvFailed, Templates: "t1", Reason: "no value", At: at}
waiting, failed, silent = walkReports(plan(now.Add(-time.Minute)), reports, now, prov)
if len(waiting) != 0 || len(silent) != 0 || len(failed) != 1 || !strings.HasPrefix(failed[0], "g14 (") ||
!strings.Contains(failed[0], "no value") {
t.Fatalf("b: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// (c) no client report past ApplySilentAfter: named silent, never applied.
delete(says, "g14/app")
waiting, failed, silent = walkReports(plan(now.Add(-inventory.ApplySilentAfter-time.Minute)), reports, now, prov)
if len(waiting) != 0 || len(failed) != 0 || !slices.Equal(silent, []string{"g14"}) {
t.Fatalf("c: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// both said it: nothing waits.
says["g14/app"] = applied
if waiting, failed, silent = walkReports(plan(now.Add(-time.Minute)), reports, now, prov); len(waiting)+len(failed)+len(silent) != 0 {
t.Fatalf("both said: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// (d) a send handing nothing to the provisioner is the engine's report alone, as before.
bare := plan(now.Add(-time.Minute))
bare.Modules["app"].Rest = map[string]inventory.SentDeclaration{"ace": {Digest: "d-ace", At: now.Add(-time.Minute)}}
if waiting, failed, silent = walkReports(bare, reports, now, func(string, string) (inventory.ProvReport, bool) {
return inventory.ProvReport{}, false
}); len(waiting)+len(failed)+len(silent) != 0 {
t.Fatalf("d: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
}
+9 -1
View File
@@ -121,6 +121,10 @@ func run() error {
return imagesCommand(ctx, args[1:])
case "mirrors":
return mirrorsCommand(ctx, args[1:])
case "move-placement":
return movePlacementCommand(ctx, args[1:])
case "move-config":
return moveConfigCommand(ctx, args[1:])
case "plans":
return plansCommand(ctx, args[1:])
case "delivery":
@@ -275,6 +279,10 @@ func usage() {
artifacts [--json] [--repository <r>] [--collected] every artifact the mesh made: kept and why, eligible, collected (ADR 0251)
collect [--json] [--most <n>] what the store's sweep would let go of, now; nothing is done
collect --confirm --why <text> [--most <n>] run the sweep now: hold every kept archive, let go of the eligible
move-placement [--json] which settings layers still hold placement keys (endpoints, expose, reach, ports); nothing is moved
move-placement --confirm --why <text> move them beside the assignment, once, after the delivery (issue 736)
move-config [--json] every config value the settings layers hold, and where in mesh-config it goes; nothing is set
move-config --confirm --why <text> set them in mesh-config, read them back, and read config from there (issue 664)
images <node> [--json] every container image the machine's declaration names, now and as last sent
seats [--json] every seat this mesh defines, what it delivers, and who holds it
seat rename <from> <to> rename a seat; its former name still resolves (ADR 0122)
@@ -331,7 +339,7 @@ func usage() {
licence add|list|use|key model access, under the name a person calls it
licence manager <name> <node> the node that holds a refreshable licence's refresh token
licence refresh <name> mint a new access token and seal it to every holder
rotate <provision> [--consumer <n>] [--module <m>] a new credential for every holder, both ends at once
rotate <provision> [--consumer <n>] [--module <m>] [--why <text>] a new credential for every holder, both ends at once (--why where the vault holds it)
ask <module> <tool> [json] call one of a module's tools over the broker, and print its answer
pin <node> <provision> <from-node> <module>
which provider this one gets a provision from: the module, and its node
+30 -3
View File
@@ -550,8 +550,17 @@ func shelfWithChange(base map[string]catalogue.Manifest, sources map[string]inve
out[k] = m
}
var failures []string
// The list of predefined keys at the tree's root (novox/hq ADR 0348 §7): what the build of this tree judges each
// manifest against and registration publishes, so what the gate judges each against too. A list that is
// wrong is refused by name, and every manifest is then judged against none.
predefined, err := catalogue.ReadPredefined(tree)
if err != nil {
failures = append(failures, fmt.Sprintf("%s: the controller judging this (%s) refuses it — %s",
catalogue.PredefinedFile, version, oneLine(err.Error())))
predefined = nil
}
found := map[string]string{} // module → its directory in the tree
err := filepath.WalkDir(tree, func(path string, d fs.DirEntry, err error) error {
err = filepath.WalkDir(tree, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
@@ -575,7 +584,12 @@ func shelfWithChange(base map[string]catalogue.Manifest, sources map[string]inve
if err != nil {
return err
}
m, err := catalogue.ParseManifest(raw)
m, err := catalogue.ParseManifestWith(raw, predefined)
if err == nil {
// Each template's text from the tree beside the manifest, as the build reads it back from the archive it
// packs (novox/hq ADR 0349 §2): so a change to a template alone is a change to its module.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
if err != nil {
if inScope(dir) {
failures = append(failures, fmt.Sprintf("%s: the controller judging this (%s) refuses it — %s",
@@ -605,7 +619,7 @@ func shelfWithChange(base map[string]catalogue.Manifest, sources map[string]inve
// As registration reads a built manifest: strictly, again (takeIn).
var raw []byte
if raw, err = json.Marshal(resolved); err == nil {
resolved, err = catalogue.ParseManifest(raw)
resolved, err = catalogue.ParseManifestWith(raw, predefined)
}
}
if err == nil {
@@ -1083,6 +1097,19 @@ func raiseFromFacts(ctx context.Context, open *stores, f snapshot.Facts, shelf m
continue
}
}
// And the provisioner client's account, the same way (novox/hq ADR 0349 clause 11, design 57 P3).
if a.Provider == "" && a.Name == broker.ProvisionerSecret {
if pm, held := shelf[a.Module]; !held || !runsTheProvisionerClient(pm) {
continue
}
if _, err := inv.MintBusPassword(ctx, inventory.BusUser{Kind: inventory.BusProvisioner, Node: m.Name,
Module: a.Module, Username: broker.Principal{Kind: broker.KindProvisioner, Node: m.Name,
Module: a.Module}.Username()}); err != nil {
notes = append(notes, fmt.Sprintf("%s's provisioner client account for %s is not kept: %s",
m.Described(), a.Module, oneLine(err.Error())))
continue
}
}
var err error
if a.Provider == "" {
err = inv.AcceptSecretForModule(ctx, m.Name, a.Module, a.Name, standIn)
+15 -1
View File
@@ -33,7 +33,21 @@ import (
func aMesh(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t) // raises the store, migrates it, and points the environment at it
licences.ForTest(t) // planning reaches this one too, by name and never by connection
return aMeshOnTheStore(t)
}
// aMeshBefore is aMesh on a store migrated only below migration `number`, with the function that runs the
// rest: for a test of what a migration that moves data does to what the mesh composes (novox/hq issue 651).
func aMeshBefore(t *testing.T, number int) (*stores, func()) {
t.Helper()
_, migrate := inventory.ForTestBefore(t, number)
return aMeshOnTheStore(t), migrate
}
// aMeshOnTheStore is the mesh aMesh makes, on the store the environment points at.
func aMeshOnTheStore(t *testing.T) *stores {
t.Helper()
licences.ForTest(t) // planning reaches this one too, by name and never by connection
open, err := openStores(t.Context())
if err != nil {
+74 -1
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"net"
"os"
"path/filepath"
"sort"
"strings"
"time"
@@ -127,6 +128,10 @@ func moduleCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// A manifest handed over by hand takes each template's text from the tree beside it (novox/hq ADR 0349 §2).
if m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(positionals[0])); err != nil {
return err
}
from, err := whereItComesFrom(*repo, *ref, *commit, *path, *self)
if err != nil {
return err
@@ -441,6 +446,10 @@ func settingsCommand(ctx context.Context, args []string) error {
if len(positionals) == 2 {
values = positionals[1]
}
// Once its config values are mesh-config's, they are proposed there (issue 664).
if forwarded, err := forwardPropose(ctx, inv, positionals[0], *node, values, *clear); forwarded || err != nil {
return err
}
return proposeCommand(ctx, positionals[0], *node, values, *clear, *replace)
case "proposals":
if len(positionals) > 1 || *node != "" || *clear || *replace || *history {
@@ -461,6 +470,15 @@ func settingsCommand(ctx context.Context, args []string) error {
where = *node
}
// Once the move of config has run, a module in the config form is read and set through mesh-config (issue 664).
if (args[0] == "set" && len(positionals) == 2) || ((args[0] == "show" && !*history) || args[0] == "clear") &&
len(positionals) == 1 {
if forwarded, err := forwardSettings(ctx, inv, args[0], positionals[0], *node, positionals, *replace); forwarded ||
err != nil {
return err
}
}
switch args[0] {
case "set":
if len(positionals) != 2 {
@@ -537,6 +555,10 @@ func settingsCommand(ctx context.Context, args []string) error {
if p.SetBy != "" {
by = "; " + p.SetBy
}
if p.Why != "" {
// A layer whose placement moved out of it (migration 0097): still the layer, as it stands.
by += "; " + p.Why
}
fmt.Printf("%s on %s, until %s (%s%s):\n %s\n", positionals[0], where,
p.ReplacedAt.Local().Format("2006-01-02 15:04:05"), p.ReplacedBy, by, shown)
}
@@ -940,7 +962,55 @@ func issueOnTheNewBus(ctx context.Context, inv *inventory.Inventory, m catalogue
if err != nil {
return err
}
return issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password)
if err := issueWith(ctx, inv, m, node, busAddress, known, reachable, user, password); err != nil {
return err
}
if !runsTheProvisionerClient(m) {
return nil
}
return issueProvisionerClient(ctx, inv, m, node, known, reachable)
}
// runsTheProvisionerClient says a module runs its machine's provisioner client, which reaches the bus on an
// account of its own (novox/hq ADR 0349 clause 11, design 57 P3): it claims the seat and declares the own secret
// that account is delivered in. The same test the bus's user list makes (inventory's declaredFor).
func runsTheProvisionerClient(m catalogue.Manifest) bool {
_, reads := m.OwnSecrets[broker.ProvisionerSecret]
return reads && m.ClaimsSeat(broker.NodeProvisionerSeat)
}
// issueProvisionerClient gives a machine's provisioner client its account: minted under the client's own name,
// never the module's or the tool runner's, and sealed to the machine as the module's own secret
// broker.ProvisionerSecret, which the client's service reads (novox/hq ADR 0349 clause 11, design 57 P3). No
// claims and no consumer: the client holds no seat on this account and is delivered nothing but its answers.
func issueProvisionerClient(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string,
known broker.Broker, reachable string) error {
user := broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: m.Module}.Username()
password, err := inv.MintBusPassword(ctx, inventory.BusUser{
Username: user, Kind: inventory.BusProvisioner, Node: node, Module: m.Module,
})
if err != nil {
return err
}
held, err := json.Marshal(struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
Node string `json:"node"`
User string `json:"user"`
Password string `json:"password"`
}{URL: "nats://" + reachable, Fingerprint: known.Fingerprint, Node: node, User: user, Password: password})
if err != nil {
return err
}
if err := inv.AcceptSecretForModule(ctx, node, m.Module, broker.ProvisionerSecret, string(held)); err != nil {
return err
}
if err := handIssuedToVault(ctx, inv, m, node, broker.ProvisionerSecret, string(held)); err != nil {
return err
}
fmt.Printf("bus user %s minted for %s's provisioner client, scoped to %s's keys, its reports and the vault\n",
user, m.Module, node)
return nil
}
// busKindOf is what a module's bus user is recorded as: the node's tool runtime where the module is
@@ -985,6 +1055,9 @@ func issueWith(ctx context.Context, inv *inventory.Inventory, m catalogue.Manife
if err := inv.AcceptSecretForModule(ctx, node, m.Module, "broker", string(held)); err != nil {
return err
}
if err := handIssuedToVault(ctx, inv, m, node, "broker", string(held)); err != nil {
return err
}
// And how it hears what it consumes. Derived from its declaration, and only when it declared
// something: a module that consumes nothing needs no consumer, and creating one would be a
+99
View File
@@ -0,0 +1,99 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
)
// movePlacementAnswer is what the move of placement says it did, or would do (issue 736).
type movePlacementAnswer struct {
// DryRun is true when nothing was moved: asked without --confirm.
DryRun bool `json:"dry-run"`
// Moved is each layer that held a placement key among its settings, and which keys.
Moved []inventory.MovedLayer `json:"moved"`
}
// movePlacementCommand is `move-placement`: the move of placement out of the settings layers (novox/hq ADR 0348
// clause 11, issue 736), one of the moves of data a release runs once after its delivery (design 57).
func movePlacementCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("move-placement", flag.ContinueOnError)
asJSON := set.Bool("json", false, "answer as JSON")
confirm := set.Bool("confirm", false, "move the keys, rather than only say which layers hold them")
f := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 0 {
return errors.New("move-placement [--json] [--confirm --why <text>]")
}
if !*confirm && f.given() {
return errors.New("move-placement takes --why only with --confirm: without it nothing is moved. Nothing was done")
}
if *confirm {
// A move a person asks for says why, before anything is asked of the store.
if err := f.require("move-placement"); err != nil {
return err
}
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if *confirm {
f.record(ctx, "move-placement", nil)
}
answer, err := runMovePlacement(ctx, open.inventory, *confirm)
if err != nil {
return err
}
if *asJSON {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
return encoder.Encode(answer)
}
fmt.Print(describeMovePlacement(answer))
return nil
}
// runMovePlacement is the move against a store: a dry run says what it would move.
func runMovePlacement(ctx context.Context, inv *inventory.Inventory, apply bool) (movePlacementAnswer, error) {
moved, err := inv.MovePlacement(ctx, apply)
if err != nil {
return movePlacementAnswer{}, err
}
return movePlacementAnswer{DryRun: !apply, Moved: moved}, nil
}
// describeMovePlacement says the answer as a person reads it.
func describeMovePlacement(a movePlacementAnswer) string {
if len(a.Moved) == 0 {
return "no settings layer holds a placement key: nothing to move\n"
}
var b strings.Builder
verb := "moved"
if a.DryRun {
verb = "would move"
}
for _, m := range a.Moved {
where := m.Node
if where == "" {
where = "the whole mesh"
}
fmt.Fprintf(&b, "%s on %s: %s %s\n", m.Module, where, verb, strings.Join(m.Keys, ", "))
}
if a.DryRun {
fmt.Fprintf(&b, "%d layer(s); nothing was moved — with --confirm --why <text> they are\n", len(a.Moved))
} else {
fmt.Fprintf(&b, "%d layer(s) moved beside their assignments, each kept whole in its history first\n", len(a.Moved))
}
return b.String()
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// **Placement leaves the settings layers and no machine is told anything new** (novox/hq ADR 0348 clause
// 11, issue 651). A mesh is seeded as the store holds it before migration 0097 — `places`, `accesses` and a
// kept network among the settings, at the mesh's level and a machine's, one layer of placement only — then
// migrated, and every machine's declaration is composed. What each machine is told of the seeded modules is
// byte for byte what the controller before this change composed from the same seed: the golden files were
// written by the integration branch's controller at bfc79c6 running this same test (MESH_WRITE_GOLDEN=1),
// where there was no migration 0097 to run.
func TestADeclarationIsTheSameAfterPlacementMoves(t *testing.T) {
open, migrate := aMeshBefore(t, 97)
ctx := t.Context()
inv := open.inventory
image := "registry.invalid:5000/library@sha256:" + strings.Repeat("a", 64)
register(t, open, catalogue.Manifest{Module: "library", Version: "1",
Accesses: []catalogue.Access{{ID: "series"}, {ID: "downloads", Mode: catalogue.AccessRead}},
Resources: []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "server", "type": "container", "image": image,
"volumes": []any{"${dir:config}:/config", "${access:series}:/series", "${access:downloads}:/downloads:ro"}},
// Today's form: a mergeable file naming no keys takes every key of the layer (ADR 0326 §3).
{"id": "runtime-config", "type": "file", "path": "${dir:config}/runtime.json", "mode": "0640",
"content": `{"log":"info"}`, "merge": "json"},
}})
register(t, open, catalogue.Manifest{Module: "found", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "image": image}}})
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
for _, l := range []struct{ node, module, values string }{
{"", "library", `{"accesses": {"downloads": "/storage/downloads"}, "log": "warn"}`},
{"anchor", "library", `{"places": {"config": {"path": "/var/lib/library/config", "owner": "1001:2000"}},
"accesses": {"series": "/storage/media/series"}, "workers": 4}`},
{"laptop", "library", `{"accesses": {"series": "/home/media/series"}}`},
{"anchor", "found", `{"networks": {"server": ["predecessor_default"]}}`},
} {
if l.node == "" {
inv.ExecForTest(t, `insert into settings (node, module, values) values (null, $1, $2)`, l.module, l.values)
continue
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.node, l.module, l.values)
}
migrate()
// Assigned once the store is migrated: an assignment reads the layers, as this controller holds them.
for _, a := range [][]string{{"anchor", "library", "found"}, {"laptop", "library"}} {
if _, err := assign(ctx, open, a[0], a[1:]...); err != nil {
t.Fatalf("assign %v: %v", a, err)
}
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatal(err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatal(err)
}
if len(composed.LeftOut) > 0 {
t.Fatalf("%s leaves out %v", node, composed.LeftOut)
}
var told []map[string]any
for _, r := range composed.Resources {
if owner := composed.Owner[r["id"].(string)]; owner == "library" || owner == "found" {
told = append(told, r)
}
}
sort.Slice(told, func(i, j int) bool { return told[i]["id"].(string) < told[j]["id"].(string) })
got, err := json.MarshalIndent(told, "", " ")
if err != nil {
t.Fatal(err)
}
golden := filepath.Join("testdata", "placement-moved", node+".json")
if os.Getenv("MESH_WRITE_GOLDEN") != "" {
if err := os.MkdirAll(filepath.Dir(golden), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(golden, append(got, '\n'), 0o644); err != nil {
t.Fatal(err)
}
continue
}
want, err := os.ReadFile(golden)
if err != nil {
t.Fatal(err)
}
if string(want) != string(got)+"\n" {
t.Errorf("%s is told something else after placement moved:\n%s\nwas:\n%s", node, got, want)
}
}
}
@@ -0,0 +1,243 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// wordedMesh is every live layer that says `endpoints`, `expose`, `reach` or `ports` (issue 736) — 36, read with
// the settings verb on 2026-10-14, ace's as anchor's and novox's as laptop's, one stand-in for an address — split as
// the store holds them once migration 0097 has run: the four words still among the settings, `places` and
// `accesses` already beside them. With each module shaped as its definition in the catalogue (mesh-catalog's, and
// the media catalogue's) is where the four words reach: its listens, what its containers publish and on which
// network, its route, its directories and accesses, and its mergeable files — every key of the layer, or the keys
// they name. The shapes are the modules'; the files' text and the images are the test's.
type wordedMesh struct {
Modules []json.RawMessage `json:"modules"`
Layers []struct {
Node string `json:"node"`
Module string `json:"module"`
Settings map[string]any `json:"settings"`
Placement map[string]any `json:"placement"`
} `json:"layers"`
}
func theWordedMesh(t *testing.T) wordedMesh {
t.Helper()
raw, err := os.ReadFile(filepath.Join("testdata", "placement-words-moved", "mesh.json"))
if err != nil {
t.Fatal(err)
}
var w wordedMesh
if err := json.Unmarshal(raw, &w); err != nil {
t.Fatal(err)
}
return w
}
// aMeshWithTheWordsInItsLayers is aMesh with every module of the worded mesh registered, a route provider on both
// machines, and its layers held as the store holds them before the four words move.
func aMeshWithTheWordsInItsLayers(t *testing.T) (*stores, wordedMesh) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
w := theWordedMesh(t)
image := "registry.invalid:5000/library@sha256:" + strings.Repeat("a", 64)
register(t, open, catalogue.Manifest{Module: "proxy", Version: "1", Provides: catalogue.FromAnywhere("route"),
Receives: map[string]string{"route": "/var/lib/proxy/routes.json"}})
for _, text := range w.Modules {
m, err := catalogue.ParseManifest([]byte(strings.ReplaceAll(string(text), "IMAGE", image)))
if err != nil {
t.Fatal(err)
}
register(t, open, m)
}
on := map[string][]string{"anchor": {"proxy"}, "laptop": {"proxy"}}
for _, l := range w.Layers {
settings, err := json.Marshal(l.Settings)
if err != nil {
t.Fatal(err)
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.Node, l.Module, string(settings))
if l.Placement != nil {
placement, err := json.Marshal(l.Placement)
if err != nil {
t.Fatal(err)
}
inv.ExecForTest(t, `insert into placement (node, module, values) select id, $2, $3 from node where name = $1`,
l.Node, l.Module, string(placement))
}
on[l.Node] = append(on[l.Node], l.Module)
}
for _, node := range []string{"anchor", "laptop"} {
if err := inv.SetPublicDomain(ctx, node, node+".example.test"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, node, on[node]...); err != nil {
t.Fatalf("assign %s %v: %v", node, on[node], err)
}
}
return open, w
}
// toldOfTheWords is what a machine is told of the modules whose layers say the four words, and of the route
// provider that serves their names: those modules' resources and the filter's rules, as JSON a person can diff.
func toldOfTheWords(t *testing.T, open *stores, node string) string {
t.Helper()
ctx := t.Context()
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatal(err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatal(err)
}
if len(composed.LeftOut) > 0 {
t.Fatalf("%s leaves out %v", node, composed.LeftOut)
}
rules, err := plan.Rules(with)
if err != nil {
t.Fatal(err)
}
ours := map[string]bool{"proxy": true}
for _, l := range theWordedMesh(t).Layers {
ours[l.Module] = true
}
var told []map[string]any
for _, r := range composed.Resources {
if ours[composed.Owner[r["id"].(string)]] {
told = append(told, r)
}
}
sort.Slice(told, func(i, j int) bool { return told[i]["id"].(string) < told[j]["id"].(string) })
got, err := json.MarshalIndent(map[string]any{"resources": told, "rules": rules}, "", " ")
if err != nil {
t.Fatal(err)
}
return string(got) + "\n"
}
// sameAsTheGolden holds what a machine is told to the golden file the controller before issue 736 wrote from the
// same seed (MESH_WRITE_GOLDEN=1 writes it instead).
func sameAsTheGolden(t *testing.T, open *stores, when string) {
t.Helper()
for _, node := range []string{"anchor", "laptop"} {
got := toldOfTheWords(t, open, node)
golden := filepath.Join("testdata", "placement-words-moved", node+".json")
if os.Getenv("MESH_WRITE_GOLDEN") != "" {
if err := os.MkdirAll(filepath.Dir(golden), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(golden, []byte(got), 0o644); err != nil {
t.Fatal(err)
}
continue
}
want, err := os.ReadFile(golden)
if err != nil {
t.Fatal(err)
}
if string(want) != got {
t.Errorf("%s: %s is told something else:\n%s\nwas:\n%s", when, node, got, want)
}
}
}
// **The four words leave the settings layers and no machine is told anything new** (novox/hq ADR 0348 clause 11,
// issue 736). Every live layer that says `endpoints`, `expose`, `reach` or `ports` is seeded as the store holds it
// after delivery, on a module shaped as its definition is, and every machine's declaration is composed. What each
// is told is byte for byte what the controller before this change composed from the same seed.
func TestADeclarationIsTheSameWhileTheWordsAreStillInTheLayers(t *testing.T) {
open, _ := aMeshWithTheWordsInItsLayers(t)
sameAsTheGolden(t, open, "before the move")
}
// **And after the move, still nothing new** (issue 736). The move of placement, asked without confirming, says
// every layer that holds one of the four words and moves nothing; confirmed, it moves them all, no settings layer
// holds a placement key after, and each machine is told byte for byte what it was told before the move — and what
// the controller before this change told it. Asked again, it moves nothing.
func TestADeclarationIsTheSameAfterTheMoveOfPlacement(t *testing.T) {
open, w := aMeshWithTheWordsInItsLayers(t)
ctx := t.Context()
inv := open.inventory
asked, err := runMovePlacement(ctx, inv, false)
if err != nil || !asked.DryRun || len(asked.Moved) != len(w.Layers) {
t.Fatalf("the move, not confirmed, says %+v (%v); %d layers hold the words", asked, err, len(w.Layers))
}
if held := inv.QueryForTest(t, `select module from settings where values ?| $1`, catalogue.PlacementKeys()); len(held) != len(w.Layers) {
t.Fatalf("the move, not confirmed, moved something: %d layers still hold the words", len(held))
}
sameAsTheGolden(t, open, "after the move was asked without confirming")
moved, err := runMovePlacement(ctx, inv, true)
if err != nil || moved.DryRun || !reflect.DeepEqual(moved.Moved, asked.Moved) {
t.Fatalf("the move says %+v (%v), it said it would move %+v", moved, err, asked.Moved)
}
if held := inv.QueryForTest(t, `select module from settings where values ?| $1`, catalogue.PlacementKeys()); len(held) != 0 {
t.Fatalf("a settings layer still holds a placement key: %v", held)
}
sameAsTheGolden(t, open, "after the move")
if again, err := runMovePlacement(ctx, inv, true); err != nil || len(again.Moved) != 0 {
t.Fatalf("asked again, the move moved %+v (%v)", again, err)
}
if said := describeMovePlacement(movePlacementAnswer{}); !strings.Contains(said, "nothing to move") {
t.Fatalf("an empty move says %q", said)
}
}
// The verb's command line: a dry run unless confirmed, and a confirmed move says why, recorded as a person's step
// of the release, never a repair (issue 736).
func TestTheMoveOfPlacementIsADryRunUnlessConfirmed(t *testing.T) {
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "move-placement --json"},
{map[string]any{"confirm": "false"}, "move-placement --json"},
{map[string]any{"confirm": "true", "why": "release 57 is delivered"}, "move-placement --json --confirm --why release 57 is delivered"},
} {
argv, err := argvFor("move-placement", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%v: %q %v, want %q", c.args, argv, err, c.want)
}
}
for _, args := range []map[string]any{
{"confirm": "true"}, // a move without a why
{"why": "tidy"}, // a why for a dry run, recorded nowhere
{"confirm": "yes", "why": "x"}, // a switch is true or false
{"node": "anchor"},
} {
if argv, err := argvFor("move-placement", args); err == nil {
t.Errorf("%v was composed as %q", args, argv)
}
}
if repairingCommand([]string{"move-placement", "--json", "--confirm", "--why", "x"}) != "move-placement" {
t.Error("a confirmed move through the generic verb would go unrecorded")
}
if repairingCommand([]string{"move-placement", "--json"}) != "" {
t.Error("a dry run was taken for an act by hand")
}
if !personsDecision(link.HandAct{Verb: "move-placement"}) {
t.Error("a move of the release would count toward a healer the mesh lacks")
}
}
+9
View File
@@ -186,6 +186,15 @@ var plainWordings = map[string]func(conditions.Observation) words{
}),
// The controller or the node-engine kept at the build a machine was last sent, over an older record (novox/hq
// issue 201).
kindVaultSilent: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "The vault did not give " + m + " the core's credentials",
Needs: "check the vault answers (vault_status); the next send asks it again.",
Explanation: fmt.Sprintf("The bus's and the provisioner's credentials reach %s sealed by the vault, and "+
"the vault did not answer the last send for them. %s keeps the files it has; a core module not "+
"there yet is not placed. The mesh never seals them itself instead.", m, m),
Resolved: "Resolved: the vault gave " + m + " the core's credentials"}
}),
kindCoreDowngrade: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
component := idPart(o, 0)
+128 -4
View File
@@ -150,6 +150,10 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
// password a provider is told to create is the one its consumer was given — and sealed to
// this node before it was ever written down, so nothing between here and there can read it.
for i, n := range resolved.Needs {
// **Issued for this holder alone and kept by the vault** (issue 757): where this consumer is handed to the
// provisioner, its machine's client asks the vault for it, and the declaration carries no value.
resolved.Needs[i].IssuedToHolder = issuedToHolder(shelf, n)
resolved.Needs[i].ProviderMoved = providerMoved(shelf, n)
if n.ByRecord {
// Answered by something the mesh holds, so there is no pair-wise secret between two
// machines. Its key was supplied by a person and sealed to this node then; the mesh
@@ -476,9 +480,23 @@ func declarationWith(ctx context.Context, open *stores, node string,
out := sendable{Resources: composed.Resources, Adoption: adoption,
Received: composed.Received, Mesh: with.Mesh, BusUsers: with.BusUsers,
LeftOut: sortedKeysOf(composed.LeftOut), leftOutWhy: composed.LeftOut, withheld: with.Withheld,
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced,
modules: namedModules(plan, composed.LeftOut),
unbound: with.Unbound, foreseen: composed.Foreseen, unplaced: composed.Unplaced, handedOnly: composed.HandedOnly,
modules: namedModules(plan, composed.LeftOut),
Filled: composed.Filled, plans: composed.Plans,
localNetwork: catalogue.LocalNetworkPorts(rules, with.LocalNetworks)}
if len(composed.Filled) > 0 {
out.Provisioner = with.Provisioner
}
// A handed module only pulled in by another keeps what requires it, for its value of its own in the assignments
// mirror (issue 765).
for module := range composed.Plans {
if by := plan.RequiredBy(module); by != "" {
if out.requiredBy == nil {
out.requiredBy = map[string]string{}
}
out.requiredBy[module] = by
}
}
// And which build of each module it carries, for the send to record (novox/hq issue 259, ADR
// 0221). Read only on the send path: a question about what would be sent records nothing.
if choosing == Allocating {
@@ -591,6 +609,10 @@ func reportLeftOut(node string, declared sendable) {
for _, u := range declared.unplaced {
fmt.Printf("%s: %s\n", node, u)
}
// And each file only the provisioner's client fills, left out because the module is not handed here.
for _, u := range declared.handedOnly {
fmt.Printf("%s: %s\n", node, u)
}
}
// busCredentialIssued refuses an own secret called `broker` whose bus account nobody issued.
@@ -601,10 +623,17 @@ func reportLeftOut(node string, declared sendable) {
// cannot read and report the machine applied. Refused by name, with the verb — on the send, and on
// a question asked ahead of it (Foreseeing), so that one is never told the push will make it.
func busCredentialIssued(ctx context.Context, inv *inventory.Inventory, node, module, name string) error {
if name != "broker" {
var user string
switch name {
case "broker":
user = broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
case broker.ProvisionerSecret:
// And the machine's provisioner client's account, issued with its module (novox/hq ADR 0349 clause 11,
// design 57 P3): the same refusal, for the same reason.
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
default:
return nil
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return err
} else if !minted {
@@ -728,7 +757,18 @@ func renderingFor(ctx context.Context, open *stores, node string,
// per node, so a module running on three machines has three.
needed := map[string]map[string]string{}
foreseen := map[string]map[string]bool{}
// **A module handed to the provisioner here has its secrets from the vault** (novox/hq ADR 0349 §7, §11,
// design 57 V4): its machine's client asks for each, so the controller makes, reads and seals none of them for
// this machine. Every other module's are made and sealed as before.
hands, err := engineTakesHanded(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
handing := catalogue.Rendering{Hands: hands, Provisioner: provisionerAccount(plan)}
for _, m := range plan.Modules {
if handing.Handing(m) {
continue
}
// **A secret family is never made** (novox/hq ADR 0283): each member a person gave on this machine is
// placed, and one not given is nothing — the module says it waits for it.
for _, family := range m.OwnSecrets.Families() {
@@ -822,6 +862,22 @@ func renderingFor(ctx context.Context, open *stores, node string,
needed[m.Module][name] = sealed
}
}
// **The core's own credentials as the vault sealed them** (novox/hq ADR 0349 §16, issue 755), for an engine that
// opens the vault's box (core_box.go): the declaration carries the box, never the seal made above. Below that
// contract nothing here is asked and the composition is as it was.
var core coreBoxes
opensBox, err := engineOpensVaultBox(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
if opensBox {
core, err = coreBoxesFor(ctx, inv, node, plan.Modules, func(m catalogue.Manifest) bool {
return handing.Handing(m)
}, choosing)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
}
// And a certificate for this machine's name inside the mesh, when anything on it asks. Issued
// rather than stored: the node's key does not change, so signing again produces an equally
// valid certificate and there is nothing to keep in step.
@@ -1016,8 +1072,11 @@ func renderingFor(ctx context.Context, open *stores, node string,
ReadsHealth: readsHealth,
JudgesRoot: judgesRoot,
ReadsBusy: readsBusy,
Hands: hands,
Provisioner: provisionerAccount(plan),
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
VaultSealed: core.sealed, NotPlaced: core.notPlaced,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
Machines: machines, Zones: zones, Holders: replicas,
Suffix: overlay.Suffix(), MeshRange: meshRange, TunnelInterface: overlay.Interface, Accounts: accounts, Foundation: foundation,
@@ -1059,6 +1118,35 @@ func engineReadsBusy(ctx context.Context, inv *inventory.Inventory, node string)
return had && stated.Contract >= link.BusyContract, nil
}
// engineTakesHanded says whether a machine's node-engine writes a filled file on the provisioner client's word alone
// (link.HandedContract, design 57 P7), by its own newest statement: only there is a moved module handed to the
// provisioner (design 57 P6). Anywhere else a filled file would have two writers, and the module is sent as before.
func engineTakesHanded(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.HandedContract, nil
}
// provisionerAccount is the account the provisioner's client runs as on this machine (novox/hq ADR 0349 §4): the user
// the process of the module running it runs as, empty where no such module is on the machine.
func provisionerAccount(plan catalogue.Resolution) string {
for _, m := range plan.Modules {
if !runsTheProvisionerClient(m) {
continue
}
for _, r := range m.Resources {
if r["type"] == "process" {
if user, _ := r["user"].(string); user != "" && user != "root" {
return user
}
}
}
}
return ""
}
// busyNotSent is, for a node-engine that does not read a container's busy, each container whose module declares
// one: not sent it, so replaced at once while it may hold work (novox/hq ADR 0335). Nothing for one that reads it.
func busyNotSent(modules []catalogue.Manifest, reads bool) []string {
@@ -1846,6 +1934,42 @@ func issuedOutsideBy(shelf map[string]catalogue.Manifest, n catalogue.Needed) (s
return catalogue.OffersIssuedOutside(shelf, n.Name)
}
// issuedToHolder is whether the vault keeps a need's credential for its holder alone (issues 749, 757 and 779): its
// providing module is marked moved, so the vault holds the credential in a row of its own for each holder
// (filledTheNewWay), and its offer says either that the provider's software issues the provision's credential, or
// that the provider gives each consumer a login of its own (form pair, ADR 0158 §11). A credential every holder
// shares, or one a person gives, is neither.
func issuedToHolder(shelf map[string]catalogue.Manifest, n catalogue.Needed) bool {
if n.ByRecord || n.Module == "" {
return false
}
provider, known := shelf[n.Module]
if !known || !movedProvider(provider) {
return false
}
if _, issued := provider.IssuedCredentialOf(n.Name); issued {
return true
}
return provider.CredentialFormOf(n.Name) == catalogue.CredentialFormPair
}
// providerMoved is whether a need is answered by a module marked moved (issue 788), or by one whose offer of the
// provision says the provider helper answers it (issue 792).
func providerMoved(shelf map[string]catalogue.Manifest, n catalogue.Needed) bool {
if n.ByRecord || n.Module == "" {
return false
}
provider, known := shelf[n.Module]
return known && (movedProvider(provider) || provider.AnsweredByHelperFor(n.Name))
}
// movedProvider is whether a provider's credentials are the vault's: the module is marked moved, or it holds the
// mesh-vault seat, whose pair rows have been in the vault since V3 (issue 793), so its consumers' credentials
// (gitea's internal token and admin password) are filled from those very rows with the values unchanged.
func movedProvider(m catalogue.Manifest) bool {
return m.Moved || slices.ContainsFunc(m.Claims, func(c catalogue.Claim) bool { return c.Name == "mesh-vault" })
}
func providerModuleOf(resolved catalogue.Resolution, open *stores, ctx context.Context, n catalogue.Needed) string {
for _, m := range resolved.Modules {
if _, shared := m.SharedCredentialOf(n.Name); shared {
+93
View File
@@ -0,0 +1,93 @@
package main
import (
"context"
"fmt"
"path"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The list of predefined keys the mesh publishes (novox/hq ADR 0348 §7, design 57 package K2, issue 661).
//
// **The newest trunk build of the catalogue wins.** A build of the catalogue reads predefined.json at the commit
// it builds and carries it with the commit's generation (link.PredefinedList). Registration keeps it in the
// controller's store when that generation is larger than the one kept (inventory.KeepPredefined), so an older
// build's list heard later never replaces a newer one, and every manifest the controller reads again is judged
// against the list kept (catalogue.UsePredefined). mesh-config publishes it on the bus from there (design 57
// package K3); the controller keeps no config value, only the keys.
// fromTheCatalogue is whether a build is of the catalogue's repository, whose list alone the mesh publishes: by
// its name, in the URL form or the seat form.
func fromTheCatalogue(result link.BuildResult) bool {
for _, repository := range []string{result.Repository, sourceRepository(result)} {
name := strings.TrimSuffix(path.Base(strings.TrimRight(repository, "/")), ".git")
if name == catalogue.CatalogueRepository {
return true
}
}
return false
}
func sourceRepository(result link.BuildResult) string {
if result.Source == nil {
return ""
}
return result.Source.Repository
}
// carriedList is the list a build of the catalogue carried, read strictly; nil for a build of another
// repository, or one that carried none.
func carriedList(result link.BuildResult) (catalogue.PredefinedList, error) {
if result.Predefined == nil || !fromTheCatalogue(result) {
return nil, nil
}
list, err := catalogue.ParsePredefined(result.Predefined.Keys)
if err != nil {
return nil, fmt.Errorf("%s carried a list of predefined keys that is refused: %w", result.ID, err)
}
return list, nil
}
// judgedWith is the list a build's manifest is first judged against at registration: the catalogue's own at the
// commit built, which its build judged it against, where it carried one; the list kept otherwise.
func judgedWith(result link.BuildResult) (catalogue.PredefinedList, error) {
list, err := carriedList(result)
if err != nil || list != nil {
return list, err
}
return catalogue.PublishedPredefined(), nil
}
// loadPredefined makes the list kept in the store the one this controller judges manifests against. A store
// keeping none leaves the list it has.
func loadPredefined(ctx context.Context, inv *inventory.Inventory) error {
kept, has, err := inv.PredefinedKept(ctx)
if err != nil || !has {
return err
}
catalogue.UsePredefined(kept.Keys)
return nil
}
// publishPredefined keeps a trunk build's list when it is newer than the one kept, says so, and loads what the
// store keeps then — this build's, or a newer one another controller process kept.
func publishPredefined(ctx context.Context, inv *inventory.Inventory, result link.BuildResult,
list catalogue.PredefinedList) error {
kept, err := inv.KeepPredefined(ctx, inventory.PredefinedRecord{Keys: list, Repository: result.Repository,
Commit: result.Commit, Generation: result.Predefined.Generation, Build: result.ID})
if err != nil {
return err
}
if kept {
fmt.Printf("%s: the list of predefined keys at %s (generation %d) is published: %s\n", result.ID,
short(result.Commit), result.Predefined.Generation, strings.Join(list.Keys(), ", "))
} else {
fmt.Printf("%s: the list of predefined keys at %s (generation %d) is not newer than the one kept, which "+
"stands\n", result.ID, short(result.Commit), result.Predefined.Generation)
}
return loadPredefined(ctx, inv)
}
+197
View File
@@ -0,0 +1,197 @@
package main
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// The list of predefined keys (novox/hq ADR 0348 §7, design 57 package K2, issue 661): `module check` and the
// merge gate judge each manifest against the list at the root of its tree; registration publishes the newest
// trunk build's list of the catalogue and judges every module against the list the mesh publishes.
const theFirstKeys = `{
"font-size": {"type": "number", "description": "the size text is shown in, in points", "minimum": 6, "maximum": 72},
"agent-managed-settings": {"type": "object", "description": "the managed settings of every agent session, a JSON object", "trusted": true}
}`
// notifierFallingBackTo is the notifier in the config form, its font size falling back to the key named.
func notifierFallingBackTo(key, kind string) string {
def := "10"
if kind == "string" {
def = `"10"`
}
return `{"module":"notifier","version":"1",
"config":{"font-size":{"type":"` + kind + `","description":"the notification's font size, in points",
"default":` + def + `,"predefined":"` + key + `"}},
"resources":[{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644",
"content":"font = ${config:font-size}\n"}]}`
}
// withoutAPublishedList judges the test against no list the mesh published, and puts the process's back after.
func withoutAPublishedList(t *testing.T) {
t.Helper()
was := catalogue.ReplacePublished(nil)
t.Cleanup(func() { catalogue.ReplacePublished(was) })
}
// aRepositoryTree is a repository's checkout holding one module at modules/notifier, and the list when given.
func aRepositoryTree(t *testing.T, manifest string, list *string) string {
t.Helper()
root := t.TempDir()
if err := os.Mkdir(filepath.Join(root, ".git"), 0o755); err != nil {
t.Fatal(err)
}
if list != nil {
if err := os.WriteFile(filepath.Join(root, catalogue.PredefinedFile), []byte(*list), 0o644); err != nil {
t.Fatal(err)
}
}
return aTemplatedModule(t, filepath.Join(root, "modules", "notifier"), manifest, nil)
}
func TestModuleCheckJudgesAManifestAgainstTheListAtItsRepositoryRoot(t *testing.T) {
withoutAPublishedList(t)
list := theFirstKeys
check := func(path string) (string, error) {
var out bytes.Buffer
err := moduleCheck([]string{path}, &out)
return out.String(), err
}
if out, err := check(aRepositoryTree(t, notifierFallingBackTo("font-size", "number"), &list)); err != nil ||
!strings.Contains(out, "notifier: ok") {
t.Fatalf("a module naming font-size was refused: %v\n%s", err, out)
}
for _, c := range []struct {
manifest string
list *string
says string
}{
{notifierFallingBackTo("font-scale", "number"), &list, `"font-scale", which is not on the list`},
{notifierFallingBackTo("font-size", "string"), &list, "which is a number, and this value is a string"},
{notifierFallingBackTo("font-size", "number"), nil, "is not present"},
{notifierFallingBackTo("font-size", "number"), ptr(`{"font-size": {"type": "number"}}`), "no description"},
} {
if out, err := check(aRepositoryTree(t, c.manifest, c.list)); err == nil || !strings.Contains(out, c.says) {
t.Errorf("wanted the check refused saying %q, got %v:\n%s", c.says, err, out)
}
}
}
func ptr(s string) *string { return &s }
// writeTheList puts the list at the root of a tree the gate judges.
func writeTheList(t *testing.T, tree, list string) string {
t.Helper()
if err := os.WriteFile(filepath.Join(tree, catalogue.PredefinedFile), []byte(list), 0o644); err != nil {
t.Fatal(err)
}
return tree
}
func TestTheMergeGateJudgesEachManifestAgainstTheListOfTheTree(t *testing.T) {
withoutAPublishedList(t)
f, manifests := templatedMesh(t)
v := gateJudged(t, f, writeTheList(t, aTree(t, withEdit(manifests, "notifier",
notifierFallingBackTo("font-size", "number"))), theFirstKeys), "modules/notifier/module.json")
if v.Verdict == "fail" || v.Modules["notifier"] != "changed" {
t.Fatalf("a module naming font-size on the tree's list failed the gate (%v):\n%s", v.Modules, v.Report())
}
for _, c := range []struct{ manifest, list, says string }{
{notifierFallingBackTo("font-scale", "number"), theFirstKeys, `"font-scale", which is not on the list`},
{notifierFallingBackTo("font-size", "string"), theFirstKeys, "which is a number"},
{inlineNotifier, `{"font-size": {"type": "number", "description": "points", "unit": "pt"}}`, `unknown field "unit"`},
} {
v := gateJudged(t, f, writeTheList(t, aTree(t, withEdit(manifests, "notifier", c.manifest)), c.list),
"modules/notifier/module.json", catalogue.PredefinedFile)
if v.Verdict != "fail" || !strings.Contains(v.Report(), c.says) {
t.Errorf("wanted the gate to fail saying %q:\n%s", c.says, v.Report())
}
}
}
// catalogueBuild is a trunk build of the catalogue carrying the list at a generation.
func catalogueBuild(id, manifest, list string, generation int) link.BuildResult {
r := link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/notifier", Ref: "main", On: "anchor", Commit: strings.Repeat(id[len(id)-1:], 12),
Manifest: json.RawMessage(manifest), Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
if list != "" {
r.Predefined = &link.PredefinedList{Keys: json.RawMessage(list), Generation: generation}
}
return r
}
func TestRegistrationPublishesTheNewestTrunkListAndJudgesEveryModuleAgainstIt(t *testing.T) {
withoutAPublishedList(t)
open := aMesh(t)
ctx := t.Context()
// A manifest naming font-size registers, and its build's list is published.
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory,
catalogueBuild("b-1", notifierFallingBackTo("font-size", "number"), theFirstKeys, 10))); err != nil {
t.Fatalf("a module naming font-size was not registered: %v", err)
}
kept, has, err := open.inventory.PredefinedKept(ctx)
if err != nil || !has || kept.Generation != 10 || kept.Build != "b-1" {
t.Fatalf("the list was not published: %+v %v %v", kept, has, err)
}
if got := catalogue.PublishedPredefined(); len(got) != 2 {
t.Fatalf("the controller judges against %v, not the list published", got)
}
// One naming a key not on the list, or of another type, is refused and never enters the catalogue.
for i, c := range []struct{ manifest, says string }{
{notifierFallingBackTo("font-scale", "number"), "not on the list"},
{notifierFallingBackTo("font-size", "string"), "which is a number"},
} {
id := []string{"b-2", "b-3"}[i]
if _, _, err := takeIn(ctx, open.inventory, catalogueBuild(id, c.manifest, theFirstKeys, 10)); err == nil ||
!strings.Contains(err.Error(), c.says) {
t.Errorf("wanted the registration refused saying %q, got %v", c.says, err)
}
}
src, err := open.inventory.SourceOf(ctx, "notifier")
if err != nil || src.BuiltFrom != strings.Repeat("1", 12) {
t.Fatalf("a refused build replaced the registered one: %+v %v", src, err)
}
// A newer build's list, which drops agent-managed-settings, replaces the published one.
smaller := `{"font-size": {"type": "number", "description": "points", "minimum": 6, "maximum": 72}}`
if _, _, err := takeIn(ctx, open.inventory, catalogueBuild("b-4",
notifierFallingBackTo("font-size", "number"), smaller, 12)); err != nil {
t.Fatal(err)
}
// The older build's outcome heard later registers nothing new about the list: the newer stands.
if _, _, err := takeIn(ctx, open.inventory, catalogueBuild("b-5",
notifierFallingBackTo("font-size", "number"), theFirstKeys, 11)); err != nil {
t.Fatal(err)
}
kept, _, _ = open.inventory.PredefinedKept(ctx)
if kept.Build != "b-4" || len(kept.Keys) != 1 || len(catalogue.PublishedPredefined()) != 1 {
t.Fatalf("an older build's list heard later replaced the newer one: %+v", kept)
}
// A build of another repository publishes nothing, and its manifest is judged against the published list.
other := catalogueBuild("b-6", notifierFallingBackTo("font-size", "number"), theFirstKeys, 99)
other.Repository, other.Source.Repository = "http://forge.internal:20000/novox/notifier.git", "novox/notifier"
other.Manifest = json.RawMessage(strings.Replace(notifierFallingBackTo("font-size", "number"), `"notifier"`,
`"chimes"`, 1))
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, other)); err != nil {
t.Fatalf("a module of another repository naming a published key: %v", err)
}
if kept, _, _ = open.inventory.PredefinedKept(ctx); kept.Build != "b-4" {
t.Fatalf("a build of another repository published its list: %+v", kept)
}
other.ID, other.Manifest = "b-7", json.RawMessage(strings.Replace(notifierFallingBackTo("agent-managed-settings",
"number"), `"notifier"`, `"chimes"`, 1))
if _, _, err := takeIn(ctx, open.inventory, other); err == nil || !strings.Contains(err.Error(), "not on the list") {
t.Fatalf("a key only another repository's list holds was registered: %v", err)
}
}
+16
View File
@@ -54,6 +54,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -906,6 +907,21 @@ func setLayerIn(open *stores) setOnWarrant {
return "", fmt.Errorf("the layer of %s on %s changed since the operator was shown the change: it is not set; "+
"propose it again", p.Module, p.where())
}
// A proposal that carries config values, raised before the module's config moved, is not applied after it: its
// values in the layer would turn ReadsConfig off and undo the move (issue 830, novox/hq ADR 0348). A proposal
// of placement only is set as before: forwardPropose sends it down this path on purpose.
carried := p.Values
if p.Clear {
carried = before
}
if config, _ := catalogue.SplitPlacement(carried); len(config) > 0 {
if reads, err := inv.ReadsConfig(ctx, p.Module); err != nil {
return "", err
} else if reads {
return "", fmt.Errorf("the config of %s moved to mesh-config since the operator was shown the change: it is "+
"not set; propose it again (it then goes to mesh-config)", p.Module)
}
}
var changed string
if p.Clear {
if err := inv.ClearSettingsBy(ctx, p.Node, p.Module, setBy); err != nil {
+100
View File
@@ -0,0 +1,100 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// What a machine's provisioner client said of a module's files (novox/hq ADR 0362, ADR 0349 §15, design 57 P6,
// issue 672), read where a walk's end and a gate count a machine's send applied: a module the send handed to the
// provisioner is applied there only once the client reported `prov.applied` on that build's templates as well.
// provAppliedOn is the client's newest report of a module on a machine, from the stream that keeps it
// (broker.ProvAppliedStream); false when it has said nothing of it. A variable so a test can say the client's words.
var provAppliedOn = func(ctx context.Context, node, module string) (inventory.ProvReport, bool, error) {
var raw []byte
var said bool
read := func(js nats.JetStreamContext) error {
var err error
raw, said, err = broker.LastProvApplied(js, node, module)
return err
}
var err error
if d := doctorFrom; d != nil && d.js != nil {
err = read(d.js.Context())
} else {
err = onTheBus(func(conn *nats.Conn) error {
js, err := conn.JetStream(nats.Context(ctx))
if err != nil {
return err
}
return read(js)
})
}
if err != nil || !said {
return inventory.ProvReport{}, false, err
}
var r inventory.ProvReport
if err := json.Unmarshal(raw, &r); err != nil {
return inventory.ProvReport{}, false, fmt.Errorf("what %s's provisioner client said of %s is not its report: %w",
node, module, err)
}
return r, true, nil
}
// provLookup answers the clients' reports for one reading; a report that cannot be read is not counted, which
// leaves the machine waiting, then silent, never applied — and is said.
func provLookup(ctx context.Context) inventory.ProvLookup {
return func(node, module string) (inventory.ProvReport, bool) {
r, said, err := provAppliedOn(ctx, node, module)
if err != nil {
fmt.Fprintf(os.Stderr, "what %s's provisioner client said of %s could not be read: %v\n", node, module, err)
return inventory.ProvReport{}, false
}
return r, said
}
}
// provisioningFailed is every module whose files a machine's provisioner client last said it failed to place, from
// the stream that keeps their reports.
func provisioningFailed(js nats.JetStreamContext) ([]inventory.ProvReport, error) {
all, err := broker.ProvAppliedReports(js)
if err != nil {
return nil, err
}
var out []inventory.ProvReport
for subject, raw := range all {
var r inventory.ProvReport
if err := json.Unmarshal(raw, &r); err != nil {
return nil, fmt.Errorf("what was said on %s is not a provisioner client's report: %w", subject, err)
}
if r.Status == inventory.ProvFailed {
out = append(out, r)
}
}
return out, nil
}
// provisionedOutcome is a machine's report of one send as a delivery's end counts it (novox/hq ADR 0362 §1-5, issue
// 840): the engine's own report, combined with its provisioner client's for a send that handed modules to it. The
// outcome of a client's failure begins inventory.OutcomeProvisioningFailed; a report not yet applied on both sides
// is none (false). A send handing nothing is the engine's report unchanged. onSend says the report is about the send.
func provisionedOutcome(node string, sent inventory.SentDeclaration, r inventory.Reported, onSend bool,
prov inventory.ProvLookup) (string, bool) {
if len(sent.Provisioned) == 0 {
return r.Outcome, onSend
}
engine := inventory.AppliedReport{Outcome: r.Outcome}
if r.At != nil {
engine.At = *r.At
}
got, ok := inventory.ProvisionedSend(node, sent, engine, onSend, prov)
return got.Outcome, ok
}
+114
View File
@@ -0,0 +1,114 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The controller hands a moved module to the provisioner, and counts it applied only once its files are placed
// (design 57 P6, issue 672; novox/hq ADR 0349 §4, §15, ADR 0362).
// A declaration that hands files says which, and whose client fills them; one that hands none is byte for byte
// what it was.
func TestADeclarationNamesTheFilledFilesOnlyWhenItHandsAny(t *testing.T) {
plain := sendable{Resources: []map[string]any{{"id": "notify.configuration", "type": "file", "path": "/x",
"content": "a"}}, Sequence: 4}
before, err := plain.Body()
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(before), "filled_files") || strings.Contains(string(before), "provisioner_account") {
t.Fatalf("a declaration handing nothing names filled files: %s", before)
}
handing := plain
handing.Filled, handing.Provisioner = []string{"/x"}, "mesh-provisioner-client"
body, err := handing.Body()
if err != nil {
t.Fatal(err)
}
var envelope struct {
Filled []string `json:"filled_files"`
Provisioner string `json:"provisioner_account"`
}
if err := json.Unmarshal(body, &envelope); err != nil || len(envelope.Filled) != 1 || envelope.Filled[0] != "/x" ||
envelope.Provisioner != "mesh-provisioner-client" {
t.Fatalf("the handing declaration says %s", body)
}
}
func TestTheProvisionersAccountIsTheClientsProcessUser(t *testing.T) {
client := catalogue.Manifest{Module: "mesh-provisioner-client",
OwnSecrets: catalogue.OwnSecrets{"provisioner-broker": {Path: "/var/lib/x/broker"}},
Claims: []catalogue.Claim{{Name: "node-provisioner", Scope: "node"}},
Resources: []map[string]any{{"id": "account", "type": "user", "name": "mesh-provisioner-client"},
{"id": "client", "type": "process", "name": "mesh-provisioner-client", "user": "mesh-provisioner-client"}}}
if got := provisionerAccount(catalogue.Resolution{Modules: []catalogue.Manifest{client}}); got != "mesh-provisioner-client" {
t.Fatalf("the client's account is %q", got)
}
if got := provisionerAccount(catalogue.Resolution{}); got != "" {
t.Fatalf("a machine without a client names %q", got)
}
}
// The first-machine gate counts a handed module healthy only once its node's prov.applied for the new build has
// come (ADR 0349 §15, kept by ADR 0362 for a walk finishing at its gate); a failure there fails it.
func TestTheGateWaitsForTheProvisionersReport(t *testing.T) {
now := time.Date(2026, 10, 14, 23, 0, 0, 0, time.UTC)
since := now.Add(-time.Minute)
reported := now.Add(-50 * time.Second)
digest := "sha256:" + strings.Repeat("d", 64)
facts := func(said map[string]inventory.ProvReport) gateFacts {
return gateFacts{now: now,
reports: map[string]inventory.Reported{"g14": {Node: "g14", Outcome: inventory.OutcomeApplied,
At: &reported, Current: true}},
sent: map[string]inventory.SentDeclaration{"g14": {Digest: "sent", At: since,
Provisioned: map[string]string{"dunst": digest}}},
provisioned: func(node, module string) (inventory.ProvReport, bool) {
r, ok := said[node+"."+module]
return r, ok
}}
}
m := catalogue.Manifest{Module: "dunst"}
if h, why := judgeHealth("dunst", "", m, "g14", since, facts(nil)); h != healthNotYet ||
!strings.Contains(why, "has not reported dunst's files placed") {
t.Fatalf("judged without the provisioner's report: %v %q", h, why)
}
failed := map[string]inventory.ProvReport{"g14.dunst": {Status: inventory.ProvFailed, Templates: digest,
Reason: "a value is not there", At: now.Add(-10 * time.Second)}}
if h, why := judgeHealth("dunst", "", m, "g14", since, facts(failed)); h != healthBroken ||
!strings.Contains(why, "a value is not there") {
t.Fatalf("a failed placing is not the build's failure there: %v %q", h, why)
}
placed := map[string]inventory.ProvReport{"g14.dunst": {Status: inventory.ProvApplied, Templates: digest,
At: now.Add(-10 * time.Second)}}
if h, why := judgeHealth("dunst", "", m, "g14", since, facts(placed)); strings.Contains(why, "provisioner") {
t.Fatalf("placed, the module is still held on the provisioner: %v %q", h, why)
}
// A module the send did not hand is judged as before.
if h, why := judgeHealth("picom", "", catalogue.Manifest{Module: "picom"}, "g14", since, facts(nil)); strings.Contains(why, "provisioner") {
t.Fatalf("a module not handed waits on the provisioner: %v %q", h, why)
}
}
// A machine whose provisioner client failed to place a module's files has a failed apply, said as apply-failing
// once the failure outlasts an apply interval, with the client's reason (ADR 0362 §3).
func TestAFailedPlacingRaisesApplyFailing(t *testing.T) {
now := time.Date(2026, 10, 4, 14, 0, 0, 0, time.UTC)
f := calm(now)
f.provFailed = []inventory.ProvReport{{Node: "anchor", Module: "dunst", Status: inventory.ProvFailed,
Reason: "the templates cannot be fetched", At: now.Add(-time.Minute)}}
if got := watchApplyFailing(f); len(got) != 0 {
t.Fatalf("raised within an apply interval: %+v", got)
}
f.provFailed[0].At = now.Add(-10 * time.Minute)
got := watchApplyFailing(f)
if len(got) != 1 || got[0].Key() != "machine.anchor.apply-failing" ||
!strings.Contains(got[0].Summary, "dunst") || !strings.Contains(got[0].Said, "the templates cannot be fetched") {
t.Fatalf("a failed placing is not raised as anchor's apply-failing: %+v", got)
}
}
@@ -0,0 +1,62 @@
package main
import (
"slices"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
func aProvisioner() catalogue.Manifest {
return catalogue.Manifest{Module: "mesh-provisioner", Version: "1",
DefinesSeats: []catalogue.SeatDeclaration{{Name: broker.NodeProvisionerSeat, Scope: catalogue.ScopeNode,
Serves: []catalogue.Verb{{Name: "get"}}}},
Claims: []catalogue.Claim{{Name: broker.NodeProvisionerSeat, Scope: catalogue.ScopeNode}},
OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/run/mesh-provisioner/broker"},
broker.ProvisionerSecret: {Path: "/run/mesh-provisioner/client"}},
}
}
// A machine's provisioner client is issued an account of its own, under its own name, sealed to the machine as the
// module's own secret its service reads (novox/hq ADR 0349 clause 11, design 57 P3); until it is, a push refuses
// to invent one, as it does for a module's `broker`.
func TestTheProvisionerClientsAccountIsIssuedForItsServiceAndNeverInvented(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
m := aProvisioner()
register(t, open, m)
if _, err := assign(ctx, open, "laptop", m.Module); err != nil {
t.Fatal(err)
}
if err := busCredentialIssued(ctx, inv, "laptop", m.Module, broker.ProvisionerSecret); err == nil {
t.Fatal("a push would seal a placeholder for the client's account before it was issued")
}
if err := issueProvisionerClient(ctx, inv, m, "laptop", broker.Broker{Fingerprint: "ab"}, "10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
if _, minted, err := inv.BusUserHash(ctx, "provisioner.laptop"); err != nil || !minted {
t.Fatalf("the client's account is not minted: %v %v", minted, err)
}
states, err := inv.SecretsOf(ctx, "laptop", m.Module)
if err != nil {
t.Fatal(err)
}
if !slices.ContainsFunc(states, func(s inventory.SecretState) bool { return s.Name == broker.ProvisionerSecret }) {
t.Fatalf("the client's account is not sealed to the machine for its service: %+v", states)
}
if err := busCredentialIssued(ctx, inv, "laptop", m.Module, broker.ProvisionerSecret); err != nil {
t.Fatalf("an issued client account is refused: %v", err)
}
if !runsTheProvisionerClient(m) {
t.Fatal("the module is not read as running the client")
}
unread := m
unread.OwnSecrets = catalogue.OwnSecrets{"broker": {Path: "/run/broker"}}
if runsTheProvisionerClient(unread) {
t.Fatal("a module with nowhere to read the client's account is read as running it")
}
}
+7 -1
View File
@@ -265,6 +265,12 @@ func serve(ctx context.Context) (err error) {
// The facts snapshot a merge check is fed (novox/hq to-be 45 §9): kept current while this
// controller holds the lease, read by the build seat from the artifact store.
go exportingFacts(ctx, open, bus.Conn.ConnectedServerVersion)
// The record mirrors (novox/hq ADR 0289, design 57 F3): the assignments and each registered module's
// declared config on the bus, written from the store only when a record changed, by the lease holder.
// The provisioner's plan of each handed module, as its machine was last sent it (design 57 P6), in the module's
// assignment.
inv.SetProvisioningPlans(inv.SentProvisioningPlans)
go mirroringRecords(ctx, inv, bus.Conn, bus)
// Every call carries the lease's epoch, and its record is written only under the lease (novox/hq
// to-be 45 §6).
link.Calls.UnderLease(func() (uint64, error) { return theLease.epoch(ctx) })
@@ -1695,7 +1701,7 @@ func recordSent(ctx context.Context, inv *inventory.Inventory, node string, body
// send that is away look failed: it is said, loudly, and the machine's own record stands.
err = inv.RecordSentWith(kept, record.ID, digest, builds, epoch, sent.toldGeneration, inventory.Send{
Sequence: sent.sequence, Epoch: int64(sent.epoch), Sender: sent.sender, Generation: sent.generation,
Digest: digest, Modules: sent.modules})
Digest: digest, Modules: sent.modules, Provisioning: sent.plans})
var unrecorded *inventory.SendNotRecordedError
switch {
case errors.As(err, &unrecorded):
+106
View File
@@ -0,0 +1,106 @@
package main
import (
"context"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/recordmirror"
)
// The controller's record mirrors on the bus (novox/hq ADR 0289, design 54, design 57 F3): the assignments,
// and each registered module's declared config, which mesh-config resolves a module's config values from.
//
// **Built from the store, on a tick of their own, by the lease holder alone.** A record changes through a
// verb that runs as a command of its own, a reconcile, a merge's registration or a pending assignment made,
// so no one call site sees every change; what every change has in common is that it is committed to the
// store. So the serving controller reads the store every mirrorEvery and writes only what differs from what
// the bucket holds (ADR 0289 §1): a quiet mesh writes nothing, and a change rolled back is never read. A
// mirror is therefore at most mirrorEvery behind its store.
// mirrorEvery is how often the mirrors are compared with the store: two small reads of the store.
const mirrorEvery = 10 * time.Second
// mirroringRecords keeps the mirrors current for as long as this controller serves: ctx ends when it stops.
func mirroringRecords(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn, announce recordmirror.Announcer) {
api, err := jetstream.New(conn)
if err != nil {
fmt.Printf("the record mirrors are not kept: %v\n", err)
return
}
holds := func() bool {
epoch, err := theLease.epoch(ctx)
return err == nil && epoch > 0
}
epoch := func() uint64 {
e, _ := theLease.epoch(ctx)
return e
}
sources := map[string]func(context.Context) (map[string][]byte, error){
recordmirror.Assignments.Name: inv.AssignmentsMirror,
recordmirror.DeclaredConfig.Name: inv.DeclaredConfigMirror,
recordmirror.Predefined.Name: inv.PredefinedMirror,
}
writers := map[string]*recordmirror.Writer{}
failing := map[string]string{}
tick := time.NewTicker(mirrorEvery)
defer tick.Stop()
for {
if holds() {
for _, m := range recordmirror.Mirrors() {
why := mirrorOnce(ctx, api, m, writers, sources[m.Name], announce, epoch, holds)
if why != failing[m.Name] {
if why != "" {
fmt.Printf("the %s mirror cannot be kept: %s\n", m.Name, why)
} else {
fmt.Printf("the %s mirror is kept again\n", m.Name)
}
failing[m.Name] = why
}
}
}
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}
// mirrorOnce brings one mirror to what the store holds, and answers why it could not; empty when it could.
func mirrorOnce(ctx context.Context, api jetstream.JetStream, m recordmirror.Mirror, writers map[string]*recordmirror.Writer,
source func(context.Context) (map[string][]byte, error), announce recordmirror.Announcer, epoch func() uint64,
holds func() bool) string {
reading, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
w := writers[m.Name]
if w == nil {
kv, err := api.KeyValue(reading, m.Bucket)
if err != nil {
return fmt.Sprintf("the bucket %s cannot be read: %v", m.Bucket, err)
}
w = &recordmirror.Writer{Mirror: m, KV: kv, Announce: announce, Epoch: epoch, Holds: holds}
writers[m.Name] = w
}
want, err := source(reading)
if err != nil {
return fmt.Sprintf("the store cannot be read: %v", err)
}
lost := w.LostEvents
r, err := w.Sync(reading, want)
if r.Written+r.Removed > 0 {
fmt.Printf("the %s mirror moved: %d written, %d removed\n", m.Name, r.Written, r.Removed)
}
if w.LostEvents > lost {
fmt.Printf("the %s mirror lost %d event(s): the bucket holds the change, its notice was refused twice\n",
m.Name, w.LostEvents-lost)
}
if err != nil {
return err.Error()
}
return ""
}
+20 -4
View File
@@ -1013,7 +1013,16 @@ func advanceRest(ctx context.Context, open *stores, p *inventory.Plan, now time.
}
// **A machine that says nothing on its send is left silent, not failed** (ADR 0353 §4): it is not judged
// until it reports, and past ApplySilentAfter it is listed and left.
if !reportedOnSend(byNode[node], node, g) {
outcome, reported := byNode[node].Outcome, reportedOnSend(byNode[node], node, g)
if sent, kept := g.Sent[node]; kept && !gateFirstMachine {
// A module handed to the provisioner is applied only once its client said so too (ADR 0362, issue 840).
carried := make([]string, 0, len(g.Carried))
for _, c := range g.Carried {
carried = append(carried, c.Module)
}
outcome, reported = provisionedOutcome(node, sent.Only(carried...), byNode[node], reported, provLookup(ctx))
}
if !reported {
if now.Sub(*g.Since) > inventory.ApplySilentAfter {
g.Silent = true
r.Silent = append(r.Silent, node)
@@ -1028,18 +1037,25 @@ func advanceRest(ctx context.Context, open *stores, p *inventory.Plan, now time.
if !gateFirstMachine {
// **Judged by its report alone** (novox/hq ADR 0360 §7): applied passes; failed or refused is named and
// fails the walk, with nothing put back — apply-failing says it.
switch rep := byNode[node]; rep.Outcome {
switch outcome {
case inventory.OutcomeApplied:
decide(g, inventory.GatePassed, node+" reported it applied its send (novox/hq ADR 0360)", now)
passCarried(ctx, open, p, g, "")
r.Done = append(r.Done, node)
changed = true
case inventory.OutcomeFailed, inventory.OutcomeRefused:
decide(g, inventory.GateFailed, fmt.Sprintf("%s %s its send", node, rep.Outcome), now)
decide(g, inventory.GateFailed, fmt.Sprintf("%s %s its send", node, outcome), now)
r.Failed = append(r.Failed, node)
changed = true
notes = append(notes, fmt.Sprintf("%s %s its send: nothing put back, apply-failing says it", node, rep.Outcome))
notes = append(notes, fmt.Sprintf("%s %s its send: nothing put back, apply-failing says it", node, outcome))
default:
if strings.HasPrefix(outcome, inventory.OutcomeProvisioningFailed) {
decide(g, inventory.GateFailed, fmt.Sprintf("%s: %s", node, outcome), now)
r.Failed = append(r.Failed, node)
changed = true
notes = append(notes, fmt.Sprintf("%s: %s: nothing put back, apply-failing says it", node, outcome))
continue
}
judging = append(judging, node+": not yet reported on its send")
}
continue
+21 -4
View File
@@ -1153,7 +1153,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
if err != nil {
return false, err
}
waiting, failed, silent := walkReports(*p, reports, time.Now().UTC())
waiting, failed, silent := walkReports(*p, reports, time.Now().UTC(), provLookup(ctx))
if len(waiting) > 0 {
note := "sent; waiting for " + strings.Join(waiting, ", ") + " to report on their send"
changed := p.State != inventory.PlanRolling || p.Note != note
@@ -1189,17 +1189,29 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
// walkReports is, for every machine a walk sent its rest, whether it reported on that send (novox/hq ADR 0360 §4):
// those still waiting, those that reported a failed or refused apply, and those silent past ApplySilentAfter,
// each sorted. Pure.
func walkReports(p inventory.Plan, reports []inventory.Reported, now time.Time) (waiting, failed, silent []string) {
func walkReports(p inventory.Plan, reports []inventory.Reported, now time.Time, prov inventory.ProvLookup) (waiting, failed, silent []string) {
byNode := map[string]inventory.Reported{}
for _, r := range reports {
byNode[r.Node] = r
}
// The modules of this delivery sent to each machine: a machine is judged on those, not on every module its
// declaration hands to the provisioner (ADR 0362 §1-2, issue 840).
carried := map[string][]string{}
for m, s := range p.Modules {
if s == nil {
continue
}
for node := range s.Rest {
carried[node] = append(carried[node], m)
}
}
seen := map[string]bool{}
for _, s := range p.Modules {
if s == nil {
continue
}
for node, sent := range s.Rest {
sent = sent.Only(carried[node]...)
if seen[node] {
continue
}
@@ -1208,11 +1220,16 @@ func walkReports(p inventory.Plan, reports []inventory.Reported, now time.Time)
if at.IsZero() && s.SentAt != nil {
at = *s.SentAt
}
// A module handed to the provisioner is applied only once its client said so too (ADR 0362 §1-5).
outcome, reported := provisionedOutcome(node, sent, r, r.At != nil && sent.ReportsOn(r), prov)
switch {
case r.At != nil && sent.ReportsOn(r):
case reported:
seen[node] = true
if r.Outcome == inventory.OutcomeFailed || r.Outcome == inventory.OutcomeRefused {
switch {
case outcome == inventory.OutcomeFailed || outcome == inventory.OutcomeRefused:
failed = append(failed, node)
case strings.HasPrefix(outcome, inventory.OutcomeProvisioningFailed):
failed = append(failed, node+" ("+outcome+")")
}
case !at.IsZero() && now.Sub(at) > inventory.ApplySilentAfter:
seen[node] = true
@@ -0,0 +1,172 @@
package main
import (
"context"
"encoding/json"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// handingAnchor makes every send to anchor hand the provisioner the given modules (module -> templates digest), and
// the clients' words the given reports (key `node/module`), as the stream would give them (issue 840, ADR 0362).
func handingAnchor(t *testing.T, b *backlogMesh, handed map[string]string) map[string]inventory.ProvReport {
t.Helper()
says := map[string]inventory.ProvReport{}
wasSays := provAppliedOn
provAppliedOn = func(_ context.Context, node, module string) (inventory.ProvReport, bool, error) {
r, ok := says[node+"/"+module]
return r, ok, nil
}
inner := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
got, err := inner(ctx, open, names)
if err != nil {
return got, err
}
for _, n := range names {
if n != "anchor" {
continue
}
inv := open.inventory
s, _, err := inv.SentTo(ctx, n)
if err != nil {
return nil, err
}
builds, _, err := inv.SentBuilds(ctx, n)
if err != nil {
return nil, err
}
plans := map[string]inventory.Provisioning{}
for m, d := range handed {
plans[m] = inventory.Provisioning{Templates: inventory.TemplatesArchive{Digest: d}}
}
raw, err := json.Marshal(plans)
if err != nil {
return nil, err
}
id := nodeID(t, open, n)
if err := inv.RecordSentWith(ctx, id, s.Digest, builds, 0, 0, inventory.Send{Node: id, Provisioning: raw}); err != nil {
return nil, err
}
}
return got, nil
}
t.Cleanup(func() { sendRollout, provAppliedOn = inner, wasSays })
return says
}
func stepRelease(t *testing.T, b *backlogMesh, n int) inventory.Plan {
t.Helper()
for i := 0; i < n; i++ {
advancePlans(t.Context(), b.open)
}
return b.release(t)
}
// Issue 840 (novox/hq ADR 0362 §1-5): the release's rest gate, which every delivery passes since ADR 0360, counts a
// machine's send of a provisioned module applied only once the engine and the machine's provisioner client both said
// so; a failed client is a failed apply named with its reason; none past ApplySilentAfter is silent, never done.
func TestTheRestGateWaitsForTheProvisioner(t *testing.T) {
judgeLeavesAsTheCore(t)
applied := inventory.ProvReport{Status: inventory.ProvApplied, Templates: "t1", At: time.Now()}
t.Run("a engine applied, the client has not said: no verdict, the release stays open", func(t *testing.T) {
b := aBacklog(t)
handingAnchor(t, b, map[string]string{"app": "t1"})
p := stepRelease(t, b, 8)
g := p.Release.Gates["anchor"]
if !p.Open() || g == nil || g.Verdict != "" || g.Silent || slices.Contains(p.Release.Done, "anchor") {
t.Fatalf("anchor is judged before its provisioner reported: %s %s %+v", p.State, p.Note, g)
}
if !slices.Contains(p.Release.Done, "laptop") {
t.Fatalf("laptop hands nothing and passes on the engine's report: %+v", p.Release)
}
})
t.Run("b the client failed: a failed apply named with the reason", func(t *testing.T) {
b := aBacklog(t)
says := handingAnchor(t, b, map[string]string{"app": "t1"})
says["anchor/app"] = inventory.ProvReport{Status: inventory.ProvFailed, Templates: "t1", Reason: "no value", At: time.Now()}
p := stepRelease(t, b, 8)
g := p.Release.Gates["anchor"]
if g == nil || g.Verdict != inventory.GateFailed || !slices.Contains(p.Release.Failed, "anchor") ||
!strings.Contains(p.Note, "provisioning failed (app: no value)") {
t.Fatalf("the failure is not named: %s %q %+v %+v", p.State, p.Note, g, p.Release)
}
})
t.Run("c no client report past ApplySilentAfter: silent, never done", func(t *testing.T) {
b := aBacklog(t)
handingAnchor(t, b, map[string]string{"app": "t1"})
p := stepRelease(t, b, 8)
g := p.Release.Gates["anchor"]
if g == nil {
t.Fatalf("no gate: %+v", p.Release)
}
since := time.Now().Add(-inventory.ApplySilentAfter - time.Minute)
g.Since = &since
if err := b.open.inventory.SavePlan(t.Context(), &p); err != nil {
t.Fatal(err)
}
p = stepRelease(t, b, 1)
if !slices.Equal(p.Release.Silent, []string{"anchor"}) || slices.Contains(p.Release.Done, "anchor") {
t.Fatalf("anchor is not silent: %s %s %+v", p.State, p.Note, p.Release)
}
})
t.Run("d both applied: the gate passes", func(t *testing.T) {
b := aBacklog(t)
says := handingAnchor(t, b, map[string]string{"app": "t1"})
says["anchor/app"] = applied
p := stepRelease(t, b, 8)
if p.State != inventory.PlanDone || !slices.Contains(p.Release.Done, "anchor") || len(p.Release.Failed) > 0 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
})
t.Run("e a send handing nothing passes on the engine's report", func(t *testing.T) {
b := aBacklog(t)
p := stepRelease(t, b, 8)
if p.State != inventory.PlanDone || !slices.Contains(p.Release.Done, "anchor") {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
})
t.Run("f a module the delivery does not carry does not touch its end", func(t *testing.T) {
// anchor's declaration also hands `same`, rebuilt with the very bytes and so not carried: its failed report
// is none of this delivery's.
b := aBacklog(t)
says := handingAnchor(t, b, map[string]string{"app": "t1", "same": "t9"})
says["anchor/app"] = applied
says["anchor/same"] = inventory.ProvReport{Status: inventory.ProvFailed, Templates: "t9", Reason: "stuck", At: time.Now()}
p := stepRelease(t, b, 8)
if p.State != inventory.PlanDone || !slices.Contains(p.Release.Done, "anchor") || len(p.Release.Failed) > 0 {
t.Fatalf("another module's failure ended this delivery: %s %s %+v", p.State, p.Note, p.Release)
}
})
}
// Issue 840: a walk's end waits on the modules it sent to a machine, not on every module that machine's declaration
// hands to the provisioner.
func TestAWalkEndWaitsOnlyOnTheModulesItSent(t *testing.T) {
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
at := now.Add(-10 * time.Second)
handed := map[string]string{"app": "t1", "other": "t2"}
p := inventory.Plan{Modules: map[string]*inventory.PlanModule{"app": {Rest: map[string]inventory.SentDeclaration{
"ace": {Digest: "d-ace", At: now.Add(-time.Minute), Provisioned: handed}}}}}
reports := []inventory.Reported{{Node: "ace", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-ace"}}
says := map[string]inventory.ProvReport{"ace/app": {Status: inventory.ProvApplied, Templates: "t1", At: at}}
prov := func(node, module string) (inventory.ProvReport, bool) { r, ok := says[node+"/"+module]; return r, ok }
// other has said nothing: not this walk's to wait for.
if waiting, failed, silent := walkReports(p, reports, now, prov); len(waiting)+len(failed)+len(silent) != 0 {
t.Fatalf("missing: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
says["ace/other"] = inventory.ProvReport{Status: inventory.ProvFailed, Templates: "t2", Reason: "stuck", At: at}
if waiting, failed, silent := walkReports(p, reports, now, prov); len(waiting)+len(failed)+len(silent) != 0 {
t.Fatalf("failed: waiting %v, failed %v, silent %v", waiting, failed, silent)
}
// once the walk carries it too, it does.
p.Modules["other"] = &inventory.PlanModule{Rest: p.Modules["app"].Rest}
if _, failed, _ := walkReports(p, reports, now, prov); len(failed) != 1 || !strings.Contains(failed[0], "other: stuck") {
t.Fatalf("a carried module's failure is not named: %v", failed)
}
}
+13
View File
@@ -379,6 +379,19 @@ func rolloutMint(ctx context.Context, again bool) error {
}
modules++
case broker.KindProvisioner:
// A machine's provisioner client, on an account of its own (novox/hq ADR 0349 clause 11), delivered
// to the module running it.
m, inShelf := shelf[p.Module]
if !inShelf || !runsTheProvisionerClient(m) {
skipped++
continue
}
if err := issueProvisionerClient(ctx, inv, m, p.Node, known, busAddress); err != nil {
return err
}
modules++
default:
skipped++
}
+141 -11
View File
@@ -6,6 +6,7 @@ import (
"flag"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
)
@@ -39,6 +40,9 @@ func rotateCommand(ctx context.Context, args []string) error {
// of one provision, each with its own credential; one module that leaked its credential (novox/hq
// issue 268) is no reason to restart every other one on the machine.
module := set.String("module", "", "only this consuming module's credential")
// Why, which the vault asks of every rotation (issue 767) and the hand-act log records; required only where a
// provider's credential is the vault's, so nothing the operator typed before changes meaning.
why := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
@@ -89,17 +93,6 @@ func rotateCommand(ctx context.Context, args []string) error {
// Every machine at both ends, named before anything changes. A person about to rotate a
// production credential is entitled to know the blast radius before it is the past tense.
affected := map[string]bool{}
for _, h := range holders {
affected[h.Consumer] = true
affected[h.Provider] = true
}
machines := make([]string, 0, len(affected))
for name := range affected {
machines = append(machines, name)
}
sort.Strings(machines)
fmt.Printf("rotating %s for %d holder(s):\n", provision, len(holders))
for _, h := range holders {
// The module, because a machine may hold several credentials for one provision and
@@ -107,6 +100,35 @@ func rotateCommand(ctx context.Context, args []string) error {
fmt.Printf(" %s on %s, from %s%s\n", h.ConsumerModule, h.Consumer, h.Provider, asLocal(h.Local))
}
// **The vault's, where the provider is filled the new way** (issue 767, ADR 0349 clause 8): its credential is
// the vault's, so the vault remakes it and its holders take it from there; the controller writes no value of its
// own and sends nothing for it. Which holders go where is read before anything changes.
toVault, today, err := rotateRoutes(holders,
func(provider, provision string) (string, error) { return offeringModuleOn(ctx, provider, provision) },
filledTheNewWay)
if err != nil {
return fmt.Errorf("nothing was rotated: %w", err)
}
if len(toVault) > 0 && !why.given() {
return errors.New("not rotated: the vault holds this credential and rotates it only with a reason, " +
"`--why \"…\"`; nothing was asked")
}
if why.given() {
why.record(ctx, "rotate", append([]string{provision}, rotateNarrowArgs(*only, *module)...))
}
for _, g := range toVault {
words, err := rotateProvisionInVault(theVault(ctx), g, *only, *module, actingAs(), *why.why)
if err != nil {
return err
}
fmt.Println(words)
}
if len(today) == 0 {
return nil
}
holders = today
machines := machinesOf(holders)
for _, h := range holders {
if err := inv.RotateSecret(ctx, h.Provision, h.Consumer, h.ConsumerModule, h.Provider, h.Local); err != nil {
// Partly rotated, and said so plainly. What is gone is remade on the next push, so
@@ -137,6 +159,21 @@ func rotateCommand(ctx context.Context, args []string) error {
return nil
}
// machinesOf is every machine at both ends of these holders, in order.
func machinesOf(holders []inventory.Holder) []string {
affected := map[string]bool{}
for _, h := range holders {
affected[h.Consumer] = true
affected[h.Provider] = true
}
machines := make([]string, 0, len(affected))
for name := range affected {
machines = append(machines, name)
}
sort.Strings(machines)
return machines
}
// ofModule is the holders whose consuming module is this one; all of them when none is named.
func ofModule(holders []inventory.Holder, module string) []inventory.Holder {
if module == "" {
@@ -165,3 +202,96 @@ func asLocal(local string) string {
}
return " (as " + local + ")"
}
// rotateGroup is the holders of one provider's credential for one provision whose provider is filled the new way:
// the vault's to rotate (issue 767).
type rotateGroup struct {
Provider, Provision, Module string
Holders []inventory.Holder
}
// rotateRoutes says which holders the vault rotates, grouped by provider and provision, and which today's path
// does: the vault's are those whose provider's offering module on its machine is filled the new way (the switch
// `accept --provider` reads too). Anything that cannot be read is an error, never today's path by default: a value
// remade where its holders no longer look would be reported rotated and change nothing.
func rotateRoutes(holders []inventory.Holder, offering func(provider, provision string) (string, error),
ofTheVault vaultSwitch) ([]rotateGroup, []inventory.Holder, error) {
type key struct{ provider, provision string }
decided := map[key]int{} // index into groups, or -1 for today's path
var groups []rotateGroup
var today []inventory.Holder
for _, h := range holders {
k := key{h.Provider, h.Provision}
i, seen := decided[k]
if !seen {
module, err := offering(h.Provider, h.Provision)
if err != nil {
return nil, nil, fmt.Errorf("which module offers %s on %s cannot be read: %w", h.Provision, h.Provider, err)
}
ours := false
if module != "" {
if ours, err = ofTheVault(module); err != nil {
return nil, nil, err
}
}
i = -1
if ours {
i = len(groups)
groups = append(groups, rotateGroup{Provider: h.Provider, Provision: h.Provision, Module: module})
}
decided[k] = i
}
if i < 0 {
today = append(today, h)
continue
}
groups[i].Holders = append(groups[i].Holders, h)
}
return groups, today, nil
}
// rotateNarrowArgs is --consumer and --module as typed, for the hand-act log.
func rotateNarrowArgs(consumer, module string) []string {
var out []string
if consumer != "" {
out = append(out, "--consumer", consumer)
}
if module != "" {
out = append(out, "--module", module)
}
return out
}
// rotateProvisionInVault asks the vault to rotate one provider's credential for a provision (issue 767): a shared
// value is made anew for every holder, and the vault refuses it for one consumer alone; for a provider whose
// consumers keep a login each (ADR 0158 §11), consumer and module name whose own login is remade, and only theirs.
// The vault says secret-changed and each holder takes the new value from it; nothing is sealed or sent here.
func rotateProvisionInVault(ask vaultAsker, g rotateGroup, consumer, module, by, why string) (string, error) {
if strings.TrimSpace(why) == "" {
return "", errors.New("not rotated: the vault rotates a credential only with a reason, `--why \"…\"`; nothing was asked")
}
args := provisionArgs(g.Provider, g.Provision)
args["by"], args["why"] = by, strings.TrimSpace(why)
if consumer != "" {
args["consumer"] = consumer
}
if module != "" {
args["consumer_module"] = module
}
changed, err := endVaultCall(ask, "rotate", args)
if err != nil {
return "", fmt.Errorf("%s from %s not rotated, and nothing else was: %w", g.Provision, g.Provider, err)
}
if len(changed.Each) > 0 {
each := make([]string, 0, len(changed.Each))
for _, c := range changed.Each {
each = append(each, fmt.Sprintf("%s (generation %d)", c.Name, c.Generation))
}
return fmt.Sprintf("rotated in the vault, each consumer's own login of %s from %s, asked by %s: %s; the values "+
"are sealed and not shown, and each holder takes its own from the vault", g.Provision, g.Provider,
changed.By, strings.Join(each, ", ")), nil
}
return fmt.Sprintf("rotated %s in the vault (generation %d), one value for every holder of %s from %s, asked by "+
"%s; the value is sealed and not shown, and each holder takes it from the vault", changed.Name,
changed.Generation, g.Provision, g.Provider, changed.By), nil
}
+124
View File
@@ -0,0 +1,124 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
)
// `rotate <provision>` reaches the vault for a provider filled the new way (issue 767), and the controller writes no
// value of its own for it: its holders go to the vault, named by provider and provision, and no other holder does.
func TestARotationOfAProvisionTheVaultOwnsGoesToTheVaultAndNotTheControllersStore(t *testing.T) {
holders := []inventory.Holder{
{Provision: "postgres-database", Consumer: "g14", ConsumerModule: "gitea", Provider: "novox"},
{Provision: "postgres-database", Consumer: "ace", ConsumerModule: "umami", Provider: "novox"},
{Provision: "postgres-database", Consumer: "ace", ConsumerModule: "n8n", Provider: "ace"},
}
offering := func(provider, provision string) (string, error) {
if provider == "novox" {
return "postgres", nil
}
return "postgres-old", nil
}
var asked []string
moved := func(module string) (bool, error) {
asked = append(asked, module)
return module == "postgres", nil
}
vault, today, err := rotateRoutes(holders, offering, moved)
if err != nil {
t.Fatal(err)
}
if len(vault) != 1 || vault[0].Provider != "novox" || vault[0].Provision != "postgres-database" ||
vault[0].Module != "postgres" || len(vault[0].Holders) != 2 {
t.Fatalf("to the vault: %+v", vault)
}
if len(today) != 1 || today[0].Provider != "ace" {
t.Fatalf("today's path rotates a holder the vault owns, or misses one it does not: %+v", today)
}
if fmt.Sprint(asked) != "[postgres postgres-old]" {
t.Fatalf("the switch was read for %v, not the offering module", asked)
}
}
// A switch that cannot be read, or a provider whose offering module cannot be found, routes nothing: an error
// before anything changes, never today's path by default.
func TestARotationWhoseOwnerCannotBeReadChangesNothing(t *testing.T) {
holders := []inventory.Holder{{Provision: "postgres-database", Consumer: "g14", ConsumerModule: "gitea",
Provider: "novox"}}
broken := func(string) (bool, error) { return false, errors.New("the catalogue is unreadable") }
if _, _, err := rotateRoutes(holders, func(string, string) (string, error) { return "postgres", nil }, broken); err == nil {
t.Fatal("an unreadable switch fell back to today's path")
}
fine := func(string) (bool, error) { return true, nil }
if _, _, err := rotateRoutes(holders, func(string, string) (string, error) { return "", errors.New("no store") },
fine); err == nil {
t.Fatal("an unreadable provider fell back to today's path")
}
}
// What the vault is told: the provider and provision, and --consumer and --module as the one consumer whose own
// login is remade; nothing named when the operator named nobody, so a shared value is rotated for every holder.
func TestTheVaultIsToldTheProvisionAndTheConsumerTheOperatorNamed(t *testing.T) {
group := rotateGroup{Provider: "novox", Provision: "postgres-database", Module: "postgres"}
for _, c := range []struct {
consumer, module string
want, absent []string
}{
{"", "", []string{`"provider":"novox"`, `"provision":"postgres-database"`, `"why":"yearly"`,
`"by":"g14/claude-code"`}, []string{`"consumer"`, `"consumer_module"`}},
{"g14", "", []string{`"consumer":"g14"`}, []string{`"consumer_module"`}},
{"", "gitea", []string{`"consumer_module":"gitea"`}, []string{`"consumer":`}},
{"g14", "gitea", []string{`"consumer":"g14"`, `"consumer_module":"gitea"`}, nil},
} {
var said string
ask := func(verb string, args map[string]any) (json.RawMessage, error) {
raw, _ := json.Marshal(map[string]any{"verb": verb, "args": args})
said = string(raw)
return json.Marshal(vaultChanged{Name: "provision/novox/postgres-database", By: "g14/claude-code",
Each: []vaultChanged{{Name: "pair/novox/postgres-database/g14/gitea", Generation: 3}}})
}
words, err := rotateProvisionInVault(ask, group, c.consumer, c.module, "g14/claude-code", "yearly")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, `"verb":"rotate"`) {
t.Fatalf("%+v: the vault was asked %s", c, said)
}
for _, w := range c.want {
if !strings.Contains(said, w) {
t.Errorf("%+v: the vault was not told %s: %s", c, w, said)
}
}
for _, a := range c.absent {
if strings.Contains(said, a) {
t.Errorf("%+v: the vault was told %s: %s", c, a, said)
}
}
if !strings.Contains(words, "pair/novox/postgres-database/g14/gitea (generation 3)") {
t.Errorf("%+v: the operator was not told whose login was remade: %s", c, words)
}
}
}
// The vault rotates only with a why: asked of the vault without one, nothing is sent; and the vault's refusal (a
// shared value named for one consumer) is the operator's answer, not a rotation in the controller's store.
func TestARotationInTheVaultSaysWhyAndPassesOnItsRefusal(t *testing.T) {
group := rotateGroup{Provider: "anchor", Provision: "stand-in-api", Module: "stand-in"}
sent := 0
ask := func(string, map[string]any) (json.RawMessage, error) {
sent++
return nil, errors.New("not rotated: provision/anchor/stand-in-api is one value every holder shares")
}
if _, err := rotateProvisionInVault(ask, group, "", "", "g14/claude-code", " "); err == nil || sent != 0 {
t.Fatalf("a rotation without a why reached the vault: %v", err)
}
if _, err := rotateProvisionInVault(ask, group, "g14", "", "g14/claude-code", "suspected"); err == nil ||
!strings.Contains(err.Error(), "every holder shares") {
t.Fatalf("the vault's refusal was not said: %v", err)
}
}
@@ -0,0 +1,55 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq ADR 0366 §3: a module's own runtime stated unhealthy by its runtime-health raises the module's usual
// condition, module.<module>.<machine>.unhealthy, through ADR 0240's path, saying the tool's reason, and the first
// statement that says it healthy clears it.
func TestAnUnhealthyOwnRuntimeRaisesTheModulesUsualCondition(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv, k := open.inventory, conditionsFrom
const key = "module.mesh-vault.anchor.unhealthy"
said := func(at time.Time, state, reason string) link.Health {
return link.Health{Contract: link.LivenessContract, At: at, Resources: []link.ResourceHealth{{
Module: "mesh-vault", Resource: "mesh-vault.own-runtime", Kind: "process", Target: "mesh-vault-runtime.service",
State: state, Reason: reason, Since: at, Check: "tool"}}}
}
why := "re-sealing to the operator key failed on 3 passes in a row: the operator key is not readable"
for i := 0; i < 2; i++ {
at := h0.Add(time.Duration(i) * time.Minute)
if err := stateHealth(ctx, inv, k, "anchor", said(at, link.StateUnhealthy, why), at); err != nil {
t.Fatal(err)
}
}
c, had, err := k.Get(ctx, key)
if err != nil || !had {
t.Fatalf("two unhealthy statements of its own runtime raised no %s: %v", key, err)
}
evidence := ""
for _, e := range c.Evidence {
evidence += e.Said + "\n"
}
if !strings.Contains(c.Summary+evidence, "operator key is not readable") {
t.Errorf("the condition does not say the tool's reason: %s / %s", c.Summary, evidence)
}
at := h0.Add(3 * time.Minute)
if err := stateHealth(ctx, inv, k, "anchor", said(at, link.StateHealthy, ""), at); err != nil {
t.Fatal(err)
}
list, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
for _, o := range list {
if o.Key == key {
t.Fatalf("a healthy statement did not clear %s", key)
}
}
}
+66 -1
View File
@@ -730,6 +730,38 @@ func (a *verbArguments) commandLine() ([]string, error) {
return nil, fmt.Errorf("%w: letting go of artifacts is a hand act, which says why. Nothing was done", err)
}
return append(argv, "--confirm", "--why", why), nil
// The move of placement out of the settings layers (issue 736), a dry run unless confirmed.
case "move-placement":
argv := []string{"move-placement", "--json"}
why := str("why")
if !on("confirm") {
if why != "" {
return nil, errors.New("move-placement takes why only with confirm: without confirm nothing is " +
"moved, and a reason for nothing would be recorded nowhere. Nothing was done")
}
return argv, nil
}
if err := need("why"); err != nil {
return nil, fmt.Errorf("%w: moving the settings layers' placement is an act by hand, which says why. "+
"Nothing was done", err)
}
return append(argv, "--confirm", "--why", why), nil
// The move of config into mesh-config (issue 664), a dry run unless confirmed.
case "move-config":
argv := []string{"move-config", "--json"}
why := str("why")
if !on("confirm") {
if why != "" {
return nil, errors.New("move-config takes why only with confirm: without confirm nothing is set, " +
"and a reason for nothing would be recorded nowhere. Nothing was done")
}
return argv, nil
}
if err := need("why"); err != nil {
return nil, fmt.Errorf("%w: moving the settings layers into mesh-config is an act by hand, which says why. "+
"Nothing was done", err)
}
return append(argv, "--confirm", "--why", why), nil
case "images":
if err := need("node"); err != nil {
return nil, err
@@ -837,6 +869,13 @@ func (a *verbArguments) commandLine() ([]string, error) {
return nil, err
}
return []string{"secret", "ask", str("node"), str("module"), str("secret"), "--at", str("at")}, nil
case "secret-move":
// The move of the controller's secrets into the vault (design 57's V3, issue 676): no value in the arguments
// or the answer, a machine's name at most.
if n := str("node"); n != "" {
return []string{"secret", "move", n}, nil
}
return []string{"secret", "move"}, nil
case "issued":
// The credential a provider's software issued, taken from the provider (novox/hq ADR 0305): no value in
// the arguments, the provision alone.
@@ -866,6 +905,13 @@ func (a *verbArguments) commandLine() ([]string, error) {
if m := str("module"); m != "" {
argv = append(argv, "--module", m)
}
// Why, which the vault asks of a rotation of a credential it holds (issue 767); a cause only beside it.
if w := str("why"); w != "" {
argv = append(argv, "--why", w)
if c := str("cause"); c != "" {
argv = append(argv, "--cause", c)
}
}
return argv, nil
}
_, node := a.given["node"]
@@ -1036,6 +1082,10 @@ var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "co
"hand-acts": true, "durations": true, "conditions": true, "doctor": true, "retire": true, "cleanup": true, "data": true,
// What the records say the registries may keep (novox/hq ADR 0251).
"artifacts": true, "collect": true, "images": true, "mirrors": true,
// The move of placement says each layer it moved (issue 736).
"move-placement": true,
// The move of config says each value it set and where from (issue 664).
"move-config": true,
// The delivery's owner's verbs answer JSON where they read (plan, order, check, walks) — novox/hq ADR 0239.
"delivery": true,
// The build queue, each ask with its position among the waiting (novox/hq issue 455).
@@ -1081,6 +1131,10 @@ func repairingCommand(argv []string) string {
return "collect"
case argv[0] == "mirrors" && slices.Contains(argv, "--confirm"):
return "mirrors"
case argv[0] == "move-placement" && slices.Contains(argv, "--confirm"):
return "move-placement"
case argv[0] == "move-config" && slices.Contains(argv, "--confirm"):
return "move-config"
}
return ""
}
@@ -1732,6 +1786,16 @@ func takenFromTheProvider(argv []string) bool {
!strings.HasPrefix(argv[2], "-")
}
// movedByTheirMachines is exactly the line the `secret-move` verb composes: `secret move`, or `secret move <node>`
// (design 57's V3, issue 676). It carries no value and answers none: each machine opens what was sealed to it and
// hands it to the vault, which never replaces a value it holds.
func movedByTheirMachines(argv []string) bool {
if len(argv) < 2 || argv[0] != "secret" || argv[1] != "move" {
return false
}
return len(argv) == 2 || (len(argv) == 3 && argv[2] != "" && !strings.HasPrefix(argv[2], "-"))
}
// terminalOnly refuses, through any verb, a command that is the operator's at the controller's terminal
// alone (novox/hq ADR 0266). **Every `node` subcommand that is not a read**: `node account` and
// `node agent-account` above all. Whoever may call a verb includes agents, and an agent that named itself
@@ -1748,7 +1812,8 @@ func terminalOnly(argv []string) error {
// Of a secret's commands only rotation, which seals the new value to the machine that uses it, and the
// `give` verb's own line: an own secret typed by the operator into the desk's hidden prompt, sealed to this
// call and then to the module's machine, so no value travels in the verb or its answer (hq ADR 0259 §10).
if argv[0] == "secret" && (len(argv) < 2 || argv[1] != "rotate") && !givenAtTheDesk(argv) && !takenFromTheProvider(argv) {
if argv[0] == "secret" && (len(argv) < 2 || argv[1] != "rotate") && !givenAtTheDesk(argv) && !takenFromTheProvider(argv) &&
!movedByTheirMachines(argv) {
return terminalRefusal("secret %s is run at the controller's terminal only, never through a verb: accepting, "+
"recovering or exporting a secret hands it to whoever asks, and that includes agents (novox/hq ADR "+
"0266). Nothing was done", strings.Join(argv[1:], " "))
@@ -285,6 +285,8 @@ var accountedFlags = map[string]map[string]string{
"collect": {"json": "set by the verb: the answer is data"},
"images": {"json": "set by the verb: the answer is data"},
"mirrors": {"json": "set by the verb: the answer is data"},
"move-placement": {"json": "set by the verb: the answer is data"},
"move-config": {"json": "set by the verb: the answer is data"},
"conditions history": {"json": "set by the verb: the answer is data"},
"conditions show": {"json": "set by the verb: the answer is data"},
"healers": {"json": "set by the verb: the answer is data"},
+124 -6
View File
@@ -53,6 +53,9 @@ func secretCommand(ctx context.Context, args []string) error {
return secretRecover(ctx, args[1:])
case "export":
return secretExport(ctx, args[1:])
case "move":
// The controller's secrets moved into the vault, each by the machine it was sealed to (design 57's V3).
return secretMove(ctx, args[1:])
default:
return errors.New(secretUsage)
}
@@ -111,7 +114,12 @@ func secretCommand(ctx context.Context, args []string) error {
if value == "" {
return errors.New("there is nothing to seal")
}
return acceptForProvision(ctx, node, module, name, value)
// The provider's module decides the path: filled the new way, the vault keeps the provision's credential.
return bySwitch(module, filledTheNewWay,
func() error { return acceptForProvision(ctx, node, module, name, value) },
func() error {
return said(acceptIntoVault(theVault(ctx), provisionArgs(node, name), value, actingAs()))
})
}
if *desk != "" {
if *from != "" || *provider != "" {
@@ -128,23 +136,112 @@ func secretCommand(ctx context.Context, args []string) error {
if value == "" {
return errors.New("there is nothing to seal")
}
// The module whose secrets hold the value decides the path: with --provider, the value is the provider's one
// credential for the provision, so the provider's module decides, as for an outside offer above.
decides := module
if *provider != "" {
if decides, err = offeringModuleOn(ctx, *provider, name); err != nil {
return err
}
}
return bySwitch(decides, filledTheNewWay,
func() error { return acceptKept(ctx, node, module, name, *provider, *local, value) },
func() error {
// Sealed here to the vault's key for this one call; only the sealed box leaves this process.
target := ownSecretArgs(node, module, name)
if *provider != "" {
if *local != "" {
return errors.New("--local names a credential inside the module; the vault keeps one credential " +
"per provider and provision, so it takes no --local: nothing was sealed")
}
target = provisionArgs(*provider, name)
}
if *provider != "" {
return said(acceptIntoVault(theVault(ctx), target, value, actingAs()))
}
return giveIntoVault(ctx, node, module, name, target, value)
})
}
// announceTheGiven is the announcement of a secret given at the terminal (announceSecretGiven, the step acceptKept
// runs); a variable so a test can make it fail.
var announceTheGiven = announceSecretGiven
// giveIntoVault is `secret accept` of a module's own secret for a module filled the new way, under the same
// N1-give rule as acceptKept, through the same steps (keepGiven, announceSecretGiven): a module that runs as an
// account of its own is announced on the operator's channels before the vault is asked, and when the announcement
// fails nothing is sealed and the vault is never called; any other is sealed first and announced after.
func giveIntoVault(ctx context.Context, node, module, name string, target map[string]any, value string) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
trusted, err := open.inventory.RunsAsItsOwnAccount(ctx, module)
if err != nil {
return err
}
var words string
askedBy := link.Caller()
_, unannounced, err := keepGiven(trusted,
func() error {
return announceTheGiven(ctx, node, module, name, "at the controller's terminal", askedBy)
},
func() (bool, error) {
var err error
words, err = acceptIntoVault(theVault(ctx), target, value, actingAs())
return false, err
})
if err != nil {
if trusted && unannounced != nil {
return fmt.Errorf("%s runs as an account of its own, and the change of its %s could not be announced on "+
"your channels first, so nothing was sealed and the vault was not asked: %w", module, name, err)
}
return err
}
fmt.Println(words)
if unannounced != nil {
fmt.Printf(" this change could NOT be announced on the operator's channels: %v\n", unannounced)
}
// The giving is recorded, never the value (novox/hq ADR 0380 §6), as acceptKept records its own.
act := terminalGiveHandAct(node, module, name, askedBy)
if err := onTheBus(func(conn *nats.Conn) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
}); err != nil {
fmt.Printf(" this act could NOT be recorded in the hand-act log, and is done anyway: %v\n", err)
}
return nil
}
// said prints what was done, or answers why not.
func said(words string, err error) error {
if err != nil {
return err
}
fmt.Println(words)
return nil
}
// acceptKept is today's `secret accept`: the value sealed into the controller's store, for a module whose secrets
// are filled the old way.
func acceptKept(ctx context.Context, node, module, name, provider, local, value string) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if *provider != "" {
if provider != "" {
// Into the pair, not into the module's own secrets: what the provider is asked to create
// and what the consumer reads are the same value, and neither end can be told a different
// one later without the other (novox/hq 04-ISSUES/070).
if err := open.inventory.AcceptSecretForPair(ctx, name, node, module, *provider, *local, value); err != nil {
if err := open.inventory.AcceptSecretForPair(ctx, name, node, module, provider, local, value); err != nil {
return err
}
fmt.Printf("%s on %s now holds %q from %s%s, sealed to both machines.\n", module, node, name, *provider, asLocal(*local))
fmt.Printf("%s on %s now holds %q from %s%s, sealed to both machines.\n", module, node, name, provider, asLocal(local))
fmt.Printf(" the mesh cannot read it back, will not replace it with one of its own, and will not rotate it\n")
fmt.Printf(" run `push %s` and `push %s` to send it\n", *provider, node)
fmt.Printf(" run `push %s` and `push %s` to send it\n", provider, node)
return nil
}
// A trusted party's secret is announced before it is kept (the confirmation review of 2026-10-09, N1-give):
@@ -221,7 +318,8 @@ const secretUsage = "secret rotate <node> <module> <name> [--why <text> [--cause
"secret accept <node> <module> <name> [--from <file> | --at-desk <machine>] [--provider <node> [--local <name>]]\n" +
"secret accept <provider's node> <provider module> <provision> [--from <file>] (a credential issued outside the mesh, for every consumer)\n" +
"secret recover <node> <module> <name> --key <operator-key> [--out <file>] [--from-export <file>] [--provider <node>]\n" +
"secret export [--out <file>]"
"secret export [--out <file>]\n" +
"secret move [<node>] (every secret the controller holds, moved into the vault by the machine it was sealed to)"
// secretRecover is break-glass: a secret opened with the operator's key, written to a file.
//
@@ -500,6 +598,26 @@ func secretRotate(ctx context.Context, args []string) error {
return errors.New(secretUsage)
}
node, module, name := rest[0], rest[1], rest[2]
return bySwitch(module, filledTheNewWay,
func() error { return rotateKept(ctx, node, module, name, why) },
func() error {
reason := ""
if why.why != nil {
reason = strings.TrimSpace(*why.why)
}
words, err := rotateInVault(theVault(ctx), ownSecretArgs(node, module, name), actingAs(), reason)
if err != nil {
return err
}
why.record(ctx, "secret rotate", []string{node, module, name})
fmt.Println(words)
return nil
})
}
// rotateKept is today's `secret rotate`: made anew in the controller's store and sent, for a module whose
// secrets are filled the old way.
func rotateKept(ctx context.Context, node, module, name string, why handActFlags) error {
open, err := openStores(ctx)
if err != nil {
return err
+336
View File
@@ -0,0 +1,336 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// `secret move` — the controller's secrets move into the vault (novox/hq ADR 0349 clause 7; design 57's V3, issue 676;
// the operator's decision of 2026-10-14 on that issue). Run once, after the feature's delivery.
//
// **The controller cannot open what it holds, so each machine moves its own.** For every machine — or the ones named —
// the controller reads every row sealed to it, as the store holds it, and asks that machine's node-engine to move each
// one, signed (link.AskMoveSecret): the engine opens it with its sealing key and hands it to the vault's move-in, sealed
// to the call's key, keeping its origin and generation. The vault takes it from that engine alone, never over a value
// it holds, and reads back what it kept to the engine, which compares; the engine answers only then. What moved is
// recorded here, never a value. Nothing is rotated, and nothing points at the vault because of this: a module's
// secrets are filled from the vault only once it moves (V4, issue 647).
//
// **A machine asleep is moved when it is next heard from**: once the move has started, a machine that reports is moved
// in the background (moveWhenHeard), and what is left of it is said on the next `secret move`.
//
// The controller's record of asks at a desk, which holds no value, is copied as it is, once (`move-asks`).
const secretMoveUsage = "secret move [<node>] — move every secret the controller holds into the vault, as it is: each " +
"machine's node-engine opens what was sealed to it and hands it to the vault, which reads it back; no value is " +
"rotated, and a machine not answering is moved when it is next heard from"
// moveAsk is the ask of one row of a machine's engine.
type moveAsk func(row inventory.SecretToMove, ask link.MoveSecretAsk) (link.MoveSecretAnswer, error)
// nodeMove is what the move did on one machine.
type nodeMove struct {
Node string
Moved int
Already int
// Refused are the rows not moved, each with why; Changed the rows changed in the store after they moved.
Refused []string
Changed []string
// Unreached is why the machine's engine could not be asked: it is moved when next heard from.
Unreached error
// Left is how many rows were not asked because the machine was not reached.
Left int
// Failed is why its rows could not be read.
Failed error
}
// done says nothing is left to move on the machine.
func (m nodeMove) done() bool {
return m.Failed == nil && m.Unreached == nil && len(m.Refused) == 0
}
func (m nodeMove) String() string {
var b strings.Builder
if m.Failed != nil {
fmt.Fprintf(&b, "%s: its rows could not be read, nothing was moved: %v\n", m.Node, m.Failed)
return b.String()
}
fmt.Fprintf(&b, "%s: %d moved into the vault and read back the same, %d there already", m.Node, m.Moved, m.Already)
if n := len(m.Refused); n > 0 {
fmt.Fprintf(&b, ", %d not moved", n)
}
b.WriteString("\n")
for _, r := range m.Refused {
fmt.Fprintf(&b, " not moved: %s\n", r)
}
for _, c := range m.Changed {
fmt.Fprintf(&b, " %s was changed in the controller's store after it moved; the vault keeps what it took, "+
"and the move never replaces it\n", c)
}
if m.Unreached != nil {
fmt.Fprintf(&b, " %d left: %s was not reached (%v); it is moved when it is next heard from\n", m.Left, m.Node,
m.Unreached)
}
return b.String()
}
// moveRowAsk is a row as its machine's engine is asked to move it.
func moveRowAsk(r inventory.SecretToMove) link.MoveSecretAsk {
a := link.MoveSecretAsk{Origin: r.Origin, Generation: r.Generation, MadeAt: r.MadeAt.UTC(), Sealed: r.Sealed,
SealedTo: r.SealedTo}
if r.Own() {
a.Module, a.Name = r.Module, r.Name
} else {
a.Provider, a.Provision, a.Consumer, a.ConsumerModule, a.Local = r.Provider, r.Provision, r.Consumer,
r.ConsumerModule, r.Local
}
return a
}
// moveNode moves every row sealed to one machine that has not moved, one ask each, and records what the vault kept.
// The first ask the engine does not answer ends the machine's turn: the rest wait for it to be heard from again.
func moveNode(node string, rows func(string) ([]inventory.SecretToMove, error), ask moveAsk,
record func(inventory.SecretToMove, int) error) nodeMove {
m := nodeMove{Node: node}
all, err := rows(node)
if err != nil {
m.Failed = err
return m
}
for n, r := range all {
switch {
case r.Moved != nil && r.Moved.Digest == r.Digest:
m.Already++
continue
case r.Moved != nil:
m.Changed = append(m.Changed, r.Secret)
continue
case !r.Current:
m.Refused = append(m.Refused, fmt.Sprintf("%s is sealed to an earlier sealing key of %s, which no longer "+
"opens it: nothing can move it; it moves once it is made or accepted again", r.Secret, node))
continue
}
answer, err := ask(r, moveRowAsk(r))
if err != nil {
m.Unreached, m.Left = err, len(all)-n
return m
}
if answer.Refused != "" {
m.Refused = append(m.Refused, fmt.Sprintf("%s: %s", r.Secret, answer.Refused))
continue
}
if answer.Secret != r.Secret {
m.Refused = append(m.Refused, fmt.Sprintf("%s: the vault kept it as %s, not under the name the move "+
"gives it", r.Secret, answer.Secret))
continue
}
if err := record(r, answer.Generation); err != nil {
m.Refused = append(m.Refused, fmt.Sprintf("%s is in the vault, but that it moved could not be recorded: %v",
r.Secret, err))
continue
}
if answer.Unchanged {
m.Already++
} else {
m.Moved++
}
}
return m
}
// movedAsksAtOnce is how many rows of the record of asks one call to the vault carries (mesh-vault MovedAsksAtOnce).
const movedAsksAtOnce = 500
// moveAsks copies the controller's record of asks into the vault, in parts, and answers how many it copied.
func moveAsks(asks []inventory.SecretAsk, vault vaultAsker) (int, error) {
copied := 0
for start := 0; start < len(asks); start += movedAsksAtOnce {
end := min(start+movedAsksAtOnce, len(asks))
var rows []map[string]any
for _, a := range asks[start:end] {
row := map[string]any{"id": a.ID, "node": a.Node, "module": a.Module, "name": a.Name, "desk": a.Desk,
"asked_by": a.AskedBy, "opened_at": a.OpenedAt.UTC().Format(time.RFC3339Nano)}
if a.EndedAt != nil {
row["ended_at"], row["outcome"] = a.EndedAt.UTC().Format(time.RFC3339Nano), a.Outcome
}
rows = append(rows, row)
}
raw, err := vault("move-asks", map[string]any{"asks": rows})
if err != nil {
return copied, fmt.Errorf("the vault did not take the record of asks after %d rows: %w", copied, err)
}
var said struct {
Copied int `json:"copied"`
Already int `json:"already"`
}
_ = json.Unmarshal(raw, &said)
copied += said.Copied + said.Already
}
return copied, nil
}
// moveOnTheBus is the move's ask over the bus, signed with the mesh's key.
func moveOnTheBus(ctx context.Context, open *stores, conn *nats.Conn, by string) (moveAsk, error) {
ident, err := open.Identity(ctx)
if err != nil {
return nil, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked: %w", err)
}
return func(r inventory.SecretToMove, row link.MoveSecretAsk) (link.MoveSecretAnswer, error) {
return link.AskMoveSecret(ctx, conn, ident, r.Node, by, row, link.MoveSecretWithin)
}, nil
}
// secretMove is `secret move [<node>]`: the record of asks copied once, then every machine's rows — or one's — and
// what moved said per machine. Any row not moved fails the command, said by name.
func secretMove(ctx context.Context, args []string) error {
if len(args) > 1 || (len(args) == 1 && (args[0] == "" || strings.HasPrefix(args[0], "-"))) {
return errors.New(secretMoveUsage)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
by := actingAs()
if by == "" {
by = handOverBy()
}
move, err := inv.StartSecretMove(ctx, by)
if err != nil {
return err
}
var nodes []string
if len(args) == 1 {
if _, err := inv.NodeByName(ctx, args[0]); err != nil {
return fmt.Errorf("nothing was moved: %w", err)
}
nodes = args
} else {
all, err := inv.Nodes(ctx)
if err != nil {
return err
}
for _, n := range all {
nodes = append(nodes, n.Name)
}
}
fmt.Printf("the move of the secrets into the vault, started %s by %s\n", move.StartedAt.Local().Format("2006-01-02 15:04"),
move.StartedBy)
if move.AsksMovedAt == nil {
asks, err := inv.SecretAsks(ctx, time.Time{})
if err != nil {
return err
}
copied, err := moveAsks(asks, theVault(ctx))
if err != nil {
return fmt.Errorf("the record of asks was not copied, and no secret was moved: %w", err)
}
if err := inv.RecordAsksMoved(ctx); err != nil {
return err
}
fmt.Printf("the record of asks at a desk: %d rows copied into the vault as they are\n", copied)
}
failed := 0
err = onTheBus(func(conn *nats.Conn) error {
ask, err := moveOnTheBus(ctx, open, conn, "the move of the secrets, by "+by)
if err != nil {
return err
}
for _, node := range nodes {
m := moveNode(node, func(n string) ([]inventory.SecretToMove, error) { return inv.SecretsToMoveFrom(ctx, n) },
ask, func(r inventory.SecretToMove, g int) error { return inv.RecordSecretMoved(ctx, r, g) })
fmt.Print(m.String())
if m.Failed != nil || len(m.Refused) > 0 {
failed++
}
}
return nil
})
if err != nil {
return err
}
if failed > 0 {
return fmt.Errorf("on %d machine(s) a secret was not moved, said above by name", failed)
}
return nil
}
// moveHeard is the machines being moved in the background now, and when each was last moved, so a machine that reports
// often is not asked again before moveHeardEvery has passed.
var moveHeard = struct {
sync.Mutex
running map[string]bool
last map[string]time.Time
}{running: map[string]bool{}, last: map[string]time.Time{}}
// moveHeardEvery is how often a machine heard from is moved at most.
const moveHeardEvery = 10 * time.Minute
// moveWhenHeard moves what is left of a machine that was just heard from, once the move has started: a machine asleep
// when it started is moved when it next reports. What it did is said on the controller's output, and recorded for the
// next `secret move`.
func moveWhenHeard(ctx context.Context, open *stores, conn *nats.Conn, node string) {
if open == nil || conn == nil || node == "" {
return
}
moveHeard.Lock()
if moveHeard.running[node] || time.Since(moveHeard.last[node]) < moveHeardEvery {
moveHeard.Unlock()
return
}
moveHeard.running[node], moveHeard.last[node] = true, time.Now()
moveHeard.Unlock()
defer func() {
moveHeard.Lock()
delete(moveHeard.running, node)
moveHeard.Unlock()
}()
inv := open.inventory
move, started, err := inv.SecretMoveStarted(ctx)
if err != nil || !started {
return
}
rows, err := inv.SecretsToMoveFrom(ctx, node)
if err != nil {
return
}
left := false
for _, r := range rows {
left = left || (r.Moved == nil && r.Current)
}
if !left {
return
}
ask, err := moveOnTheBus(ctx, open, conn, "the move of the secrets, by "+move.StartedBy+", on hearing from "+node)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-controller: %s was heard from and its secrets not moved: %v\n", node, err)
return
}
m := moveNode(node, func(string) ([]inventory.SecretToMove, error) { return rows, nil }, ask,
func(r inventory.SecretToMove, g int) error { return inv.RecordSecretMoved(ctx, r, g) })
if m.done() {
fmt.Printf("mesh-controller: %s was heard from; the move of its secrets is done: %s", node, m.String())
return
}
fmt.Fprintf(os.Stderr, "mesh-controller: %s was heard from; the move of its secrets is not done: %s", node, m.String())
}
// moveHeardOn is the connection a machine heard from is moved over: the serving controller's bus.
func moveHeardOn() *nats.Conn {
if js := servingBus.Load(); js != nil {
return js.Conn()
}
return nil
}
+158
View File
@@ -0,0 +1,158 @@
package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func aRowToMove(secret string, current bool) inventory.SecretToMove {
return inventory.SecretToMove{Secret: secret, Node: "anchor", Module: "postgres", Name: strings.TrimPrefix(secret, "own/anchor/postgres/"),
Origin: "made", Generation: "g-1", MadeAt: time.Unix(1, 0), Sealed: "c2VhbGVk", SealedTo: "k", Current: current,
Digest: "sha256:" + secret}
}
// Each row sealed to a machine is asked of its engine once, as the store holds it; what the vault kept and read back
// the same is recorded; a row the vault held already counts as there; a row moved before is not asked again, and one
// changed in the store since it moved is said; a row sealed to a key the machine no longer has is said, not asked.
func TestAMachinesRowsAreMovedOnceEachAndWhatMovedIsSaid(t *testing.T) {
rows := []inventory.SecretToMove{
aRowToMove("own/anchor/postgres/superuser", true),
aRowToMove("own/anchor/postgres/replication", true),
aRowToMove("own/anchor/postgres/old-key", false),
aRowToMove("own/anchor/postgres/refused", true),
aRowToMove("own/anchor/postgres/moved-before", true),
aRowToMove("own/anchor/postgres/changed-since", true),
}
rows[4].Moved = &inventory.SecretMoved{Node: "anchor", Generation: 1, Digest: rows[4].Digest}
rows[5].Moved = &inventory.SecretMoved{Node: "anchor", Generation: 1, Digest: "sha256:something before"}
var asked []link.MoveSecretAsk
ask := func(r inventory.SecretToMove, row link.MoveSecretAsk) (link.MoveSecretAnswer, error) {
asked = append(asked, row)
switch r.Secret {
case "own/anchor/postgres/refused":
return link.MoveSecretAnswer{Refused: "the vault already holds another value; nothing was moved"}, nil
case "own/anchor/postgres/replication":
return link.MoveSecretAnswer{Secret: r.Secret, Generation: 3, Unchanged: true}, nil
}
return link.MoveSecretAnswer{Secret: r.Secret, Generation: 1}, nil
}
recorded := map[string]int{}
record := func(r inventory.SecretToMove, generation int) error { recorded[r.Secret] = generation; return nil }
m := moveNode("anchor", func(string) ([]inventory.SecretToMove, error) { return rows, nil }, ask, record)
if m.Moved != 1 || m.Already != 2 || len(m.Refused) != 2 || len(m.Changed) != 1 || m.Unreached != nil {
t.Fatalf("the move said %+v", m)
}
if len(asked) != 3 || asked[0].Module != "postgres" || asked[0].Name != "superuser" || asked[0].Sealed != "c2VhbGVk" ||
asked[0].Origin != "made" || asked[0].Generation != "g-1" || asked[0].SealedTo != "k" {
t.Fatalf("asked %+v", asked)
}
if recorded["own/anchor/postgres/superuser"] != 1 || recorded["own/anchor/postgres/replication"] != 3 || len(recorded) != 2 {
t.Errorf("recorded %v", recorded)
}
said := m.String()
for _, want := range []string{"anchor: 1 moved into the vault and read back the same, 2 there already",
"own/anchor/postgres/old-key", "earlier sealing key", "own/anchor/postgres/refused", "already holds another value",
"own/anchor/postgres/changed-since", "changed in the controller's store after it moved"} {
if !strings.Contains(said, want) {
t.Errorf("the move did not say %q:\n%s", want, said)
}
}
if strings.Contains(said, "c2VhbGVk") {
t.Error("the move said a sealed value")
}
}
// A machine that does not answer is left for when it is next heard from: nothing more is asked of it now, and that
// is said; a pair is asked with both its ends and never as an own secret.
func TestAMachineNotAnsweringIsMovedWhenNextHeardFrom(t *testing.T) {
pair := inventory.SecretToMove{Secret: "pair/anchor/database/shanks/gitea", Node: "shanks", Provider: "anchor",
Provision: "database", Consumer: "shanks", ConsumerModule: "gitea", Origin: "accepted", Sealed: "x",
SealedTo: "k", Current: true}
rows := []inventory.SecretToMove{pair, pair}
calls := 0
var asked link.MoveSecretAsk
ask := func(_ inventory.SecretToMove, row link.MoveSecretAsk) (link.MoveSecretAnswer, error) {
calls++
asked = row
return link.MoveSecretAnswer{}, errors.New("nothing on shanks answers a move")
}
m := moveNode("shanks", func(string) ([]inventory.SecretToMove, error) { return rows, nil }, ask,
func(inventory.SecretToMove, int) error { t.Fatal("recorded"); return nil })
if calls != 1 || m.Unreached == nil || !strings.Contains(m.String(), "moved when it is next heard from") {
t.Fatalf("calls %d, %+v: %s", calls, m, m.String())
}
if asked.Module != "" || asked.Provider != "anchor" || asked.Consumer != "shanks" || asked.ConsumerModule != "gitea" {
t.Errorf("the pair was asked as %+v", asked)
}
failing := moveNode("x", func(string) ([]inventory.SecretToMove, error) { return nil, errors.New("no store") }, nil, nil)
if failing.Failed == nil {
t.Error("a store that cannot be read was not said")
}
}
// The record of asks is copied into the vault in parts, as it is: every row's fields, nothing more.
func TestTheRecordOfAsksIsCopiedInParts(t *testing.T) {
ended := time.Unix(100, 0).UTC()
var asks []inventory.SecretAsk
for n := 1; n <= movedAsksAtOnce+3; n++ {
asks = append(asks, inventory.SecretAsk{ID: int64(n), Node: "anchor", Module: "telegram", Name: "token",
AskedBy: "g14/claude-code", Desk: "laptop", OpenedAt: time.Unix(int64(n), 0).UTC(), EndedAt: &ended,
Outcome: "given"})
}
var parts []int
vault := func(verb string, args map[string]any) (json.RawMessage, error) {
if verb != "move-asks" {
t.Fatalf("asked %s", verb)
}
rows := args["asks"].([]map[string]any)
parts = append(parts, len(rows))
if rows[0]["desk"] != "laptop" || rows[0]["asked_by"] != "g14/claude-code" || rows[0]["outcome"] != "given" ||
rows[0]["id"] == nil || rows[0]["opened_at"] == nil || rows[0]["ended_at"] == nil {
t.Fatalf("a row went as %v", rows[0])
}
return json.Marshal(map[string]int{"copied": len(rows)})
}
copied, err := moveAsks(asks, vault)
if err != nil || copied != movedAsksAtOnce+3 || len(parts) != 2 || parts[0] != movedAsksAtOnce {
t.Fatalf("copied %d in %v, %v", copied, parts, err)
}
if _, err := moveAsks(asks, func(string, map[string]any) (json.RawMessage, error) {
return nil, errors.New("the vault is away")
}); err == nil {
t.Error("a copy the vault refused was not said")
}
}
// The verb runs the move's one line through the seat and nothing else of the secret commands.
func TestTheMoveIsAVerbOfItsOwn(t *testing.T) {
argv, err := argvFor("secret-move", map[string]any{})
if err != nil || strings.Join(argv, " ") != "secret move" {
t.Fatalf("argv %v, %v", argv, err)
}
argv, err = argvFor("secret-move", map[string]any{"node": "anchor"})
if err != nil || strings.Join(argv, " ") != "secret move anchor" {
t.Fatalf("argv %v, %v", argv, err)
}
if err := terminalOnly([]string{"secret", "move", "anchor"}); err != nil {
t.Fatalf("the move's line was refused through its verb: %v", err)
}
for _, line := range [][]string{{"secret", "move", "--from", "x"}, {"secret", "move", "a", "b"}} {
if err := terminalOnly(line); err == nil {
t.Errorf("%q ran through a verb", line)
}
}
found := false
for _, v := range catalogue.ControllerVerbs {
found = found || v.Name == "secret-move"
}
if !found {
t.Fatal("the controller's seat does not declare the secret-move verb")
}
}
+17
View File
@@ -77,6 +77,9 @@ type sendable struct {
// unplaced is every contribution its holder's template could not render, naming its module and
// why (novox/hq ADR 0255): left out of this declaration, for push and plan to say, never on the wire.
unplaced []string
// handedOnly is each file left out because only the provisioner's client fills it and the module is not handed
// here (a provider's consumers file, ADR 0373).
handedOnly []string
// Builds is the build of each module this declaration carries — module to the commit its build
// was made from — recorded with the send and never on the wire (novox/hq issue 259, ADR 0221).
// Composed only on the send path; nil records that it is not known.
@@ -84,6 +87,16 @@ type sendable struct {
// Bindings is where each consumer of a provision that keeps its data is bound in this declaration
// (novox/hq ADR 0232), recorded with the send and never on the wire. Composed only on the send path.
Bindings []inventory.Binding
// Filled is the path of every file this declaration hands to the provisioner, and Provisioner the account the
// client that fills them runs as (novox/hq ADR 0349 §4, §5, design 57 P6): on the wire as `filled_files` and
// `provisioner_account`, both absent when nothing is handed, so every other declaration is byte for byte what
// it was. plans is each handed module's plan, kept with the send for the assignments mirror; never on the wire.
Filled []string
Provisioner string
plans map[string]catalogue.Plan
// requiredBy is, for each handed module only pulled in by another, the module that requires it (issue 765): kept
// beside its plan, so the assignments mirror gives the plan a value of its own. Never on the wire.
requiredBy map[string]string
}
// adoptionEnvelope is what an adopted node is told about its mode. Taken is every module taken on
@@ -113,6 +126,10 @@ func (s sendable) Body() ([]byte, error) {
if s.Generation > 0 {
envelope["generation"] = s.Generation
}
if len(s.Filled) > 0 {
envelope["filled_files"] = s.Filled
envelope["provisioner_account"] = s.Provisioner
}
// An empty declaration is deliberate here — the node owns nothing the mesh put there
// (novox/hq issue 127) — and the host refuses an empty body unless it is told the emptiness
// is meant, so a truncated or mis-composed body is never mistaken for "own nothing".
@@ -0,0 +1,125 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A send keeps, beside each handed module's plan, what requires a module only pulled in (issue 765), so the
// assignments mirror gives that plan a value of its own; an assigned module's plan is kept as the bytes it was.
func TestASendKeepsWhatRequiresAPulledInModule(t *testing.T) {
plan := catalogue.Plan{Templates: catalogue.PlanTemplates{Source: "http://store:5000/v2/notify/files",
Digest: "sha256:" + strings.Repeat("d", 64)},
Files: []catalogue.PlanFile{{ID: "rc", Path: "/home/op/rc", Template: "rc", OwnerAtRuntime: "owned"}}}
r := sentRecordOf(t.Context(), sendable{plans: map[string]catalogue.Plan{"dunst": plan, "notify": plan},
requiredBy: map[string]string{"notify": "desk"}})
kept := string(r.plans)
want := `{"dunst":{"templates":{"source":"http://store:5000/v2/notify/files","digest":"sha256:` +
strings.Repeat("d", 64) + `"},"files":[{"id":"rc","path":"/home/op/rc","template":"rc","owner-at-runtime":"owned"}]},` +
`"notify":{"templates":{"source":"http://store:5000/v2/notify/files","digest":"sha256:` + strings.Repeat("d", 64) +
`"},"files":[{"id":"rc","path":"/home/op/rc","template":"rc","owner-at-runtime":"owned"}],"required-by":"desk"}}`
if kept != want {
t.Fatalf("the send keeps\n%s\nwant\n%s", kept, want)
}
}
// **A module only pulled in by another is handed on a machine that can take it** (issue 765): on a contract-11
// node running a client, a moved module that nothing assigned there, required by one that is, has its files and its
// own secret handed like an assigned one's, no secret of it made or sealed by the controller, and its plan mirrored
// under its own key, saying what requires it. A module not moved, pulled in the same way, is sent as before.
func TestAModulePulledInIsHandedOnAMachineThatCanTakeItAndMirrored(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
client := aClientModule()
moved, kept := aBot("moved-bot", true), aBot("kept-bot", false)
moved.Provides = []catalogue.Offer{{Name: "moved-bot-api"}}
kept.Provides = []catalogue.Offer{{Name: "kept-bot-api"}}
desk := catalogue.Manifest{Module: "desk", Version: "1", Requires: []string{"moved-bot-api", "kept-bot-api"}}
for _, m := range []catalogue.Manifest{client, moved, kept, desk} {
register(t, open, m)
}
for _, m := range []string{client.Module, desk.Module} {
if _, err := assign(ctx, open, "laptop", m); err != nil {
t.Fatal(err)
}
}
if err := issueProvisionerClient(ctx, inv, client, "laptop", broker.Broker{Fingerprint: "ab"}, "10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "laptop", Contract: link.HandedContract,
SaidAt: time.Now()}); err != nil {
t.Fatal(err)
}
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if plan.RequiredBy("moved-bot") != "desk" || plan.RequiredBy("kept-bot") != "desk" {
t.Fatalf("the bots are on laptop because %v", plan.Because)
}
declared, err := declarationFor(ctx, open, "laptop", plan, settings)
if err != nil {
t.Fatal(err)
}
var movedToken, keptToken, movedConf map[string]any
for _, r := range declared.Resources {
switch r["id"] {
case "moved-bot.needs-token":
movedToken = r
case "kept-bot.needs-token":
keptToken = r
case "moved-bot.conf":
movedConf = r
}
}
if movedToken == nil || movedToken["sealed"] != nil || movedToken["owner-at-runtime"] != "owned" {
t.Fatalf("the pulled-in moved module's secret file is sent as %v: placed, with no value", movedToken)
}
if movedConf == nil || movedConf["content"] != nil || movedConf["owner-at-runtime"] != "owned" {
t.Fatalf("the pulled-in moved module's templated file is sent as %v: placed, with no text", movedConf)
}
if keptToken == nil || keptToken["sealed"] == nil {
t.Fatalf("the pulled-in module not moved has its secret sent as %v: sealed, as before", keptToken)
}
if _, held, err := inv.ModuleSecretIfIssued(ctx, "laptop", "moved-bot", "token"); err != nil || held {
t.Fatalf("the send made the pulled-in handed module's secret in the controller's store (held %v, %v)", held, err)
}
if _, has := declared.plans["kept-bot"]; has || len(declared.plans) != 1 {
t.Fatalf("the send hands %v", declared.plans)
}
record, err := inv.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := inv.RecordSentWith(ctx, record.ID, "first", nil, 0, 0, inventory.Send{Digest: "first",
Provisioning: sentRecordOf(ctx, declared).plans}); err != nil {
t.Fatal(err)
}
inv.SetProvisioningPlans(inv.SentProvisioningPlans)
mirror, err := inv.AssignmentsMirror(ctx)
if err != nil {
t.Fatal(err)
}
var value inventory.AssignmentMirrored
if err := json.Unmarshal(mirror["laptop.moved-bot"], &value); err != nil {
t.Fatalf("no value for the pulled-in module: %s (%v)", mirror["laptop.moved-bot"], err)
}
if value.RequiredBy != "desk" || !value.Assigned.IsZero() || value.Provisioning == nil ||
len(value.Provisioning.Files) != 1 || value.Provisioning.Files[0].Path != "/etc/moved-bot/bot.conf" ||
len(value.Provisioning.Secrets) != 1 || value.Provisioning.Secrets[0].Name != "token" {
t.Fatalf("the pulled-in module is mirrored as %s", mirror["laptop.moved-bot"])
}
if _, has := mirror["laptop.kept-bot"]; has {
t.Fatalf("a pulled-in module not handed is mirrored: %s", mirror["laptop.kept-bot"])
}
}
+372
View File
@@ -0,0 +1,372 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **`settings` answers by forwarding to `config`** (novox/hq ADR 0348 §1, design 57 K5, issue 664). Once the move of
// config has run, a module in the config form keeps its config values in mesh-config (inventory.ReadsConfig), so the
// settings verb reads and writes them there: `show` lists what `config list` lists, `set` sets each value as the
// module's own on the node and `clear` clears them, and `propose` proposes each through mesh-config. Placement stays
// the controller's and is read and written beside the assignment as before (ADR 0348 §11). A module still in the
// settings form, and every module before the move, keeps today's verb unchanged.
// configAsk asks mesh-config's verb in the controller's own name, so mesh-config takes a value as the controller
// carries it (ADR 0348 §8); a test replaces it.
var configAsk = func(ctx context.Context, args map[string]any) (json.RawMessage, error) {
var result json.RawMessage
err := onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatToolAs(ctx, conn, broker.ControllerName, "mesh-config", "config", args, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
result = answer.Result
return nil
})
return result, err
}
// configListed is what `config list` answers, as the settings verb reads it.
type configListed struct {
Resolved []resolvedEntry `json:"resolved"`
Set []struct {
Level configLevel `json:"level"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
By string `json:"by"`
} `json:"set"`
}
func listConfig(ctx context.Context, module, node string) (configListed, error) {
args := map[string]any{"action": "list", "module": module}
if node != "" {
args["node"] = node
}
raw, err := configAsk(ctx, args)
if err != nil {
return configListed{}, fmt.Errorf("mesh-config does not answer %s's config values: %w", module, err)
}
var l configListed
if err := json.Unmarshal(raw, &l); err != nil {
return configListed{}, fmt.Errorf("mesh-config's listing cannot be read: %w", err)
}
return l, nil
}
// ownValues is the module's own values on the node, as mesh-config holds them; none for the whole mesh, which has no
// module value (ADR 0348 §3).
func (l configListed) ownValues(module, node string) map[string]any {
out := map[string]any{}
if node == "" {
return out
}
for _, s := range l.Set {
if s.Level.Level == "module" && s.Level.Node == node && s.Level.Module == module {
var v any
if json.Unmarshal(s.Value, &v) == nil {
out[s.Key] = v
}
}
}
return out
}
// forwardSettings answers a settings action for a module whose config values are mesh-config's, and says whether it
// did: false leaves the action to today's verb.
func forwardSettings(ctx context.Context, inv *inventory.Inventory, action, module, node string, positionals []string,
replace bool) (bool, error) {
if module == "" {
return false, nil
}
reads, err := inv.ReadsConfig(ctx, module)
if err != nil || !reads {
return false, err
}
where := whereOf(node)
switch action {
case "show":
listed, err := listConfig(ctx, module, node)
if err != nil {
return true, err
}
layer, _, err := inv.Layer(ctx, node, module)
if err != nil {
return true, err
}
if layer == nil {
layer = map[string]any{}
}
for k, v := range listed.ownValues(module, node) {
layer[k] = v
}
if len(layer) == 0 {
fmt.Printf("%s on %s: no layer — the module's definition says\n", module, where)
} else {
shown, err := json.MarshalIndent(layer, "", " ")
if err != nil {
return true, err
}
fmt.Println(string(shown))
}
fmt.Printf(" its config values are mesh-config's since the move of config (issue 664): mesh-config.config\n")
fmt.Printf("%s on %s, every value and where it comes from:\n", module, where)
for _, r := range listed.Resolved {
value := string(r.Value)
if value == "" {
value = "unset"
}
on := ""
if node == "" {
on = " on " + r.Node
}
fmt.Printf(" %s%s = %s (%s)\n", r.Key, on, value, r.Source)
}
return true, nil
case "set":
var raw []byte
if strings.HasPrefix(strings.TrimSpace(positionals[1]), "{") {
raw = []byte(positionals[1])
} else {
return true, errors.New("the settings of a module whose config values are mesh-config's are given as " +
"{…}, never a file; or set each with mesh-config.config")
}
var values map[string]any
if err := json.Unmarshal(raw, &values); err != nil {
return true, fmt.Errorf("%s is not a settings object: %w", positionals[1], err)
}
return true, forwardSet(ctx, inv, module, node, values, replace)
case "clear":
listed, err := listConfig(ctx, module, node)
if err != nil {
return true, err
}
own := listed.ownValues(module, node)
layer, _, err := inv.Layer(ctx, node, module)
if err != nil {
return true, err
}
before := map[string]any{}
for k, v := range layer {
before[k] = v
}
for k, v := range own {
before[k] = v
}
if err := refuseTerminalSettingsThroughAVerb(ctx, inv, before, nil, module, where); err != nil {
return true, err
}
if err := refuseTrustedThroughAVerb(ctx, inv, module, where, movedKeys(own)); err != nil {
return true, err
}
for _, k := range movedKeys(own) {
if _, err := configAsk(ctx, map[string]any{"action": "clear", "module": module, "node": node, "key": k,
"why": "settings clear " + module + " on " + where, "by": setByWords()}); err != nil {
return true, fmt.Errorf("%s of %s on %s was not cleared: %w", k, module, where, err)
}
}
if len(layer) > 0 {
if err := inv.ClearSettingsBy(ctx, node, module, setByWords()); err != nil {
return true, err
}
}
fmt.Printf("%s on %s is back to what the module says\n", module, where)
return true, nil
}
return false, nil
}
// forwardSet sets a layer of a module whose config values are mesh-config's: its placement beside the assignment
// as before, each config value as the module's own on the node, and each it no longer names cleared.
func forwardSet(ctx context.Context, inv *inventory.Inventory, module, node string, values map[string]any,
replace bool) error {
where := whereOf(node)
config, placement := catalogue.SplitPlacement(values)
listed, err := listConfig(ctx, module, node)
if err != nil {
return err
}
own := listed.ownValues(module, node)
layer, _, err := inv.Layer(ctx, node, module)
if err != nil {
return err
}
before := map[string]any{}
for k, v := range layer {
before[k] = v
}
for k, v := range own {
before[k] = v
}
if err := refuseTerminalSettingsThroughAVerb(ctx, inv, before, values, module, where); err != nil {
return err
}
added, changed, removed := settingsChange(before, values)
if len(removed) > 0 && !replace {
return fmt.Errorf("%s on %s: this layer would no longer set %s. A layer is replaced whole; read it with "+
"`settings show %s%s` and include what should stay, or add --replace if the removal is meant (novox/hq "+
"ADR 0217). Nothing was changed", module, where, strings.Join(removed, ", "), module, nodeFlag(node))
}
var toSet, toClear []string
for _, k := range movedKeys(config) {
if was, had := own[k]; !had || !sameJSON(valueJSON(was), config[k]) {
toSet = append(toSet, k)
}
}
for _, k := range movedKeys(own) {
if _, kept := config[k]; !kept {
toClear = append(toClear, k)
}
}
if node == "" && len(toSet)+len(toClear) > 0 {
return fmt.Errorf("%s has no own value for the whole mesh (novox/hq ADR 0348 §3): name the node, or set mesh "+
"config for the predefined key a value falls back to with mesh-config.config. Nothing was changed", module)
}
if err := refuseTrustedThroughAVerb(ctx, inv, module, where, append(append([]string{}, toSet...), toClear...)); err != nil {
return err
}
if placement == nil {
placement = map[string]any{}
}
if _, placed := catalogue.SplitPlacement(layer); !sameLayer(placed, placement) {
if err := inv.SetSettingsBy(ctx, node, module, placement, setByWords()); err != nil {
return err
}
}
for _, k := range toSet {
if _, err := configAsk(ctx, map[string]any{"action": "set", "module": module, "node": node, "key": k,
"value": config[k], "why": "settings set " + module + " on " + where, "by": setByWords()}); err != nil {
return fmt.Errorf("%s of %s on %s was not set: %w", k, module, where, err)
}
}
for _, k := range toClear {
if _, err := configAsk(ctx, map[string]any{"action": "clear", "module": module, "node": node, "key": k,
"why": "settings set " + module + " on " + where + " --replace", "by": setByWords()}); err != nil {
return fmt.Errorf("%s of %s on %s was not cleared: %w", k, module, where, err)
}
}
fmt.Printf("%s on %s:\n", module, where)
if len(added)+len(changed)+len(removed) == 0 {
fmt.Println(" nothing changed")
}
for _, p := range added {
fmt.Printf(" + %s\n", p)
}
for _, p := range changed {
fmt.Printf(" ~ %s\n", p)
}
for _, p := range removed {
fmt.Printf(" - %s\n", p)
}
fmt.Println(" its config values are mesh-config's (issue 664): each was set there, with its history; the module " +
"takes them when its files are filled again")
return nil
}
func valueJSON(v any) json.RawMessage {
raw, _ := json.Marshal(v)
return raw
}
// refuseTrustedThroughAVerb refuses, through a verb, a trusted config value (ADR 0348 §8): mesh-config takes the
// controller's word for one as given at its terminal, and whoever may call a verb includes agents.
func refuseTrustedThroughAVerb(ctx context.Context, inv *inventory.Inventory, module, where string, keys []string) error {
verb, through := throughAVerb()
if !through || len(keys) == 0 {
return nil
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return fmt.Errorf("which config values of %s are trusted cannot be read, so nothing was changed: %w", module, err)
}
for _, k := range keys {
if shelf[module].Settings[k].Trusted {
return fmt.Errorf("%s of %s on %s is trusted (novox/hq ADR 0348 §8): it is set at the controller's terminal "+
"or on the operator's Approve, never through a verb alone (this line came through %q). Propose it with "+
"mesh-config.config, action propose. Nothing was changed", k, module, where, verb)
}
}
return nil
}
// forwardPropose proposes, through mesh-config, each config value of a layer proposed for a module whose config
// values are mesh-config's (ADR 0348 §8): mesh-config asks the operator under its own name, one ask per value. A
// proposal naming placement too is refused, so the operator is never asked half of it; placement alone is proposed
// as before. Says whether it answered.
func forwardPropose(ctx context.Context, inv *inventory.Inventory, module, node, valuesArg string, clear bool) (bool, error) {
reads, err := inv.ReadsConfig(ctx, module)
if err != nil || !reads {
return false, err
}
where := whereOf(node)
if node == "" {
return true, fmt.Errorf("%s has no own value for the whole mesh (novox/hq ADR 0348 §3): name the node, or "+
"propose mesh config for the predefined key a value falls back to with mesh-config.config. Nothing was "+
"proposed", module)
}
proposals := map[string]any{}
var clears []string
if clear {
listed, err := listConfig(ctx, module, node)
if err != nil {
return true, err
}
layer, _, err := inv.Layer(ctx, node, module)
if err != nil {
return true, err
}
if len(layer) > 0 {
return true, fmt.Errorf("%s on %s keeps placement beside its assignment (%s): propose clearing it apart "+
"from its config values. Nothing was proposed", module, where, strings.Join(movedKeys(layer), ", "))
}
clears = movedKeys(listed.ownValues(module, node))
} else {
var values map[string]any
if err := json.Unmarshal([]byte(valuesArg), &values); err != nil {
return true, fmt.Errorf("%s is not a settings object: %w", valuesArg, err)
}
config, placement := catalogue.SplitPlacement(values)
if len(placement) > 0 {
if len(config) > 0 {
return true, fmt.Errorf("%s names placement (%s) and config values: propose them apart, the config "+
"values through mesh-config. Nothing was proposed", module, strings.Join(movedKeys(placement), ", "))
}
return false, nil
}
proposals = config
}
for _, k := range movedKeys(proposals) {
answer, err := configAsk(ctx, map[string]any{"action": "propose", "module": module, "node": node, "key": k,
"value": proposals[k], "why": "settings propose " + module + " on " + where})
if err != nil {
return true, fmt.Errorf("%s of %s on %s was not proposed: %w", k, module, where, err)
}
fmt.Printf("%s of %s on %s, proposed through mesh-config: %s\n", k, module, where, answer)
}
for _, k := range clears {
answer, err := configAsk(ctx, map[string]any{"action": "propose", "module": module, "node": node, "key": k,
"clear": true, "why": "settings propose " + module + " on " + where + " --clear"})
if err != nil {
return true, fmt.Errorf("clearing %s of %s on %s was not proposed: %w", k, module, where, err)
}
fmt.Printf("clearing %s of %s on %s, proposed through mesh-config: %s\n", k, module, where, answer)
}
return true, nil
}
+5
View File
@@ -224,5 +224,10 @@ func (l nudgingListener) Heard(ctx context.Context, report link.Report) (bool, e
// nothing it caused (novox/hq ADR 0228).
go replaceGiven(context.WithoutCancel(ctx), l.open, report)
}
// And, once the move of the secrets into the vault has started, what is left of this machine's (design 57's V3,
// issue 676): a machine asleep when it started is moved when it is next heard from. Off the report's path.
if err == nil && l.open != nil && report.Superseded == "" && report.Rekey == nil && report.Node != "" {
go moveWhenHeard(context.WithoutCancel(ctx), l.open, moveHeardOn(), report.Node)
}
return news, err
}
+8
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"fmt"
"os"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -124,6 +125,13 @@ func openInventory(ctx context.Context) (*inventory.Inventory, error) {
if aliases, err := inv.Aliases(ctx); err == nil {
catalogue.UseAliases(aliases)
}
// And the list of predefined keys the mesh publishes (novox/hq ADR 0348 §7), which every manifest read again
// is judged against. A store a migration has not reached yet keeps none, and nothing names a key then.
if err := loadPredefined(ctx, inv); err != nil {
fmt.Fprintf(os.Stderr, "the list of predefined keys kept cannot be read: %v\n", err)
}
// And where config values are read once the move of config has run (issue 664): mesh-config_resolved.
inv.ReadsConfigFrom(readConfigValues)
return inv, nil
}
+241
View File
@@ -0,0 +1,241 @@
package main
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
)
// A file's text kept as a template file in its module's repository (novox/hq ADR 0349 §2, design 57 package T1,
// issue 654): `module check` and the merge gate read it from the tree beside the manifest, so a change to a
// template alone is judged as a change to its module.
const templatedNotifier = `{"module":"notifier","version":"1",
"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]},
"resources":[{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644","template":"files/notifier.conf"}]}`
// inlineNotifier is the same module as the mesh holds it: built, its text inline.
const inlineNotifier = `{"module":"notifier","version":"1",
"resources":[{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644","content":"volume = 3\n"}]}`
// aTemplatedModule writes a module's directory: its manifest and its files/.
func aTemplatedModule(t *testing.T, dir, manifest string, files map[string]string) string {
t.Helper()
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, "module.json")
if err := os.WriteFile(path, []byte(manifest), 0o644); err != nil {
t.Fatal(err)
}
for name, text := range files {
full := filepath.Join(dir, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
return path
}
func TestModuleCheckReadsTemplatesFromTheTreeBesideTheManifest(t *testing.T) {
good := aTemplatedModule(t, t.TempDir(), templatedNotifier, map[string]string{"files/notifier.conf": "volume = 3\n"})
var out bytes.Buffer
if err := moduleCheck([]string{good}, &out); err != nil || !strings.Contains(out.String(), "notifier: ok") {
t.Fatalf("a module with its text in files/ was refused: %v\n%s", err, out.String())
}
// A change to the template alone, to text the controller refuses inline, is refused the same.
bad := aTemplatedModule(t, t.TempDir(), templatedNotifier,
map[string]string{"files/notifier.conf": "volume = 3\nstore = ${dir:nowhere}\n"})
out.Reset()
if err := moduleCheck([]string{bad}, &out); err == nil || !strings.Contains(out.String(), "nowhere") {
t.Fatalf("a template whose text is refused inline passed:\n%s", out.String())
}
// A symlink under files/ that no template names: the build's packing refuses it, so the check does too.
linked := t.TempDir()
aTemplatedModule(t, linked, templatedNotifier, map[string]string{"files/notifier.conf": "volume = 3\n"})
if err := os.Symlink("/etc/hostname", filepath.Join(linked, "files", "elsewhere")); err != nil {
t.Fatal(err)
}
out.Reset()
if err := moduleCheck([]string{filepath.Join(linked, "module.json")}, &out); err == nil ||
!strings.Contains(out.String(), "files/elsewhere is a symlink") {
t.Fatalf("a symlink under files/ passed the check:\n%s", out.String())
}
// And a template the tree lacks, by name.
missing := aTemplatedModule(t, t.TempDir(), templatedNotifier, nil)
out.Reset()
if err := moduleCheck([]string{missing}, &out); err == nil || !strings.Contains(out.String(), "files/notifier.conf") {
t.Fatalf("a missing template was not refused by name:\n%s", out.String())
}
}
// templatedMesh is catalogueMesh with the notifier on the laptop, registered as built: its text inline.
func templatedMesh(t *testing.T) (snapshot.Facts, map[string]string) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
manifests := map[string]string{
"lemurs": `{"module":"lemurs","version":"1","capabilities":["systemd"],
"resources":[{"id":"service","type":"service","unit":"lemurs.service","state":"running","boot":"enabled"}]}`,
"notifier": inlineNotifier,
}
for name, raw := range manifests {
m, err := catalogue.ParseManifest([]byte(raw))
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if err := open.inventory.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog",
Path: "modules/" + name, BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", name); err != nil {
t.Fatalf("assign %s: %v", name, err)
}
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
return f, manifests
}
// aTemplatedTree is the catalogue's checkout with the notifier's text in files/.
func aTemplatedTree(t *testing.T, manifests map[string]string, text string, has bool) string {
t.Helper()
tree := aTree(t, withEdit(manifests, "notifier", ""))
files := map[string]string{}
if has {
files["files/notifier.conf"] = text
}
aTemplatedModule(t, filepath.Join(tree, "modules", "notifier"), templatedNotifier, files)
return tree
}
const notifierTemplate = "modules/notifier/files/notifier.conf"
func TestTheMergeGateJudgesAChangeToATemplateAlone(t *testing.T) {
f, manifests := templatedMesh(t)
// The module moved to its template, the same text: nothing changes on any machine.
v := gateJudged(t, f, aTemplatedTree(t, manifests, "volume = 3\n", true), "modules/notifier/module.json",
notifierTemplate)
if v.Verdict == "fail" || v.Modules["notifier"] != "" {
t.Fatalf("the module moved to its template with the same text reads as a change (%v):\n%s", v.Modules, v.Report())
}
// The template alone changed: its module changed, and composes.
v = gateJudged(t, f, aTemplatedTree(t, manifests, "volume = 4\n", true), notifierTemplate)
if v.Verdict == "fail" || v.Modules["notifier"] != "changed" {
t.Fatalf("a change to the template alone is not a change to its module (%v):\n%s", v.Modules, v.Report())
}
if v.Width == nil || strings.Join(v.Width.Modules, ",") != "notifier" {
t.Fatalf("a change to the template rebuilds %+v, not its module", v.Width)
}
// The template alone changed to text the controller refuses: the pull request fails, naming the module.
v = gateJudged(t, f, aTemplatedTree(t, manifests, "store = ${dir:nowhere}\n", true), notifierTemplate)
if v.Verdict != "fail" || !strings.Contains(v.Report(), "notifier") {
t.Fatalf("a template the controller refuses passed the gate:\n%s", v.Report())
}
// A symlinked files/ fails: nothing is read from outside the tree.
tree := aTree(t, withEdit(manifests, "notifier", ""))
elsewhere := t.TempDir()
if err := os.WriteFile(filepath.Join(elsewhere, "notifier.conf"), []byte("volume = 3\n"), 0o644); err != nil {
t.Fatal(err)
}
aTemplatedModule(t, filepath.Join(tree, "modules", "notifier"), templatedNotifier, nil)
if err := os.Symlink(elsewhere, filepath.Join(tree, "modules", "notifier", "files")); err != nil {
t.Fatal(err)
}
v = gateJudged(t, f, tree, notifierTemplate)
if v.Verdict != "fail" || !strings.Contains(v.Report(), "files/ is a symlink") {
t.Fatalf("a symlinked files/ passed the gate:\n%s", v.Report())
}
// And the template removed from the tree fails, naming it.
v = gateJudged(t, f, aTemplatedTree(t, manifests, "", false), notifierTemplate)
if v.Verdict != "fail" || !strings.Contains(v.Report(), "files/notifier.conf") {
t.Fatalf("a missing template passed the gate:\n%s", v.Report())
}
}
// What the gate composes for the module in its template form is what the mesh composes for it inline.
func TestTheMergeGateResolvesATemplatedModuleToItsInlineTwin(t *testing.T) {
dir := t.TempDir()
path := aTemplatedModule(t, dir, templatedNotifier, map[string]string{"files/notifier.conf": "volume = 3\n"})
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
m, err = catalogue.WithTemplatesFrom(m, dir)
if err != nil {
t.Fatal(err)
}
resolved, err := standInBuild(m)
if err != nil {
t.Fatal(err)
}
inline, err := catalogue.ParseManifest([]byte(inlineNotifier))
if err != nil {
t.Fatal(err)
}
if builtAlike(resolved) != builtAlike(inline) {
got, _ := json.Marshal(resolved)
t.Fatalf("the templated module resolves to %s, not its inline twin", got)
}
}
// A module whose files/ is unpacked onto the machine names its own templates directory (issue 731): `module check`
// reads its templates from there and says of it exactly what it says of the same module with its text inline.
func TestModuleCheckReadsTemplatesFromTheTemplatesDirectoryTheManifestNames(t *testing.T) {
const inline = `{"module":"notifier","version":"1",
"build":{"artifacts":[{"name":"scripts","kind":"archive","from":"files"}]},
"resources":[{"id":"scripts","type":"archive","path":"/usr/local/lib/notifier","artifact":"scripts"},
{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644","content":"volume = 3\n"}]}`
const templated = `{"module":"notifier","version":"1","templates":"templates",
"build":{"artifacts":[{"name":"scripts","kind":"archive","from":"files"},
{"name":"templates","kind":"archive","from":"templates"}]},
"resources":[{"id":"scripts","type":"archive","path":"/usr/local/lib/notifier","artifact":"scripts"},
{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644","template":"templates/notifier.conf"}]}`
scripts := map[string]string{"files/bin/notify": "#!/bin/sh\n"}
var want bytes.Buffer
if err := moduleCheck([]string{aTemplatedModule(t, t.TempDir(), inline, scripts)}, &want); err != nil {
t.Fatalf("the inline module was refused: %v\n%s", err, want.String())
}
with := map[string]string{"files/bin/notify": "#!/bin/sh\n", "templates/notifier.conf": "volume = 3\n"}
var got bytes.Buffer
if err := moduleCheck([]string{aTemplatedModule(t, t.TempDir(), templated, with)}, &got); err != nil {
t.Fatalf("a module with its text in templates/ was refused: %v\n%s", err, got.String())
}
if got.String() != want.String() {
t.Fatalf("module check says otherwise of the templated module:\ninline: %s\ntemplated: %s", want.String(),
got.String())
}
// Its text in files/ instead, where the machine unpacks it, is not where its manifest says its templates are.
misplaced := strings.Replace(templated, `"templates/notifier.conf"`, `"files/notifier.conf"`, 1)
got.Reset()
if err := moduleCheck([]string{aTemplatedModule(t, t.TempDir(), misplaced,
map[string]string{"files/notifier.conf": "volume = 3\n"})}, &got); err == nil ||
!strings.Contains(got.String(), "files/notifier.conf") {
t.Fatalf("a template outside the templates directory the manifest names passed:\n%s", got.String())
}
}
File diff suppressed because one or more lines are too long
@@ -0,0 +1,46 @@
[
{
"id": "found.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"networks": [
"predecessor_default"
],
"type": "container"
},
{
"id": "library.access-storage/downloads",
"mode": "read",
"path": "/storage/downloads",
"type": "access"
},
{
"id": "library.access-storage/media/series",
"mode": "read",
"path": "/storage/media/series",
"type": "access"
},
{
"id": "library.config",
"mode": "0750",
"owner": "1001:2000",
"path": "/var/lib/library/config",
"type": "directory"
},
{
"content": "{\n \"accesses\": {\n \"downloads\": \"/storage/downloads\",\n \"series\": \"/storage/media/series\"\n },\n \"log\": \"warn\",\n \"places\": {\n \"config\": {\n \"owner\": \"1001:2000\",\n \"path\": \"/var/lib/library/config\"\n }\n },\n \"workers\": 4\n}\n",
"id": "library.runtime-config",
"mode": "0640",
"path": "/var/lib/library/config/runtime.json",
"type": "file"
},
{
"id": "library.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"type": "container",
"volumes": [
"/var/lib/library/config:/config",
"/storage/media/series:/series",
"/storage/downloads:/downloads:ro"
]
}
]
@@ -0,0 +1,37 @@
[
{
"id": "library.access-home/media/series",
"mode": "read",
"path": "/home/media/series",
"type": "access"
},
{
"id": "library.access-storage/downloads",
"mode": "read",
"path": "/storage/downloads",
"type": "access"
},
{
"id": "library.config",
"mode": "0750",
"path": "/var/lib/library/config",
"type": "directory"
},
{
"content": "{\n \"accesses\": {\n \"downloads\": \"/storage/downloads\",\n \"series\": \"/home/media/series\"\n },\n \"log\": \"warn\"\n}\n",
"id": "library.runtime-config",
"mode": "0640",
"path": "/var/lib/library/config/runtime.json",
"type": "file"
},
{
"id": "library.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"type": "container",
"volumes": [
"/var/lib/library/config:/config",
"/home/media/series:/series",
"/storage/downloads:/downloads:ro"
]
}
]
File diff suppressed because one or more lines are too long
@@ -0,0 +1,279 @@
{
"resources": [
{
"id": "distribution.registry-data",
"mode": "0700",
"path": "/var/lib/distribution/registry-data",
"type": "directory"
},
{
"id": "distribution.state",
"mode": "0700",
"path": "/var/lib/distribution/state",
"type": "directory"
},
{
"id": "distribution.store",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"ports": [
"5100:5000"
],
"type": "container",
"volumes": [
"/var/lib/distribution/state:/state",
"/var/lib/distribution/registry-data:/registry-data"
]
},
{
"id": "gitea.data",
"mode": "0700",
"path": "/var/lib/gitea/data",
"type": "directory"
},
{
"id": "gitea.grants",
"mode": "0700",
"path": "/var/lib/gitea/grants",
"type": "directory"
},
{
"id": "gitea.mesh-state",
"mode": "0700",
"path": "/var/lib/gitea/mesh-state",
"type": "directory"
},
{
"content": "{\n \"expose\": {\n \"22\": \"anywhere\"\n }\n}\n",
"id": "gitea.runtime-config",
"mode": "0640",
"path": "/var/lib/gitea/runtime-config.json",
"type": "file"
},
{
"id": "gitea.runtime-state",
"mode": "0700",
"path": "/var/lib/gitea/runtime-state",
"type": "directory"
},
{
"id": "gitea.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"ports": [
"20000:3000",
"222:22"
],
"type": "container",
"volumes": [
"/var/lib/gitea/mesh-state:/mesh-state",
"/var/lib/gitea/runtime-state:/runtime-state",
"/var/lib/gitea/state:/state",
"/var/lib/gitea/grants:/grants",
"/var/lib/gitea/data:/data"
]
},
{
"id": "gitea.state",
"mode": "0700",
"path": "/var/lib/gitea/state",
"type": "directory"
},
{
"id": "mssql.data",
"mode": "0700",
"path": "/var/lib/mssql/data",
"type": "directory"
},
{
"id": "mssql.dumps",
"mode": "0700",
"path": "/var/lib/mssql/dumps",
"type": "directory"
},
{
"id": "mssql.grants",
"mode": "0700",
"path": "/var/lib/mssql/grants",
"type": "directory"
},
{
"id": "mssql.net",
"name": "mssql",
"type": "network"
},
{
"id": "mssql.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"network": "mssql",
"ports": [
"4848:1433"
],
"type": "container",
"volumes": [
"/var/lib/mssql/state:/state",
"/var/lib/mssql/grants:/grants",
"/var/lib/mssql/data:/data",
"/var/lib/mssql/dumps:/dumps"
]
},
{
"id": "mssql.state",
"mode": "0700",
"path": "/var/lib/mssql/state",
"type": "directory"
},
{
"id": "postgres.dumps",
"mode": "0700",
"path": "/var/lib/postgres/dumps",
"type": "directory"
},
{
"id": "postgres.grants",
"mode": "0700",
"path": "/var/lib/postgres/grants",
"type": "directory"
},
{
"id": "postgres.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"ports": [
"6852:5432"
],
"type": "container",
"volumes": [
"/var/lib/postgres/state:/state",
"/var/lib/postgres/grants:/grants",
"/var/lib/postgres/store-data:/store-data",
"/var/lib/postgres/dumps:/dumps"
]
},
{
"id": "postgres.state",
"mode": "0700",
"path": "/var/lib/postgres/state",
"type": "directory"
},
{
"id": "postgres.store-data",
"mode": "0700",
"path": "/var/lib/postgres/store-data",
"type": "directory"
},
{
"content": "{\n \"contributions\": 1,\n \"generated\": \"by the mesh — do not edit; replaced whenever a module contributing to route arrives or leaves\",\n \"given\": [\n {\n \"from\": \"gitea\",\n \"node\": \"laptop\",\n \"at\": \"laptop.internal\",\n \"as\": \"mesh_laptop_gitea\",\n \"secret\": \"/laptop.gitea.secret\",\n \"values\": {}\n },\n {\n \"from\": \"umami\",\n \"node\": \"laptop\",\n \"at\": \"laptop.internal\",\n \"as\": \"mesh_laptop_umami\",\n \"secret\": \"/laptop.umami.secret\",\n \"values\": {\n \"endpoint\": \"web\",\n \"internal-name\": \"umami.laptop.internal\",\n \"label\": \"umami\",\n \"name\": \"umami.laptop.example.test\",\n \"port\": 1212\n }\n },\n {\n \"from\": \"gitea\",\n \"values\": {\n \"deny\": true,\n \"internal-name\": \"git.laptop.internal\",\n \"label\": \"git\",\n \"name\": \"git.laptop.example.test\",\n \"path\": \"/api/internal\",\n \"priority\": 100000\n }\n },\n {\n \"from\": \"gitea\",\n \"values\": {\n \"endpoint\": \"web\",\n \"internal-name\": \"git.laptop.internal\",\n \"label\": \"git\",\n \"name\": \"git.laptop.example.test\",\n \"port\": 20000\n }\n }\n ],\n \"requirement\": \"route\"\n}\n",
"id": "proxy.received-route",
"mode": "0644",
"path": "/var/lib/proxy/routes.json",
"type": "file"
},
{
"id": "umami.grants",
"mode": "0700",
"path": "/var/lib/umami/grants",
"type": "directory"
},
{
"id": "umami.net",
"name": "umami",
"type": "network"
},
{
"id": "umami.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"network": "umami",
"ports": [
"1212:3000"
],
"type": "container",
"volumes": [
"/var/lib/umami/state:/state",
"/var/lib/umami/grants:/grants"
]
},
{
"id": "umami.state",
"mode": "0700",
"path": "/var/lib/umami/state",
"type": "directory"
}
],
"rules": [
{
"Port": 222,
"Protocol": "tcp",
"From": "anywhere",
"Because": [
"gitea"
],
"Why": [
""
]
},
{
"Port": 1212,
"Protocol": "tcp",
"From": "mesh",
"Because": [
"umami"
],
"Why": [
""
]
},
{
"Port": 4848,
"Protocol": "tcp",
"From": "mesh",
"Because": [
"mssql"
],
"Why": [
""
]
},
{
"Port": 5100,
"Protocol": "tcp",
"From": "mesh",
"Because": [
"distribution"
],
"Why": [
""
]
},
{
"Port": 6852,
"Protocol": "tcp",
"From": "mesh",
"Because": [
"postgres"
],
"Why": [
""
]
},
{
"Port": 20000,
"Protocol": "tcp",
"From": "mesh",
"Because": [
"gitea"
],
"Why": [
""
]
},
{
"Port": 51820,
"Protocol": "udp",
"From": "anywhere",
"Because": [
"mesh-wireguard"
],
"Why": [
"the private network — a node at another site has no other way in"
]
}
]
}
File diff suppressed because it is too large Load Diff
+83
View File
@@ -0,0 +1,83 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A machine's send makes and seals none of the secrets of a module handed to the provisioner there (novox/hq ADR
// 0349 §7, §11, design 57 V4, issue 647): its client asks the vault for them. A module not moved, on the same
// machine, has its secret made and sealed as before.
func aClientModule() catalogue.Manifest {
return catalogue.Manifest{Module: "mesh-provisioner-client", Version: "1",
Claims: []catalogue.Claim{{Name: broker.NodeProvisionerSeat, Scope: catalogue.ScopeNode}},
OwnSecrets: catalogue.OwnSecrets{broker.ProvisionerSecret: {Path: "/var/lib/mesh-provisioner-client/broker"}},
Resources: []map[string]any{{"id": "client", "type": "process", "name": "mesh-provisioner-client",
"run": []any{"./mesh-provisioner-client"}, "user": "mesh-provisioner-client"}}}
}
func aBot(module string, moved bool) catalogue.Manifest {
m := catalogue.Manifest{Module: module, Version: "1",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/" + module + "/token"}},
Resources: []map[string]any{{"id": "conf", "type": "file", "path": "/etc/" + module + "/bot.conf",
"content": "token = ${secret:token}\n"}}}
if moved {
m.Moved = true
m.Handed = &catalogue.Handed{Source: "https://templates.example/" + module + ".tar.gz",
Digest: "sha256:" + strings.Repeat("e", 64),
Templates: map[string]string{"conf": "bot.conf"}}
}
return m
}
func TestTheSendSealsNoSecretOfAModuleHandedThere(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
client := aClientModule()
for _, m := range []catalogue.Manifest{client, aBot("moved-bot", true), aBot("kept-bot", false)} {
register(t, open, m)
if _, err := assign(ctx, open, "laptop", m.Module); err != nil {
t.Fatal(err)
}
}
if err := issueProvisionerClient(ctx, inv, client, "laptop", broker.Broker{Fingerprint: "ab"}, "10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "laptop", Contract: link.HandedContract,
SaidAt: time.Now()}); err != nil {
t.Fatal(err)
}
sent := sentTo(t, open, "laptop")
var moved, kept map[string]any
for _, r := range sent {
switch r["id"] {
case "moved-bot.needs-token":
moved = r
case "kept-bot.needs-token":
kept = r
}
}
if moved == nil || moved["sealed"] != nil || moved["owner-at-runtime"] != "owned" {
t.Fatalf("the handed module's secret file is sent as %v: placed, with no value", moved)
}
if kept == nil || kept["sealed"] == nil {
t.Fatalf("the module not moved has its secret sent as %v: sealed, as before", kept)
}
if _, held, err := inv.ModuleSecretIfIssued(ctx, "laptop", "moved-bot", "token"); err != nil || held {
t.Fatalf("the send made the handed module's secret in the controller's store (held %v, %v)", held, err)
}
for _, r := range sent {
if r["id"] == "moved-bot.conf" && (r["content"] != nil || r["sealed"] != nil) {
t.Fatalf("the handed module's templated file carries text or a value: %v", r)
}
}
}
+403
View File
@@ -0,0 +1,403 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/secrets"
)
// The vault's verbs, reached from the controller's (novox/hq ADR 0349 clauses 8 to 10; design 57's V2, issue 675).
//
// **Each module is filled the old way or the new way, by its form** (ADR 0349 clause 17, the release lead's
// decision of 2026-10-14 on issue 675). `secret accept`, `secret ask` (and `accept --at-desk`), `secret issued`
// and `secret rotate` go to the vault only for a module whose secrets are filled the new way — its machine asks
// the vault for them — and keep today's path, the controller's store and the declaration it seals, for every other
// module. Forwarding a module still filled the old way would report success and change nothing on its machine:
// the vault hands it nothing, and its machine never asks.
//
// **What the controller carries, it cannot open; and it carries only what is given at its own terminal.** The vault
// takes a value given at this terminal in two steps of one verb: the first names the secret and answers a key the
// vault made for that one call; the second carries the value sealed here to it (clause 8). A value typed at a desk
// or issued by a provider's software the controller does not carry at all (issue 723): it asks the vault once, and
// the vault asks the desk's prompt or the provider's tool itself, in its own name on its own bus account (ADR 0259
// §8's runs-as), so the prompt and the tool know the vault asks and the value opens there alone (clauses 9 and 10).
// vaultSwitch says whether a module's secrets are filled the new way, or why that could not be read.
type vaultSwitch func(module string) (bool, error)
// secretsOfTheVault says whether a module's secrets are filled the new way: the switch the controller's secret
// verbs turn on. **A module marked moved** (design 57 V4, issue 647): where it is handed to the provisioner, its
// machine's client fills its `${secret:}` and its secret files from the vault, and the controller makes and seals
// none of them (catalogue handed.go), so a value given, asked for, issued or rotated goes to the vault. Every
// other module keeps today's path. A catalogue that cannot be read is an error, never today's path by default: a
// value kept where the module's machine no longer looks would be reported kept and change nothing.
func secretsOfTheVault(module string) (bool, error) {
ctx, cancel := context.WithTimeout(context.Background(), vaultWithin)
defer cancel()
open, err := openStores(ctx)
if err != nil {
return false, fmt.Errorf("whether %s's secrets are the vault's cannot be read: %w", module, err)
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return false, fmt.Errorf("whether %s's secrets are the vault's cannot be read: %w", module, err)
}
return movedOn(shelf, module), nil
}
// movedOn is whether the catalogue marks the module moved: its secrets are the vault's.
func movedOn(shelf map[string]catalogue.Manifest, module string) bool { return shelf[module].Moved }
// filledTheNewWay is the switch as the four commands read it — accept, secret-ask (and accept --at-desk), issued and
// rotate, each at its own call site: secretsOfTheVault. A test replaces it to prove that each call site sends a
// module filled the old way down today's path and one filled the new way to the vault (the review of issue 675).
var filledTheNewWay vaultSwitch = secretsOfTheVault
// switchOf is a switch that cannot fail, as a test writes one.
func switchOf(f func(module string) bool) vaultSwitch {
return func(module string) (bool, error) { return f(module), nil }
}
// offeringModuleOn is the module on the provider's machine that offers the provision: the one whose secrets hold
// the provision's one credential, and so the one that decides the path of `accept --provider`. "" when no module,
// or more than one, on that machine offers it: today's path then, which says what is wrong.
func offeringModuleOn(ctx context.Context, provider, provision string) (string, error) {
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
assigned, err := open.inventory.Assigned(ctx, provider)
if err != nil {
return "", err
}
return offeredOn(shelf, assigned, provision), nil
}
// offeredOn is the one module of those assigned that offers the provision, or "" for none or several.
func offeredOn(shelf map[string]catalogue.Manifest, assigned []string, provision string) string {
found := ""
for _, module := range assigned {
if slices.Contains(shelf[module].Offers(), provision) {
if found != "" {
return ""
}
found = module
}
}
return found
}
// bySwitch runs the vault's path for a module whose secrets are filled the new way, and today's for every other.
func bySwitch(module string, ofTheVault vaultSwitch, today, vault func() error) error {
if ofTheVault == nil {
return today()
}
ours, err := ofTheVault(module)
if err != nil {
return fmt.Errorf("nothing was done: %w", err)
}
if ours {
return vault()
}
return today()
}
// vaultAsker asks one of the vault's verbs in the controller's name and answers its result, or its refusal.
type vaultAsker func(verb string, args map[string]any) (json.RawMessage, error)
// vaultWithin is how long the vault is given to answer one step.
const vaultWithin = 20 * time.Second
// vaultAskWithin is how long the vault is given to answer `secret-ask`, which waits for the operator at the desk:
// its prompt's time (mesh-vault's DeskPromptFor, twenty seconds) and the bus's, under the thirty seconds the caller
// of this verb waits.
const vaultAskWithin = 28 * time.Second
// theVault asks the vault's seat on this controller's bus, on the subject that names the controller.
func theVault(ctx context.Context) vaultAsker {
return func(verb string, args map[string]any) (json.RawMessage, error) {
var result json.RawMessage
err := onTheBus(func(conn *nats.Conn) error {
within := vaultWithin
if verb == "secret-ask" {
within = vaultAskWithin
}
answer, err := link.AskMeshSeatToolAs(ctx, conn, broker.ControllerName, "mesh-vault", verb, args, within)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
result = answer.Result
return nil
})
return result, err
}
}
// vaultOpened is a first step's answer: the call, the key made for it, the secret's name in the vault, and who
// asked as the vault recorded it. Never a value.
type vaultOpened struct {
Call string `json:"call"`
SealTo string `json:"seal_to"`
Secret string `json:"secret"`
AskedBy string `json:"asked_by"`
}
// vaultChanged is what the vault kept: never the value.
type vaultChanged struct {
Name string `json:"name"`
Generation int `json:"generation"`
By string `json:"by"`
Unchanged bool `json:"unchanged"`
// Each is, for a provider whose consumers keep a login each, each consumer's own login remade (issue 767).
Each []vaultChanged `json:"each,omitempty"`
}
func openVaultCall(ask vaultAsker, verb string, args map[string]any) (vaultOpened, error) {
raw, err := ask(verb, args)
if err != nil {
return vaultOpened{}, fmt.Errorf("the vault did not open the call: %w", err)
}
var o vaultOpened
if err := json.Unmarshal(raw, &o); err != nil || o.Call == "" || o.SealTo == "" {
return vaultOpened{}, errors.New("the vault answered no call and no key to seal to")
}
return o, nil
}
func endVaultCall(ask vaultAsker, verb string, args map[string]any) (vaultChanged, error) {
raw, err := ask(verb, args)
if err != nil {
return vaultChanged{}, err
}
var c vaultChanged
if err := json.Unmarshal(raw, &c); err != nil {
return vaultChanged{}, errors.New("the vault answered something that is not what it kept")
}
return c, nil
}
// ownSecretArgs names a module's own secret on one machine, as the vault's verbs take it.
func ownSecretArgs(node, module, name string) map[string]any {
return map[string]any{"node": node, "module": module, "name": name}
}
// provisionArgs names the one credential of a provider assignment and provision.
func provisionArgs(provider, provision string) map[string]any {
return map[string]any{"provider": provider, "provision": provision}
}
// acceptIntoVault is `secret accept` for a module filled the new way: the value read at this terminal is sealed
// here to the key the vault made for this call, and only the sealed box crosses the bus (ADR 0349 clause 8).
func acceptIntoVault(ask vaultAsker, target map[string]any, value, by string) (string, error) {
first := map[string]any{"by": by}
for k, v := range target {
first[k] = v
}
opened, err := openVaultCall(ask, "accept", first)
if err != nil {
return "", fmt.Errorf("nothing was read or sealed: %w", err)
}
sealed, err := secrets.Seal(opened.SealTo, []byte(value))
value = ""
if err != nil {
return "", fmt.Errorf("the vault's key for this call is not a sealing key; nothing was sent: %w", err)
}
changed, err := endVaultCall(ask, "accept", map[string]any{"call": opened.Call, "sealed": sealed})
if err != nil {
return "", fmt.Errorf("the vault did not take it: %w", err)
}
if changed.Unchanged {
return fmt.Sprintf("the vault already holds this value as %s (generation %d); nothing changed", opened.Secret,
changed.Generation), nil
}
return fmt.Sprintf("the vault now holds %s (generation %d), accepted: sealed at this terminal to the vault's key "+
"for this call, opened in the vault alone and never remade by it; its machine takes it from the vault",
opened.Secret, changed.Generation), nil
}
// rotateInVault is `secret rotate` for a module filled the new way: the vault makes the value anew and says it
// changed; the module's machine takes it from the vault, with no declaration sent.
//
// **The vault rotates only with a why** (the review of issue 675): a rotation by hand is an act worth its reason,
// and the vault refuses one without it, so it is refused here before the vault is asked.
func rotateInVault(ask vaultAsker, target map[string]any, by, why string) (string, error) {
if strings.TrimSpace(why) == "" {
return "", errors.New("not rotated: the vault rotates a secret only with a reason, `--why \"…\"`; nothing was asked")
}
args := map[string]any{"by": by, "why": why}
for k, v := range target {
args[k] = v
}
changed, err := endVaultCall(ask, "rotate", args)
if err != nil {
return "", fmt.Errorf("not rotated: %w", err)
}
return fmt.Sprintf("rotated %s in the vault (generation %d), asked by %s; the value is sealed and not shown, and "+
"its machine takes it from the vault", changed.Name, changed.Generation, changed.By), nil
}
// vaultAsked is how the vault's ask at a desk ended: what it kept, or why nothing was given. Never the value.
type vaultAsked struct {
Secret string `json:"secret"`
Generation int `json:"generation"`
AskedBy string `json:"asked_by"`
Outcome string `json:"outcome"`
Unchanged bool `json:"unchanged"`
}
// giveToVault is the desk's path for a module filled the new way (ADR 0349 clause 9, ADR 0277 §22–§27 in the
// vault; issue 723): one ask of the vault, which records the ask and holds its bounds, asks the desk's prompt itself
// in its own name, the prompt naming the asker as the vault recorded it, and opens what is typed. The controller
// never reaches the desk, and nothing typed passes it, sealed or not.
func (d deskGive) giveToVault(node, module, name, desk string) (string, error) {
raw, err := d.vault("secret-ask", map[string]any{"node": node, "module": module, "name": name, "desk": desk,
"asked_by": askedByName(d.askedBy)})
if err != nil {
return "", fmt.Errorf("the vault did not ask at the desk on %s: %w", desk, err)
}
var asked vaultAsked
if err := json.Unmarshal(raw, &asked); err != nil || asked.Outcome == "" {
return "", fmt.Errorf("the vault answered something that is not how its ask at the desk on %s ended", desk)
}
switch asked.Outcome {
case "given":
case "timed-out":
return "", fmt.Errorf("%w: the prompt on %s was not answered in time (asked by the vault)", errNothingGiven, desk)
case "dismissed", "empty":
return "", fmt.Errorf("%w: the prompt on %s was %s (asked by the vault)", errNothingGiven, desk, asked.Outcome)
default:
return "", fmt.Errorf("the vault's ask at the desk on %s ended %s", desk, asked.Outcome)
}
act := link.HandAct{Verb: "secret accept", Args: []string{node, module, name, "--at-desk", desk},
Why: fmt.Sprintf("the operator gave %s for %s on %s at the desk on %s, asked by %s; kept in the vault", name,
module, node, desk, asked.AskedBy),
Cause: "given-at-the-desk"}
recorded := ""
if err := d.record(act); err != nil {
recorded = fmt.Sprintf("\n this act could NOT be recorded in the hand-act log, and is done anyway: %v", err)
}
if d.announce != nil {
if err := d.announce(node, module, name, "at the desk on "+desk); err != nil {
recorded += fmt.Sprintf("\n this change could NOT be announced on the operator's channels: %v", err)
}
}
return fmt.Sprintf("%s on %s now holds %q, given at the desk on %s and kept in the vault as %s (generation %d); "+
"the vault asked the desk itself and opened it alone, and %s takes it from there", module, node, name, desk,
asked.Secret, asked.Generation, node) + recorded, nil
}
// vaultIssued is what the vault kept of a value a provider's software issued. Never the value.
type vaultIssued struct {
Secret string `json:"secret"`
Generation int `json:"generation"`
Unchanged bool `json:"unchanged"`
// Holders are what the vault kept for each holder when the tool answered a value per holder, and NotAnswered the
// holders named that it gave nothing for (issue 749).
Holders []vaultIssuedHolder `json:"holders,omitempty"`
NotAnswered []issuedHolder `json:"not_answered,omitempty"`
}
// vaultIssuedHolder is what the vault kept for one holder. Never the value.
type vaultIssuedHolder struct {
Node string `json:"node"`
Module string `json:"module"`
Secret string `json:"secret"`
Generation int `json:"generation"`
Unchanged bool `json:"unchanged,omitempty"`
}
// takeIntoVault is `secret issued` for a provider filled the new way (ADR 0349 clause 10; issue 723): one ask of
// the vault, which asks the provider's tool itself, in its own name, with a key made for the call, and opens what
// the tool sealed. The controller never reaches the tool, and nothing is sealed, kept or sent here: each holder
// takes it from the vault.
func (t issuedTaker) takeIntoVault(provision string, p issuedProvider) (string, error) {
args := map[string]any{"provision": provision, "provider": p.Node, "module": p.Module, "tool": p.Tool}
if len(p.Holders) > 0 {
args["holders"] = p.Holders
}
raw, err := t.vault("issued", args)
if err != nil {
return "", fmt.Errorf("the vault did not take it from its tool %s: %w", p.Tool, err)
}
var changed vaultIssued
if err := json.Unmarshal(raw, &changed); err != nil || changed.Secret == "" {
return "", fmt.Errorf("the vault answered something that is not what it kept from its tool %s", p.Tool)
}
if len(changed.Holders) > 0 {
return t.takenPerHolder(provision, p, changed), nil
}
if changed.Unchanged {
return fmt.Sprintf("%s from %s on %s: the vault already holds the value its software issued (generation %d); "+
"nothing was written", provision, p.Module, p.Node, changed.Generation), nil
}
recorded := ""
if err := t.record(link.HandAct{Verb: handActIssued, Args: []string{provision},
Why: fmt.Sprintf("%s on %s gave the credential its software issued for %s to the vault, which asked it; kept "+
"in the vault as %s", p.Module, p.Node, provision, changed.Secret),
Cause: "issued-by-software"}); err != nil {
recorded = fmt.Sprintf("\n this was NOT recorded in the hand-act log, and is done anyway: %v", err)
}
return fmt.Sprintf("%s from %s on %s: the vault asked its tool itself, and the value its software issued is kept "+
"in the vault as %s (generation %d); each holder takes it from there%s", provision, p.Module, p.Node,
changed.Secret, changed.Generation, recorded), nil
}
// takenPerHolder says what the vault kept for each holder when the provider's tool answered a value per holder (issue
// 749), records the act when any holder's value changed, and names each holder the tool gave nothing for.
func (t issuedTaker) takenPerHolder(provision string, p issuedProvider, changed vaultIssued) string {
var kept, moved []string
for _, h := range changed.Holders {
holder := issuedHolder{Node: h.Node, Module: h.Module}
kept = append(kept, fmt.Sprintf("%s as %s (generation %d)", holder, h.Secret, h.Generation))
if !h.Unchanged {
moved = append(moved, holder.String())
}
}
missing := ""
if len(changed.NotAnswered) > 0 {
names := make([]string, 0, len(changed.NotAnswered))
for _, h := range changed.NotAnswered {
names = append(names, h.String())
}
missing = fmt.Sprintf("\n its tool gave nothing for %s, which holds no value from it", strings.Join(names, ", "))
}
if len(moved) == 0 {
return fmt.Sprintf("%s from %s on %s: the vault already holds the value its software issued for each holder (%s); "+
"nothing was written%s", provision, p.Module, p.Node, strings.Join(kept, ", "), missing)
}
recorded := ""
if err := t.record(link.HandAct{Verb: handActIssued, Args: []string{provision},
Why: fmt.Sprintf("%s on %s gave the vault, which asked it, a credential its software issued for each holder of %s; "+
"new for %s", p.Module, p.Node, provision, strings.Join(moved, ", ")),
Cause: "issued-by-software"}); err != nil {
recorded = fmt.Sprintf("\n this was NOT recorded in the hand-act log, and is done anyway: %v", err)
}
return fmt.Sprintf("%s from %s on %s: the vault asked its tool itself, and keeps the value its software issued for "+
"each holder, handed to that holder alone: %s%s%s", provision, p.Module, p.Node, strings.Join(kept, ", "), missing,
recorded)
}
// actingAs is who acts in this process, as the vault records it beside the controller that carries the act.
func actingAs() string {
return strings.TrimSpace(askedByName(link.Caller()))
}
+529
View File
@@ -0,0 +1,529 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"log"
"strings"
"sync"
"testing"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/secrets"
"github.com/novox/mesh-controller/internal/testbus"
)
// The controller's secret verbs reach the vault only for a module filled the new way (design 57's V2, issue 675,
// the release lead's decision on it; novox/hq ADR 0349 clauses 8 to 10 and 17).
// standInVault answers the vault's two-step verbs as mesh-vault does: a key made for each call, the value opened
// with it here and nowhere else. It keeps what it took and every argument it was sent, as the bus carried them.
type standInVault struct {
t *testing.T
mu sync.Mutex
keys map[string]string // call → private key
taken map[string]string // secret → value
sent []string // every argument, as JSON
refuse error // refuses every first step when set
outcome []string
again bool // the value is held already
// perHolder answers `issued` with a value per holder named, the last one not answered (issue 749).
perHolder bool
// asked is how the vault's own ask at a desk ends, as mesh-vault answers `secret-ask` (issue 723): the vault
// asks the desk itself, so the controller sends one step and is told how it ended.
asked string
}
func newStandInVault(t *testing.T) *standInVault {
return &standInVault{t: t, keys: map[string]string{}, taken: map[string]string{}}
}
func (v *standInVault) ask(verb string, args map[string]any) (json.RawMessage, error) {
v.mu.Lock()
defer v.mu.Unlock()
raw, _ := json.Marshal(map[string]any{"verb": verb, "args": args})
v.sent = append(v.sent, string(raw))
if verb == "rotate" {
return json.Marshal(vaultChanged{Name: "own/g14/dunst/token", Generation: 2, By: fmt.Sprint(args["by"])})
}
// The vault asks the desk's prompt and the provider's tool itself, in its own name (issue 723): one step from
// the controller, which carries no key and no sealed value.
switch verb {
case "secret-ask":
if v.refuse != nil {
return nil, v.refuse
}
outcome := v.asked
if outcome == "" {
outcome = "given"
}
return json.Marshal(vaultAsked{Secret: fmt.Sprintf("own/%v/%v/%v", args["node"], args["module"], args["name"]),
Generation: 1, AskedBy: "as-the-vault-recorded", Outcome: outcome})
case "issued":
if v.refuse != nil {
return nil, v.refuse
}
// A tool answering a value per holder (issue 749): the vault keeps one for each holder named, and says which
// it was not given.
if holders, ok := args["holders"].([]issuedHolder); ok && v.perHolder {
out := vaultIssued{Secret: fmt.Sprintf("provision/%v/%v", args["provider"], args["provision"]), Unchanged: v.again}
for i, h := range holders {
if i == len(holders)-1 && len(holders) > 1 {
out.NotAnswered = append(out.NotAnswered, h)
continue
}
out.Holders = append(out.Holders, vaultIssuedHolder{Node: h.Node, Module: h.Module,
Secret: fmt.Sprintf("%s/%s/%s", out.Secret, h.Node, h.Module), Generation: 1, Unchanged: v.again})
}
return json.Marshal(out)
}
return json.Marshal(vaultIssued{Secret: fmt.Sprintf("provision/%v/%v", args["provider"], args["provision"]),
Generation: 1, Unchanged: v.again})
}
call, _ := args["call"].(string)
if call == "" {
if v.refuse != nil {
return nil, v.refuse
}
public, private, err := secrets.Keypair()
if err != nil {
v.t.Fatal(err)
}
call = fmt.Sprintf("call-%d", len(v.keys)+1)
v.keys[call] = private
secret := fmt.Sprintf("%s/%v", verb, args["name"])
return json.Marshal(vaultOpened{Call: call, SealTo: public, Secret: secret, AskedBy: "as-the-vault-recorded"})
}
private, ok := v.keys[call]
if !ok {
return nil, errors.New("no such call")
}
delete(v.keys, call)
if o, _ := args["outcome"].(string); o != "" {
v.outcome = append(v.outcome, o)
return json.Marshal(vaultChanged{Unchanged: true})
}
value, err := secrets.Open(private, fmt.Sprint(args["sealed"]))
if err != nil {
return nil, errors.New("not sealed to this call")
}
// As mesh-vault takes a typed value: its line ending off (intake.go AskEnd).
v.taken[verb] = strings.TrimRight(string(value), "\r\n")
return json.Marshal(vaultChanged{Name: verb, Generation: 1, Unchanged: v.again})
}
func (v *standInVault) said() string {
v.mu.Lock()
defer v.mu.Unlock()
return strings.Join(v.sent, "\n")
}
// A module's secrets are the vault's once the catalogue marks it moved (design 57 V4, issue 647): its machine's
// client asks the vault for them. Every module not marked, and one the catalogue does not hold, keeps today's path.
func TestAModulesSecretsAreTheVaultsOnceItIsMarkedMoved(t *testing.T) {
open := aMesh(t)
register(t, open, aBot("moved-bot", true))
register(t, open, aBot("kept-bot", false))
for module, want := range map[string]bool{"moved-bot": true, "kept-bot": false, "nowhere": false} {
got, err := secretsOfTheVault(module)
if err != nil || got != want {
t.Errorf("%s: the vault's %v (%v), want %v", module, got, err, want)
}
}
}
// A switch that cannot be read does nothing, and says why: never today's path by default.
func TestASwitchThatCannotBeReadDoesNothing(t *testing.T) {
var ran []string
err := bySwitch("dunst", func(string) (bool, error) { return false, errors.New("the store is gone") },
func() error { ran = append(ran, "today"); return nil },
func() error { ran = append(ran, "vault"); return nil })
if err == nil || len(ran) > 0 || !strings.Contains(err.Error(), "the store is gone") {
t.Errorf("an unreadable switch ran %v (%v)", ran, err)
}
}
// The switch: today's path for a module filled the old way, the vault's for one filled the new way, and never
// both.
func TestTheSwitchRunsOnePathByTheModulesForm(t *testing.T) {
moved := switchOf(func(m string) bool { return m == "dunst" })
for module, want := range map[string]string{"telegram": "today", "dunst": "vault"} {
var ran []string
err := bySwitch(module, moved,
func() error { ran = append(ran, "today"); return nil },
func() error { ran = append(ran, "vault"); return nil })
if err != nil || strings.Join(ran, ",") != want {
t.Errorf("%s ran %v (%v), not %s", module, ran, err, want)
}
}
var ran []string
_ = bySwitch("dunst", nil, func() error { ran = append(ran, "today"); return nil },
func() error { ran = append(ran, "vault"); return nil })
if strings.Join(ran, ",") != "today" {
t.Errorf("no switch ran %v", ran)
}
}
// `secret accept` for a module filled the new way: the value read at the terminal is sealed there to the key the
// vault made for this one call; no clear value crosses the bus or reaches the controller's log. Over a real bus,
// with a tap on every subject.
func TestAnAcceptedValueCrossesTheBusOnlySealedToTheVault(t *testing.T) {
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
before := handActConn
handActConn = conn
t.Cleanup(func() { handActConn = before })
var tapped bytes.Buffer
var tapMu sync.Mutex
tap, err := conn.Subscribe(">", func(m *nats.Msg) {
tapMu.Lock()
defer tapMu.Unlock()
tapped.WriteString(m.Subject + " " + string(m.Data) + "\n")
})
if err != nil {
t.Fatal(err)
}
defer tap.Unsubscribe()
vault := newStandInVault(t)
var subjects []string
holder, err := conn.Subscribe(broker.CalledPattern(link.SeatToolSubject("mesh-vault", "accept")), func(m *nats.Msg) {
subjects = append(subjects, m.Subject)
var args map[string]any
_ = json.Unmarshal(m.Data, &args)
result, err := vault.ask("accept", args)
answer := link.Answer{Result: result}
if err != nil {
answer = link.Answer{Error: err.Error()}
}
raw, _ := json.Marshal(answer)
_ = m.Respond(raw)
})
if err != nil {
t.Fatal(err)
}
defer holder.Unsubscribe()
var logged bytes.Buffer
log.SetOutput(&logged)
t.Cleanup(func() { log.SetOutput(nil) })
const value = "pw-of-last-year-that-must-not-show"
words, err := acceptIntoVault(theVault(context.Background()), provisionArgs("novox", "postgres"), value, "jochen")
if err != nil {
t.Fatal(err)
}
if vault.taken["accept"] != value {
t.Fatalf("the vault opened %q", vault.taken["accept"])
}
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
tapMu.Lock()
crossed := tapped.String()
tapMu.Unlock()
if !strings.Contains(crossed, "mesh.seat.mesh-vault.call.accept.controller") {
t.Fatalf("the tap heard no call to the vault: %s", crossed)
}
for _, s := range subjects {
if !strings.HasSuffix(s, ".controller") {
t.Errorf("the vault was asked on %s, not in the controller's name", s)
}
}
for where, text := range map[string]string{"the bus": crossed, "the log": logged.String(), "the answer": words} {
if strings.Contains(text, value) || strings.Contains(text, "must-not-show") {
t.Errorf("the value crossed %s in the clear", where)
}
}
if !strings.Contains(words, "sealed at this terminal") {
t.Errorf("the answer: %q", words)
}
// A vault that refuses the first step: nothing is sealed or sent.
vault.refuse = errors.New("the vault has no operator key")
if _, err := acceptIntoVault(vault.ask, ownSecretArgs("g14", "dunst", "token"), value, "jochen"); err == nil ||
!strings.Contains(err.Error(), "nothing was read or sealed") {
t.Errorf("a refused call said %v", err)
}
}
// The desk's path for a module filled the new way (issue 723): the controller asks the vault once; the vault
// records the ask, holds its bounds and asks the desk's prompt itself, so the controller never reaches the desk and
// carries no key and no sealed value; today's store is not touched. For a module filled the old way, the vault is
// never asked.
func TestAnAskForAModuleFilledTheNewWayIsTheVaults(t *testing.T) {
d, accepted, acts, asked := aDesk(t, sealedTo(t, typed))
d.askedBy = "g14/claude-code"
d.open = func(node, module, name, desk string) (int64, error) {
t.Fatal("today's record was asked")
return 0, nil
}
vault := newStandInVault(t)
d.vault = vault.ask
words, err := d.give("anchor", "telegram", "telegram-token", "laptop")
if err != nil {
t.Fatal(err)
}
if len(*accepted) != 0 {
t.Fatal("the controller opened and kept what was typed")
}
if len(*asked) != 0 {
t.Fatalf("the controller asked the desk's prompt itself: %v", *asked)
}
said := vault.said()
if !strings.Contains(said, `"verb":"secret-ask"`) || !strings.Contains(said, `"asked_by":"g14/claude-code"`) ||
!strings.Contains(said, `"desk":"laptop"`) || strings.Count(said, "\n") != 0 {
t.Errorf("the vault was not asked once, naming who asked and the desk: %s", said)
}
if strings.Contains(said, "seal_to") || strings.Contains(said, "sealed") || strings.Contains(said, `"call"`) {
t.Errorf("the controller carried a key or a sealed value: %s", said)
}
if len(*acts) != 1 || (*acts)[0].Cause != "given-at-the-desk" || !strings.Contains((*acts)[0].Why, "kept in the vault") ||
!strings.Contains((*acts)[0].Why, "as-the-vault-recorded") {
t.Errorf("the act: %+v", *acts)
}
if !strings.Contains(words, "own/anchor/telegram/telegram-token") || strings.Contains(words, typed) {
t.Errorf("said %q", words)
}
// Dismissed and timed out: nothing given, said so.
for _, outcome := range []string{"dismissed", "timed-out", "empty"} {
d2, _, _, _ := aDesk(t, sealedTo(t, typed))
v2 := newStandInVault(t)
v2.asked = outcome
d2.vault = v2.ask
if _, err := d2.give("anchor", "telegram", "telegram-token", "laptop"); !errors.Is(err, errNothingGiven) {
t.Errorf("%s: %v", outcome, err)
}
}
// Failed at the desk: an error, not "nothing given".
d5, _, acts5, _ := aDesk(t, sealedTo(t, typed))
v5 := newStandInVault(t)
v5.asked = "failed"
d5.vault = v5.ask
if _, err := d5.give("anchor", "telegram", "telegram-token", "laptop"); err == nil || errors.Is(err, errNothingGiven) ||
len(*acts5) != 0 {
t.Errorf("a failed ask: %v, acts %d", err, len(*acts5))
}
// The vault's bounds refuse: said so, nobody is asked by the controller.
d3, _, _, asked3 := aDesk(t, sealedTo(t, typed))
v3 := newStandInVault(t)
v3.refuse = errors.New("an ask for telegram-token of telegram on anchor is still open")
d3.vault = v3.ask
if _, err := d3.give("anchor", "telegram", "telegram-token", "laptop"); err == nil ||
!strings.Contains(err.Error(), "still open") || len(*asked3) != 0 {
t.Errorf("a refused ask: %v, prompts %d", err, len(*asked3))
}
// Filled the old way: today's path, the vault never asked.
d4, accepted4, _, _ := aDesk(t, sealedTo(t, typed))
if _, err := d4.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if len(*accepted4) != 1 || d4.vault != nil {
t.Error("a module filled the old way did not take today's path")
}
}
// `secret issued` for a provider filled the new way (issue 723): the controller asks the vault once, naming the
// provider's machine, module and tool; the vault asks the tool itself; the controller never reaches the tool and
// nothing is kept, sealed or sent here. A provider filled the old way keeps today's path and the vault is never
// asked.
func TestAnIssuedValueForAProviderFilledTheNewWayGoesToTheVault(t *testing.T) {
var r issuedRecorder
vault := newStandInVault(t)
taker := r.taker(t, func(issuedProvider, map[string]any) (link.Answer, error) {
t.Fatal("the controller asked the provider's tool itself")
return link.Answer{}, nil
})
taker.keep = func(string, string, string) (inventory.IssuedKept, error) {
t.Fatal("today's store was asked to keep it")
return inventory.IssuedKept{}, nil
}
taker.ofTheVault = switchOf(func(m string) bool { return m == "plex" })
taker.vault = vault.ask
words, err := taker.take("plex-api")
if err != nil {
t.Fatal(err)
}
if len(r.sent) != 0 {
t.Errorf("machines were sent: %v", r.sent)
}
if len(r.acts) != 1 || r.acts[0].Verb != handActIssued || !strings.Contains(r.acts[0].Why, "kept in the vault") {
t.Errorf("the act: %+v", r.acts)
}
said := vault.said()
for _, want := range []string{`"verb":"issued"`, `"provider":"ace"`, `"module":"plex"`, `"tool":"plex_issued_credential"`} {
if !strings.Contains(said, want) {
t.Errorf("the vault was told %s, without %s", said, want)
}
}
if strings.Contains(said, "seal_to") || strings.Contains(said, "sealed") || strings.Contains(said, `"call"`) {
t.Errorf("the controller carried a key or a sealed value: %s", said)
}
if !strings.Contains(words, "provision/ace/plex-api") {
t.Errorf("said %q", words)
}
// The same value again: the vault writes nothing, and nothing is recorded.
again := newStandInVault(t)
again.again = true
r2 := issuedRecorder{}
t2 := r2.taker(t, issuedSealed(t, "the-servers-own-token"))
t2.ofTheVault, t2.vault = switchOf(func(string) bool { return true }), again.ask
if words, err := t2.take("plex-api"); err != nil || len(r2.acts) != 0 || !strings.Contains(words, "nothing was written") {
t.Errorf("%q %v %+v", words, err, r2.acts)
}
// The vault refuses (its tool refused it): said, nothing recorded.
refused := newStandInVault(t)
refused.refuse = errors.New("its tool plex_issued_credential refused: this tool answers the vault alone")
r4 := issuedRecorder{}
t4 := r4.taker(t, issuedSealed(t, "x"))
t4.ofTheVault, t4.vault = switchOf(func(string) bool { return true }), refused.ask
if _, err := t4.take("plex-api"); err == nil || !strings.Contains(err.Error(), "answers the vault alone") || len(r4.acts) != 0 {
t.Errorf("a refusal: %v, acts %d", err, len(r4.acts))
}
// Filled the old way: today's path.
var r3 issuedRecorder
t3 := r3.taker(t, issuedSealed(t, "the-servers-own-token"))
t3.ofTheVault = switchOf(func(string) bool { return false })
t3.vault = func(string, map[string]any) (json.RawMessage, error) {
t.Fatal("the vault was asked for a provider filled the old way")
return nil, nil
}
if _, err := t3.take("plex-api"); err != nil || len(r3.kept) != 1 {
t.Errorf("today's path: %v, kept %d", err, len(r3.kept))
}
}
// `secret issued` names to the vault the provision's holders from that provider (issue 749): a tool may answer a
// value for each, which the vault keeps as that holder's own; the controller says what was kept for whom and which
// holder the tool gave nothing for, never a value. A provider answering one value is told the holders all the same
// and keeps the provision's one credential.
func TestAnIssuedValuePerHolderNamesTheHoldersToTheVault(t *testing.T) {
var r issuedRecorder
vault := newStandInVault(t)
vault.perHolder = true
taker := r.taker(t, func(issuedProvider, map[string]any) (link.Answer, error) {
t.Fatal("the controller asked the provider's tool itself")
return link.Answer{}, nil
})
taker.providers = func(string) ([]issuedProvider, bool, error) {
return []issuedProvider{{Node: "novox", Module: "gitea", Tool: "gitea_agent_token", Holders: []issuedHolder{
{Node: "anchor", Module: "claude-code"}, {Node: "laptop", Module: "claude-code"},
{Node: "shanks", Module: "claude-code"}}}}, true, nil
}
taker.ofTheVault = switchOf(func(m string) bool { return m == "gitea" })
taker.vault = vault.ask
words, err := taker.take("gitea-read")
if err != nil {
t.Fatal(err)
}
said := vault.said()
if !strings.Contains(said, `"holders":[{"node":"anchor","module":"claude-code"},{"node":"laptop","module":"claude-code"},{"node":"shanks","module":"claude-code"}]`) {
t.Errorf("the vault was told %s", said)
}
for _, want := range []string{"provision/novox/gitea-read/anchor/claude-code", "provision/novox/gitea-read/laptop/claude-code",
"gave nothing for claude-code on shanks"} {
if !strings.Contains(words, want) {
t.Errorf("said %q, without %q", words, want)
}
}
if len(r.acts) != 1 || !strings.Contains(r.acts[0].Why, "claude-code on anchor") || !strings.Contains(r.acts[0].Why, "claude-code on laptop") {
t.Errorf("the act: %+v", r.acts)
}
if len(r.sent) != 0 || len(r.kept) != 0 {
t.Error("the controller kept or sent something")
}
// The same values again: nothing written, nothing recorded.
again := newStandInVault(t)
again.perHolder, again.again = true, true
r2 := issuedRecorder{}
t2 := r2.taker(t, nil)
t2.providers, t2.ofTheVault, t2.vault = taker.providers, taker.ofTheVault, again.ask
if words, err := t2.take("gitea-read"); err != nil || len(r2.acts) != 0 || !strings.Contains(words, "nothing was written") {
t.Errorf("again: %q %v %+v", words, err, r2.acts)
}
}
// The holders named are the provision's holders from that provider, each consuming module on each machine once.
func TestTheHoldersNamedAreThoseOfThatProvider(t *testing.T) {
holders := []inventory.Holder{
{Provision: "gitea-read", Consumer: "anchor", ConsumerModule: "claude-code", Provider: "novox"},
{Provision: "gitea-read", Consumer: "anchor", ConsumerModule: "claude-code", Local: "second", Provider: "novox"},
{Provision: "gitea-read", Consumer: "laptop", ConsumerModule: "claude-code", Provider: "novox"},
{Provision: "gitea-read", Consumer: "ace", ConsumerModule: "claude-code", Provider: "shanks"},
}
got := issuedHoldersFrom(holders, "novox")
if fmt.Sprint(got) != "[claude-code on anchor claude-code on laptop]" {
t.Errorf("the holders named: %v", got)
}
if got := issuedHoldersFrom(holders, "g14"); len(got) != 0 {
t.Errorf("a provider holding nothing named %v", got)
}
shelf := map[string]catalogue.Manifest{"gitea": {Module: "gitea", Provides: []catalogue.Offer{{Name: "gitea-read",
Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedBySoftware, Tool: "gitea_agent_token"}}}}}
providers, err := issuedProvidersFrom(shelf, holders, func(string) ([]string, error) { return []string{"gitea"}, nil },
"gitea-read")
if err != nil || len(providers) != 2 || providers[0].Node != "novox" || fmt.Sprint(providers[0].Holders) !=
"[claude-code on anchor claude-code on laptop]" || fmt.Sprint(providers[1].Holders) != "[claude-code on ace]" {
t.Errorf("the providers asked: %+v %v", providers, err)
}
}
// `secret rotate` for a module filled the new way asks the vault, naming who asked; nothing is sent.
func TestARotationForAModuleFilledTheNewWayIsTheVaults(t *testing.T) {
vault := newStandInVault(t)
if _, err := rotateInVault(vault.ask, ownSecretArgs("g14", "dunst", "token"), "g14/claude-code", " "); err == nil ||
vault.said() != "" {
t.Fatalf("a rotation without a why reached the vault: %v", err)
}
words, err := rotateInVault(vault.ask, ownSecretArgs("g14", "dunst", "token"), "g14/claude-code", "leaked")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(vault.said(), `"verb":"rotate"`) || !strings.Contains(vault.said(), `"by":"g14/claude-code"`) ||
!strings.Contains(vault.said(), `"module":"dunst"`) || !strings.Contains(vault.said(), `"why":"leaked"`) {
t.Errorf("the vault was told %s", vault.said())
}
if !strings.Contains(words, "generation 2") || !strings.Contains(words, "takes it from the vault") {
t.Errorf("%q", words)
}
}
// The module that decides `accept --provider` is the one on the provider's machine that offers the provision; none,
// or several, is none, and today's path then says what is wrong.
func TestTheProvidersModuleIsTheOneThatOffersTheProvision(t *testing.T) {
offer := func(module string, provisions ...string) catalogue.Manifest {
m := catalogue.Manifest{Module: module}
for _, p := range provisions {
m.Provides = append(m.Provides, catalogue.Offer{Name: p})
}
return m
}
shelf := map[string]catalogue.Manifest{"tv": offer("tv", "tv-api"), "plex": offer("plex", "plex-api"),
"other-tv": offer("other-tv", "tv-api"), "dunst": offer("dunst")}
for _, c := range []struct {
assigned []string
want string
}{
{[]string{"dunst", "tv", "plex"}, "tv"},
{[]string{"dunst", "plex"}, ""},
{[]string{"tv", "other-tv"}, ""},
} {
if got := offeredOn(shelf, c.assigned, "tv-api"); got != c.want {
t.Errorf("%v: %q, want %q", c.assigned, got, c.want)
}
}
}
+413
View File
@@ -0,0 +1,413 @@
package main
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// Each of the four secret commands honours the switch at its own call site (the review of issue 675): run as the
// terminal runs them, against the real store and a real bus with a stand-in vault serving the vault's seat, a
// module filled the old way takes today's path and the vault is never asked, and one filled the new way reaches
// the vault. Sending every module to the vault at any one call site — or none — fails one of these. Needs the
// store (MESH_TEST_POSTGRES), and is named in testdata/store-tests.
// vaultOnTheBus serves the vault's four intake verbs on the test's bus with a stand-in vault, and says which verbs
// were asked, in order.
type vaultOnTheBus struct {
vault *standInVault
mu sync.Mutex
asked []string
}
func (v *vaultOnTheBus) verbs() string {
v.mu.Lock()
defer v.mu.Unlock()
return strings.Join(v.asked, ",")
}
func (v *vaultOnTheBus) reset() {
v.mu.Lock()
defer v.mu.Unlock()
v.asked = nil
}
// aMeshWithAVault is aMesh with the controller's bus pointed at a test bus where a stand-in vault serves its seat,
// and the switch saying that only the modules named are filled the new way.
func aMeshWithAVault(t *testing.T) (*stores, *vaultOnTheBus, func(moved ...string)) {
t.Helper()
open := aMesh(t)
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
before := handActConn
handActConn = conn
t.Cleanup(func() { handActConn = before })
served := &vaultOnTheBus{vault: newStandInVault(t)}
for _, verb := range []string{"accept", "secret-ask", "issued", "rotate"} {
sub, err := conn.Subscribe(broker.CalledPattern(link.SeatToolSubject("mesh-vault", verb)), func(m *nats.Msg) {
served.mu.Lock()
served.asked = append(served.asked, verb)
served.mu.Unlock()
var args map[string]any
_ = json.Unmarshal(m.Data, &args)
result, err := served.vault.ask(verb, args)
answer := link.Answer{Result: result}
if err != nil {
answer = link.Answer{Error: err.Error()}
}
raw, _ := json.Marshal(answer)
_ = m.Respond(raw)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
switchWas := filledTheNewWay
t.Cleanup(func() { filledTheNewWay = switchWas })
moveOnly := func(moved ...string) {
filledTheNewWay = switchOf(func(module string) bool {
for _, m := range moved {
if m == module {
return true
}
}
return false
})
}
return open, served, moveOnly
}
// aValueFile is a file holding a value, for `--from`.
func aValueFile(t *testing.T, value string) string {
t.Helper()
file := filepath.Join(t.TempDir(), "value")
if err := os.WriteFile(file, []byte(value+"\n"), 0o600); err != nil {
t.Fatal(err)
}
return file
}
// `secret accept` of a module's own secret.
func TestAcceptOfAnOwnSecretTakesThePathOfItsModulesForm(t *testing.T) {
open, vault, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "dunst", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/mesh/dunst/token"}}})
if _, err := assign(ctx, open, "anchor", "dunst"); err != nil {
t.Fatal(err)
}
accept := []string{"accept", "anchor", "dunst", "token", "--from", aValueFile(t, "the-bots-token")}
moveOnly("some-other-module")
if err := secretCommand(ctx, accept); err != nil {
t.Fatalf("today's path: %v", err)
}
if asked := vault.verbs(); asked != "" {
t.Fatalf("the vault was asked %s for a module filled the old way", asked)
}
if origin, _, err := open.inventory.OwnSecretOrigin(ctx, "anchor", "dunst", "token"); err != nil || origin == "" {
t.Fatalf("today's store does not hold it: %q %v", origin, err)
}
moveOnly("dunst")
if err := secretCommand(ctx, accept); err != nil {
t.Fatalf("the vault's path: %v", err)
}
if asked := vault.verbs(); asked != "accept,accept" {
t.Fatalf("the vault was asked %q for a module filled the new way", asked)
}
if vault.vault.taken["accept"] != "the-bots-token" {
t.Fatalf("the vault opened %q", vault.vault.taken["accept"])
}
}
// `secret accept --provider`: the provider's module decides, not the consumer's.
func TestAcceptForAProviderTakesThePathOfTheProvidersModule(t *testing.T) {
open, vault, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "tv", Version: "1",
Provides: []catalogue.Offer{{Name: "tv-api", Scope: catalogue.ScopeMesh}}})
register(t, open, catalogue.Manifest{Module: "requests", Version: "1", Requires: []string{"tv-api"},
Secrets: map[string]string{"tv-api": "/var/lib/mesh/requests/tv-api.secret"}})
for _, a := range [][2]string{{"anchor", "tv"}, {"laptop", "requests"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
accept := []string{"accept", "laptop", "requests", "tv-api", "--provider", "anchor", "--from",
aValueFile(t, "the-pairs-key")}
// The consumer filled the new way, the provider the old: the provider's module decides, today's path.
moveOnly("requests")
if err := secretCommand(ctx, accept); err != nil {
t.Fatalf("today's path: %v", err)
}
if asked := vault.verbs(); asked != "" {
t.Fatalf("the vault was asked %s for a provider filled the old way", asked)
}
moveOnly("tv")
if err := secretCommand(ctx, accept); err != nil {
t.Fatalf("the vault's path: %v", err)
}
if asked := vault.verbs(); asked != "accept,accept" {
t.Fatalf("the vault was asked %q for a provider filled the new way", asked)
}
if said := vault.vault.said(); !strings.Contains(said, `"provider":"anchor"`) || !strings.Contains(said, `"provision":"tv-api"`) {
t.Errorf("the vault was told %s", said)
}
}
// `secret accept` of a provision's credential issued outside the mesh, given once for its provider.
func TestAcceptOfAnOutsideOfferTakesThePathOfTheProvidersModule(t *testing.T) {
open, vault, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "tv", Version: "1",
Provides: []catalogue.Offer{{Name: "tv-api", Scope: catalogue.ScopeMesh,
Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedOutside}}}})
register(t, open, catalogue.Manifest{Module: "requests", Version: "1", Requires: []string{"tv-api"},
Secrets: map[string]string{"tv-api": "/var/lib/mesh/requests/tv-api.secret"}})
for _, a := range [][2]string{{"anchor", "tv"}, {"laptop", "requests"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
accept := []string{"accept", "anchor", "tv", "tv-api", "--from", aValueFile(t, "the-apps-key")}
moveOnly("requests")
// Today's path seals it to every consumer; whether it then sends is not this test's.
_ = secretCommand(ctx, accept)
if asked := vault.verbs(); asked != "" {
t.Fatalf("the vault was asked %s for a provider filled the old way", asked)
}
moveOnly("tv")
if err := secretCommand(ctx, accept); err != nil {
t.Fatalf("the vault's path: %v", err)
}
if asked := vault.verbs(); asked != "accept,accept" {
t.Fatalf("the vault was asked %q for a provider filled the new way", asked)
}
if vault.vault.taken["accept"] != "the-apps-key" {
t.Fatalf("the vault opened %q", vault.vault.taken["accept"])
}
}
// `secret ask` (and `secret accept --at-desk`): the desk is never answered on this bus, so today's path ends at the
// prompt; the vault's path is one ask of the vault, which asks the desk itself (issue 723), so it is done without
// the controller reaching the desk at all.
func TestAnAskAtTheDeskTakesThePathOfItsModulesForm(t *testing.T) {
open, vault, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "dunst", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/mesh/dunst/token"}}})
if _, err := assign(ctx, open, "anchor", "dunst"); err != nil {
t.Fatal(err)
}
for _, line := range [][]string{
{"ask", "anchor", "dunst", "token", "--at", "laptop"},
{"accept", "anchor", "dunst", "token", "--at-desk", "laptop"},
} {
vault.reset()
moveOnly("some-other-module")
err := secretCommand(ctx, line)
if err == nil || !strings.Contains(err.Error(), "could not be asked") {
t.Fatalf("%v: today's path did not reach the desk: %v", line, err)
}
if asked := vault.verbs(); asked != "" {
t.Fatalf("%v: the vault was asked %s for a module filled the old way", line, asked)
}
vault.reset()
moveOnly("dunst")
if err := secretCommand(ctx, line); err != nil {
t.Fatalf("%v: the vault's path reached the desk itself, or failed: %v", line, err)
}
// One ask of the vault, which asks the desk in its own name: the controller carries nothing to it.
if asked := vault.verbs(); asked != "secret-ask" {
t.Fatalf("%v: the vault was asked %q for a module filled the new way", line, asked)
}
}
}
// `secret issued`: the provider's tool is never answered on this bus, so today's path ends at it; the vault's path
// is one ask of the vault, which asks the tool itself (issue 723).
func TestAnIssuedValueTakesThePathOfItsProvidersModule(t *testing.T) {
open, vault, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "plex", Version: "1", Tools: []string{"plex_token"},
Provides: []catalogue.Offer{{Name: "plex-api", Scope: catalogue.ScopeMesh,
Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedBySoftware, Tool: "plex_token"}}}})
register(t, open, catalogue.Manifest{Module: "kometa", Version: "1", Requires: []string{"plex-api"},
Secrets: map[string]string{"plex-api": "/var/lib/mesh/kometa/plex-api.secret"}})
for _, a := range [][2]string{{"anchor", "plex"}, {"laptop", "kometa"}} {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
// The consumer's send makes the pair, so the provision has a holder to take it for.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if _, err := declarationFor(ctx, open, "laptop", plan, settings); err != nil {
t.Fatal(err)
}
moveOnly("kometa")
err = secretCommand(ctx, []string{"issued", "plex-api"})
if err == nil || !strings.Contains(err.Error(), "could not be asked") {
t.Fatalf("today's path did not reach the provider's tool: %v", err)
}
if asked := vault.verbs(); asked != "" {
t.Fatalf("the vault was asked %s for a provider filled the old way", asked)
}
moveOnly("plex")
if err := secretCommand(ctx, []string{"issued", "plex-api"}); err != nil {
t.Fatalf("the vault's path reached the provider's tool itself, or failed: %v", err)
}
if asked := vault.verbs(); asked != "issued" {
t.Fatalf("the vault was asked %q for a provider filled the new way", asked)
}
}
// `secret rotate` of a module's own secret.
func TestARotationTakesThePathOfItsModulesForm(t *testing.T) {
open, vault, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "dunst", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/mesh/dunst/token"}}})
if _, err := assign(ctx, open, "anchor", "dunst"); err != nil {
t.Fatal(err)
}
moveOnly()
if err := secretCommand(ctx, []string{"accept", "anchor", "dunst", "token", "--from", aValueFile(t, "tok")}); err != nil {
t.Fatal(err)
}
rotate := []string{"rotate", "anchor", "dunst", "token", "--why", "the test rotates it"}
moveOnly("some-other-module")
_ = secretCommand(ctx, rotate) // today's path: rotated in the store, or refused there
if asked := vault.verbs(); asked != "" {
t.Fatalf("the vault was asked %s for a module filled the old way", asked)
}
moveOnly("dunst")
if err := secretCommand(ctx, []string{"rotate", "anchor", "dunst", "token"}); err == nil ||
!strings.Contains(err.Error(), "--why") || vault.verbs() != "" {
t.Fatalf("a rotation in the vault without a why: %v, the vault asked %q", err, vault.verbs())
}
if err := secretCommand(ctx, rotate); err != nil {
t.Fatalf("the vault's path: %v", err)
}
if asked := vault.verbs(); asked != "rotate" {
t.Fatalf("the vault was asked %q for a module filled the new way", asked)
}
if said := vault.vault.said(); !strings.Contains(said, `"why":"the test rotates it"`) {
t.Errorf("the vault was not told why: %s", said)
}
}
// A moved module that runs as an account of its own is announced on the operator's channels before the vault is
// asked, and with the announcement failing the vault is never called (issue 836, the N1-give rule).
func TestAcceptForAMovedTrustedModuleAnnouncesBeforeTheVaultIsAsked(t *testing.T) {
open, vault, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "telegram", Version: "1", RunsAs: "telegram", SecretsOwner: "telegram",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/mesh/telegram/token"}}})
register(t, open, catalogue.Manifest{Module: "dunst", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/mesh/dunst/token"}}})
for _, m := range []string{"telegram", "dunst"} {
if _, err := assign(ctx, open, "anchor", m); err != nil {
t.Fatal(err)
}
}
moveOnly("telegram", "dunst")
was := announceTheGiven
t.Cleanup(func() { announceTheGiven = was })
var order []string
announceTheGiven = func(_ context.Context, node, module, name, how, askedBy string) error {
order = append(order, "announce:"+module+":"+vault.verbs())
return nil
}
accept := func(module string) error {
return secretCommand(ctx, []string{"accept", "anchor", module, "token", "--from", aValueFile(t, "the-bots-token")})
}
if err := accept("telegram"); err != nil {
t.Fatal(err)
}
if strings.Join(order, ",") != "announce:telegram:" || vault.verbs() != "accept,accept" {
t.Fatalf("announced %v, the vault was asked %q", order, vault.verbs())
}
vault.reset()
announceTheGiven = func(context.Context, string, string, string, string, string) error {
return errors.New("no channel took it")
}
err := accept("telegram")
if err == nil || !strings.Contains(err.Error(), "nothing was sealed") || vault.verbs() != "" {
t.Fatalf("a failed announcement: %v, the vault was asked %q", err, vault.verbs())
}
// Any other module is kept first and announced after; a failed announcement is said, not undone.
if err := accept("dunst"); err != nil || vault.verbs() != "accept,accept" {
t.Fatalf("an untrusted module: %v, asked %q", err, vault.verbs())
}
}
// issue 851 (hand-act): a moved module's terminal give is recorded in the hand-act log like acceptKept's
// (novox/hq ADR 0380 §6), naming where, and never carrying the value.
func TestAcceptForAMovedModuleRecordsAHandActAndNeverTheValue(t *testing.T) {
open, _, moveOnly := aMeshWithAVault(t)
ctx := t.Context()
api, err := jetstream.New(handActConn)
if err != nil {
t.Fatal(err)
}
if _, err := api.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: broker.HandActsBucket}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "dunst", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/mesh/dunst/token"}}})
if _, err := assign(ctx, open, "anchor", "dunst"); err != nil {
t.Fatal(err)
}
moveOnly("dunst")
was := announceTheGiven
t.Cleanup(func() { announceTheGiven = was })
announceTheGiven = func(context.Context, string, string, string, string, string) error { return nil }
if err := secretCommand(ctx, []string{"accept", "anchor", "dunst", "token", "--from", aValueFile(t, "the-bots-token")}); err != nil {
t.Fatal(err)
}
acts, err := link.HandActs(ctx, handActConn, time.Now().Add(-time.Minute))
if err != nil || len(acts) != 1 || acts[0].Verb != "secret accept" || acts[0].Cause != "given-at-the-terminal" ||
!strings.Contains(acts[0].Why, "at the controller's terminal") {
t.Fatalf("recorded %+v (%v)", acts, err)
}
if strings.Contains(acts[0].Why, "the-bots-token") {
t.Error("the act carries the value")
}
}
+6
View File
@@ -63,6 +63,9 @@ type signalFacts struct {
// failure (S21, novox/hq issue 233).
doing []inventory.Doing
doingErr error
// provFailed is every module whose files a machine's provisioner client last said it failed to place (S21,
// novox/hq ADR 0362 §3): that machine's failed apply.
provFailed []inventory.ProvReport
loop loopFacts
loopErr error
@@ -396,6 +399,9 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
f.handActs, f.handActsErr = w.gatherHandActs(ctx, now)
f.refusedSends, f.refusedSendsErr = inv.RefusedSendsSince(ctx, now.Add(-generationRefusalsSaid))
f.doing, f.doingErr = inv.NotDoingWhatTheyWereTold(ctx)
if f.doingErr == nil && w.js != nil {
f.provFailed, f.doingErr = provisioningFailed(w.js.Context())
}
f.facts.taken, _, f.facts.began, f.facts.err = exportedFacts.last()
return f
}
+2 -2
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-controller
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261013182924-5933eb757d44
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261015210729-d3e182503c19
github.com/jackc/pgx/v5 v5.10.0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
@@ -36,4 +36,4 @@ require (
// committed. Every build (the build agent's `go build`, the Dockerfile) compiles from vendor/ and
// fetches nothing; go refuses to build when vendor/ and this file disagree, so a pin moved without
// `go mod vendor` fails loudly, at once, everywhere.
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261013220009-b9a293f8c8bc
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261015211328-5a7ec3775b98
+4 -10
View File
@@ -1,13 +1,7 @@
git.novox.be/novox/mesh-host v0.0.0-20261013220009-b9a293f8c8bc h1:h5v2AbdjM/clMY1YxS9p7VuyYtIjkcw85w9Oz1cOYQc=
git.novox.be/novox/mesh-host v0.0.0-20261013220009-b9a293f8c8bc/go.mod h1:tcTK4LMs1d6JpZUwy3hSHMFG/MIuDr/XFs7/nbeaW/c=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c h1:l5onJwoIeH8yE/PjVlEeoPyXBsHp2NQiWLllz2fwX7s=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261013141520-9588871d6629 h1:7TYFV1Q6Mcz+lsLP/sds87tysET4rKCn3rRIQV8gNmA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261013141520-9588871d6629/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261013145622-f4745447ba6d h1:HEJyqorKktrkJPeeJdwEo0kdrJjH3fm/YVS9MGftCqY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261013145622-f4745447ba6d/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261013182924-5933eb757d44 h1:r5xIo5qWOXQdCSMxvqJy8kV4U+P/QWXFK6971Ha6wU8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261013182924-5933eb757d44/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-host v0.0.0-20261015211328-5a7ec3775b98 h1:27nPn3XO9yfrHoAOuTuuJPVZTA3Bdnbqlc2wRI7S/LY=
git.novox.be/novox/mesh-host v0.0.0-20261015211328-5a7ec3775b98/go.mod h1:5QOm3cIfu/k9fN6QqgDbKGCcDst7Ul1UlcfsiMVm9j4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261015210729-d3e182503c19 h1:mAMt3KFmrHzDiBv4MLL94opWcG9y73ISMrBUyisapQE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261015210729-d3e182503c19/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+162
View File
@@ -0,0 +1,162 @@
package broker
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"golang.org/x/crypto/bcrypt"
)
// **Only mesh-config's account writes mesh-config's buckets** (novox/hq ADR 0348 §9 and How it is checked, design
// 57 K3, issue 662): on a real server whose users are composed as the controller composes them. mesh-config runs on
// an account of its own (`runs-as`, ADR 0259 §8), so the machine's runtime does not carry it and its own principal
// alone is granted the writes of its four buckets. A put from any other account is refused by the server: a
// module reading the resolved values through `reads`, the runtime of the machine mesh-config runs on, and the
// controller.
func TestNatsOnlyMeshConfigsAccountPutsToItsBuckets(t *testing.T) {
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
states := []string{"values", "resolved", "predefined", "proposals"}
var buckets []Bucket
for _, s := range states {
buckets = append(buckets, Bucket{Module: "mesh-config", Name: s})
}
seat := Seat{Name: "mesh-config", Scope: "mesh", Serves: []string{"config"}}
users, err := Users(Records{Nodes: []string{"novox", "g14"}, Assigned: map[string][]Declared{
"novox": {
{Module: RuntimeModule},
{Module: "mesh-config", RunsAs: "mesh-config", Holds: []Seat{seat}, State: buckets,
Emits: []string{"config-changed"},
Reads: []string{"mesh-controller.assignments", "mesh-controller.declared-config", "mesh-controller.predefined"}},
{Module: "mesh-issues", State: []Bucket{{Module: "mesh-issues", Name: "issues"}}},
},
"g14": {{Module: "dunst", Reads: []string{"mesh-config.resolved"}}},
}})
if err != nil {
t.Fatal(err)
}
for i := range users {
users[i].PasswordHash = string(hash)
}
accounts, err := ComposeAccounts(users)
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
connect := func(user string) (*nats.Conn, chan error) {
t.Helper()
refused := make(chan error, 8)
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused <- err }))
if err != nil {
t.Fatalf("%s cannot connect: %v", user, err)
}
t.Cleanup(nc.Close)
return nc, refused
}
// The controller makes the buckets, as it does from a manifest's `state`.
controller, controllerRefused := connect(ControllerName)
cjs, err := jetstream.New(controller)
if err != nil {
t.Fatal(err)
}
for _, b := range buckets {
if _, err := cjs.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b.Bucket()}); err != nil {
t.Fatalf("the controller cannot make %s: %v", b.Bucket(), err)
}
}
owner, _ := connect("novox.mesh-config")
ojs, err := jetstream.New(owner)
if err != nil {
t.Fatal(err)
}
for _, b := range buckets {
kv, err := ojs.KeyValue(ctx, b.Bucket())
if err != nil {
t.Fatalf("mesh-config cannot bind its own %s: %v", b.Bucket(), err)
}
if _, err := kv.Put(ctx, "mesh.font-size", []byte(`{"value":12}`)); err != nil {
t.Errorf("mesh-config cannot write its own %s: %v", b.Bucket(), err)
}
}
// Every other account: a publish on the bucket's subject is refused by the server, and the bucket holds
// what mesh-config wrote.
reader, readerRefused := connect("g14.dunst")
runtime, runtimeRefused := connect("novox." + RuntimeModule)
for _, other := range []struct {
name string
nc *nats.Conn
refused chan error
}{{"a module reading the resolved values", reader, readerRefused},
{"the runtime of mesh-config's machine", runtime, runtimeRefused},
{"the controller", controller, controllerRefused}} {
for _, b := range buckets {
if err := other.nc.Publish("$KV."+b.Bucket()+".mesh.font-size", []byte(`{"value":99}`)); err != nil {
t.Fatal(err)
}
_ = other.nc.Flush()
select {
case err := <-other.refused:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("%s putting to %s: the server said %v, want a permissions violation", other.name, b.Bucket(), err)
}
case <-time.After(3 * time.Second):
t.Errorf("the server did not refuse %s a put to %s", other.name, b.Bucket())
}
}
}
// The reader still reads what it was granted.
rjs, err := jetstream.New(reader)
if err != nil {
t.Fatal(err)
}
resolved, err := rjs.KeyValue(ctx, "mesh-config_resolved")
if err != nil {
t.Fatalf("a module reading mesh-config.resolved cannot bind it: %v", err)
}
e, err := resolved.Get(ctx, "mesh.font-size")
if err != nil || string(e.Value()) != `{"value":12}` {
t.Errorf("the resolved value reads %v (%v), want mesh-config's own write", e, err)
}
for _, b := range buckets {
kv, err := ojs.KeyValue(ctx, b.Bucket())
if err != nil {
t.Fatal(err)
}
if e, err := kv.Get(ctx, "mesh.font-size"); err != nil || string(e.Value()) != `{"value":12}` {
t.Errorf("%s holds %v (%v) after the refused puts", b.Bucket(), e, err)
}
}
}
+67 -2
View File
@@ -40,6 +40,45 @@ var (
AskedBucket = BucketName(ControllerSeat, "asked")
)
// The controller's record mirrors (novox/hq ADR 0289, design 54, design 57 package F3): a bucket per kind of
// record, one key per record, written from the store by the lease holder only when the record changed, with
// one event on the controller's seat per write. Read by whatever names them in `reads` as
// `mesh-controller.<mirror>` (ADR 0201 §1), written by nobody else. **No secret in either** (ADR 0201): an
// assignment's value names its module and node; a module's declared config is what its manifest, which is
// public, declares.
//
// AssignmentsBucket holds which module is assigned to which node, keyed `<node>.<module>`; and
// DeclaredConfigBucket each registered module's config declarations as its manifest holds them, keyed by
// the module: what mesh-config resolves a config value from (ADR 0348 clauses 2 and 4); and PredefinedBucket the
// list of predefined keys the mesh published (the newest trunk build's of the catalogue, ADR 0348 §7), one key
// `list`: what mesh-config publishes as mesh-config_predefined (design 57 K3, issue 662).
var (
AssignmentsBucket = BucketName(ControllerSeat, MirrorAssignments)
DeclaredConfigBucket = BucketName(ControllerSeat, MirrorDeclaredConfig)
PredefinedBucket = BucketName(ControllerSeat, MirrorPredefined)
)
// The mirrors' local names, as a reader names them in `reads` after `mesh-controller.`.
const (
MirrorAssignments = "assignments"
MirrorDeclaredConfig = "declared-config"
MirrorPredefined = "predefined"
)
// ControllerMirrors are the local names of the controller's record mirrors, which a module may read.
func ControllerMirrors() []string {
return []string{MirrorAssignments, MirrorDeclaredConfig, MirrorPredefined}
}
// The mirrors' caps (design 54 §5): one value per key, no age, 64 KiB a value, 64 MiB a bucket.
const (
MirrorValueBytes = 64 << 10
MirrorBytes = 64 << 20
// MirrorMarkerTTL is the longest a purge's marker is kept (design 54 §2): the bucket allows a
// per-message age for it, and the writer gives each marker a day.
MirrorMarkerTTL = 24 * time.Hour
)
// AskedKeptFor is how long an ask is kept after it was made: a month, as the router keeps its own.
const AskedKeptFor = 30 * 24 * time.Hour
@@ -66,12 +105,14 @@ const (
// IsControllerBucket says a bucket is the controller's own, not a module's state nothing declares.
func IsControllerBucket(bucket string) bool {
return bucket == CallsBucket || bucket == HandActsBucket || bucket == ConditionsBucket ||
bucket == ConditionHistoryBucket || bucket == LeaseBucket || bucket == AskedBucket
bucket == ConditionHistoryBucket || bucket == LeaseBucket || bucket == AskedBucket ||
bucket == AssignmentsBucket || bucket == DeclaredConfigBucket || bucket == PredefinedBucket
}
// ControllerBuckets are the controller's own buckets, in the order they are asserted.
func ControllerBuckets() []string {
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket}
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket,
AssignmentsBucket, DeclaredConfigBucket, PredefinedBucket}
}
// ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
@@ -168,6 +209,30 @@ func (j *JetStream) EnsureControllerBuckets() error {
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", AskedBucket, err)
}
// **The record mirrors: no age** (design 54 §5). A key is removed when its record is, by a purge whose
// marker lives a day; a key that expired would be a record the bus forgot while the store still holds it.
for _, m := range []struct{ bucket, why string }{
{AssignmentsBucket, "which module is assigned to which node, keyed <node>.<module> (novox/hq ADR 0289, " +
"design 54): written by the controller alone from its store when an assignment changes, read through `reads`"},
{DeclaredConfigBucket, "each registered module's config declarations as its manifest holds them, keyed by " +
"the module (novox/hq ADR 0348, design 57 F3): written by the controller alone from its store when a " +
"registration changes them, read through `reads`"},
{PredefinedBucket, "the list of predefined keys the mesh published, the newest trunk build's of the " +
"catalogue, one key `list` (novox/hq ADR 0348 §7, design 57 K3): written by the controller alone from " +
"its store when a newer list is kept, read through `reads`"},
} {
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: m.bucket,
Description: m.why,
History: 1,
MaxValueSize: MirrorValueBytes,
MaxBytes: MirrorBytes,
Storage: jetstream.FileStorage,
LimitMarkerTTL: MirrorMarkerTTL,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", m.bucket, err)
}
}
return nil
}
+131 -1
View File
@@ -21,7 +21,9 @@ func TestTheControllerMayWriteEveryBucketItWrites(t *testing.T) {
if err != nil {
t.Fatal(err)
}
want := []string{"$KV." + CallsBucket + ".>", "$KV." + HandActsBucket + ".>"}
want := []string{"$KV." + CallsBucket + ".>", "$KV." + HandActsBucket + ".>",
"$KV." + AssignmentsBucket + ".>", "$KV." + DeclaredConfigBucket + ".>",
"$KV." + PredefinedBucket + ".>"}
for _, seat := range seatsTheControllerAsks {
if hasCancelledSet(seat) {
want = append(want, "$KV."+CancelledSetName(seat)+".>")
@@ -37,6 +39,35 @@ func TestTheControllerMayWriteEveryBucketItWrites(t *testing.T) {
}
}
// **A module reads a record mirror through `reads`, and never writes it** (ADR 0201 §1, design 54 §5,
// design 57 F3): granted the direct reads and watches of the bucket, and refused at composition a grant to
// write it, whose writer is the controller.
func TestAModuleReadsTheControllersMirrorsAndWritesNone(t *testing.T) {
reader := Principal{Kind: KindModule, Node: "one", Module: "mesh-config", PasswordHash: "x",
Reads: []string{"mesh-controller." + MirrorAssignments, "mesh-controller." + MirrorDeclaredConfig,
"mesh-controller." + MirrorPredefined}}
p, err := PermissionsFor(reader)
if err != nil {
t.Fatal(err)
}
for _, bucket := range []string{AssignmentsBucket, DeclaredConfigBucket, PredefinedBucket} {
for _, subject := range []string{"$JS.API.STREAM.INFO.KV_" + bucket, "$JS.API.DIRECT.GET.KV_" + bucket + ".>",
"$JS.API.CONSUMER.CREATE.KV_" + bucket + ".>"} {
if !slices.Contains(p.Publish, subject) {
t.Errorf("a module reading %s may not publish %s", bucket, subject)
}
}
for _, subject := range p.Publish {
if SubjectsOverlap(subject, "$KV."+bucket+".>") {
t.Errorf("a module reading %s may write it through %s", bucket, subject)
}
}
if err := CheckWriters(reader, []string{"$KV." + bucket + ".>"}); err == nil {
t.Errorf("a module granted a write of %s would compose", bucket)
}
}
}
// **A machine's node tools may say they are there, as that machine and no other** (novox/hq to-be 45
// S11), and the controller may hear the bus's advisories and ask who answers — read-only, named.
func TestTheWatchedSignalsMayBeSaidAndHeard(t *testing.T) {
@@ -96,3 +127,102 @@ func TestWhatTheControllerAskedIsBounded(t *testing.T) {
t.Errorf("history %d, age %s, bytes %d", status.History(), status.TTL(), info.Config.MaxBytes)
}
}
// **A listed mirror is no undeclared bucket** (ADR 0356 §3, ADR 0201 §16): the controller's list of its mirrors
// stands for their declaration, so the report of state nothing declares never names one.
func TestAListedMirrorIsNotReportedAsUndeclared(t *testing.T) {
names := []string{"gone_state"}
for _, m := range ControllerMirrors() {
names = append(names, BucketName(ControllerSeat, m))
}
undeclared, err := RaiseBuckets(fakeBuckets{names: names}, nil)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(undeclared, []string{"gone_state"}) {
t.Fatalf("the report of undeclared state names %v", undeclared)
}
}
// **The mirrors' list and the writers table agree** (ADR 0356 §3, to-be 45 §1): every listed mirror is a row of
// the table whose one writer is the controller, on its bucket alone; and every row that writes a bucket of the
// controller's named as a mirror is on the list.
func TestTheMirrorsListAndTheWritersTableAgree(t *testing.T) {
controller := Principal{Kind: KindController}
module := Principal{Kind: KindModule, Module: ControllerSeat, Node: "one"}
rowed := map[string]bool{}
for _, row := range WritersTable {
if !strings.HasSuffix(row.State, " mirror") {
continue
}
for _, subject := range row.Subjects {
bucket := strings.TrimSuffix(strings.TrimPrefix(subject, "$KV."), ".>")
rowed[bucket] = true
if row.Writes == nil || !row.Writes(controller, subject) || row.Writes(module, subject) {
t.Errorf("%q is not written by the controller alone", row.State)
}
}
}
listed := map[string]bool{}
for _, m := range ControllerMirrors() {
bucket := BucketName(ControllerSeat, m)
listed[bucket] = true
if !rowed[bucket] {
t.Errorf("the mirror %s is listed and no row of the writers table writes it", bucket)
}
if !IsControllerBucket(bucket) || !slices.Contains(ControllerBuckets(), bucket) {
t.Errorf("the mirror %s is listed and not asserted as the controller's own", bucket)
}
}
for bucket := range rowed {
if !listed[bucket] {
t.Errorf("the writers table writes the mirror %s, which the list does not name", bucket)
}
}
}
// **The controller may ask the vault's intake verbs in its own name** (design 57's V2, issue 675): the vault
// takes a value only from the controller, which it knows from the subject that names its caller — so the
// controller is granted that subject for the verbs the vault's holder serves, and no more of the vault's seat.
// `get` among them since issue 755 (ADR 0349 §16): the vault answers the controller a core credential sealed to its
// machine alone, which the controller carries and cannot open.
// Composed from the records, as the issued tools are: the installer's first user list carries none.
func TestTheControllerMayAskTheVaultInItsOwnName(t *testing.T) {
vault := Declared{Module: "mesh-vault", Holds: []Seat{{Name: "mesh-vault",
Serves: []string{"get", "accept", "secret-ask", "issued", "rotate"}}}}
users, err := Users(Records{Nodes: []string{"novox"}, Assigned: map[string][]Declared{"novox": {vault}}})
if err != nil {
t.Fatal(err)
}
controller := users[0]
if strings.Join(controller.AsksTheVault, ",") != "accept,get,issued,rotate,secret-ask" {
t.Fatalf("composed %v", controller.AsksTheVault)
}
p, err := PermissionsFor(controller)
if err != nil {
t.Fatal(err)
}
for _, verb := range []string{"accept", "secret-ask", "issued", "rotate", "get"} {
if !MayPublish(p, CalledSubject("mesh.seat.mesh-vault.tool."+verb, ControllerName)) {
t.Errorf("the controller may not ask the vault's %s in its own name", verb)
}
if MayPublish(p, CalledSubject("mesh.seat.mesh-vault.tool."+verb, "anchor.node-tools")) {
t.Errorf("the controller may ask the vault's %s in another's name", verb)
}
}
if MayPublish(p, CalledSubject("mesh.seat.mesh-vault.tool.move-in", ControllerName)) {
t.Error("the controller may move a value in: that is a node-engine's alone")
}
// A vault that serves none of them yet composes no grant, nor does genesis.
older := Declared{Module: "mesh-vault", Holds: []Seat{{Name: "mesh-vault", Serves: []string{"move-in"}}}}
users, _ = Users(Records{Nodes: []string{"novox"}, Assigned: map[string][]Declared{"novox": {older}}})
if len(users[0].AsksTheVault) != 0 {
t.Errorf("a vault serving none composed %v", users[0].AsksTheVault)
}
genesis, _ := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
for _, s := range genesis.Publish {
if strings.Contains(s, "mesh-vault") {
t.Errorf("the installer's controller may publish %s", s)
}
}
}
+6 -2
View File
@@ -172,7 +172,9 @@ func TestRaisingLogsReportsWhatNothingDeclares(t *testing.T) {
// **No path of the mesh deletes or purges a log or a bucket, or recreates one** (novox/hq ADR 0297 §2,
// ADR 0201 §15–16): no code of this repository outside its tests calls the client's stream or bucket
// delete, or purges a stream, but for the controller lease's own stream, which purges its own history
// below a sequence (internal/lease). A new call is a decision, not a refactor.
// below a sequence (internal/lease), and the controller's record mirrors, which purge the key of a record
// the store no longer holds (internal/recordmirror; novox/hq ADR 0289, design 54 §2). A new call is a decision,
// not a refactor.
func TestNoPathDeletesALogOrABucket(t *testing.T) {
removers := map[string]bool{"DeleteStream": true, "DeleteKeyValue": true, "PurgeStream": true,
"DeleteObjectStore": true}
@@ -197,7 +199,9 @@ func TestNoPathDeletesALogOrABucket(t *testing.T) {
return true
}
name := sel.Sel.Name
if removers[name] || (name == "Purge" && !strings.Contains(filepath.ToSlash(path), "internal/lease/")) {
decided := strings.Contains(filepath.ToSlash(path), "internal/lease/") ||
strings.Contains(filepath.ToSlash(path), "internal/recordmirror/")
if removers[name] || (name == "Purge" && !decided) {
found = append(found, path+": "+name)
}
return true
+88
View File
@@ -0,0 +1,88 @@
package broker
import (
"strings"
"testing"
)
// The move of the secrets into the vault (design 57's V3, issue 676): each machine's node-engine may ask the
// vault's move-in in its own name — the subject that names it, and no other machine's — once the vault's holder
// serves it; it hears the controller's ask to move a value on its own subject; and the controller may copy its
// record of asks with the vault's move-asks. Composed from the records: a vault that serves neither grants nothing.
func TestANodeEngineMayMoveItsSecretsIntoTheVaultInItsOwnName(t *testing.T) {
vault := Declared{Module: "mesh-vault", Holds: []Seat{{Name: "mesh-vault",
Serves: []string{"get", "accept", "secret-ask", "issued", "rotate", "move-in", "move-asks"}}}}
users, err := Users(Records{Nodes: []string{"anchor", "shanks"}, Assigned: map[string][]Declared{"anchor": {vault}}})
if err != nil {
t.Fatal(err)
}
var anchor, controller Principal
for _, u := range users {
switch {
case u.Kind == KindNode && u.Node == "anchor":
anchor = u
case u.Kind == KindController:
controller = u
}
}
p, err := PermissionsFor(anchor)
if err != nil {
t.Fatal(err)
}
if !MayPublish(p, CalledSubject("mesh.seat.mesh-vault.tool.move-in", "node.anchor")) {
t.Error("anchor's node-engine may not ask the vault's move-in in its own name")
}
for _, s := range []string{CalledSubject("mesh.seat.mesh-vault.tool.move-in", "node.shanks"),
CalledSubject("mesh.seat.mesh-vault.tool.move-in", ControllerName),
CalledSubject("mesh.seat.mesh-vault.tool.get", "node.anchor"),
CalledSubject("mesh.seat.mesh-vault.tool.accept", "node.anchor"),
"mesh.seat.mesh-vault.tool.move-in"} {
if MayPublish(p, s) {
t.Errorf("anchor's node-engine may publish %s", s)
}
}
if !MaySubscribe(p, AskMoveSecretSubject("anchor")) || MaySubscribe(p, AskMoveSecretSubject("shanks")) {
t.Error("anchor's node-engine does not hear its own move asks alone")
}
// The vault's holder hears the call naming the node-engine.
held := false
for _, u := range users {
if u.Kind == KindModule && u.Module == "mesh-vault" {
held = true
h, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
if !MaySubscribe(h, CalledSubject("mesh.seat.mesh-vault.tool.move-in", "node.anchor")) {
t.Error("the vault's holder does not hear move-in asked by a node-engine")
}
}
}
if !held {
t.Fatal("no principal holds the vault's seat")
}
c, err := PermissionsFor(controller)
if err != nil {
t.Fatal(err)
}
if !MayPublish(c, CalledSubject("mesh.seat.mesh-vault.tool.move-asks", ControllerName)) {
t.Error("the controller may not copy its record of asks into the vault")
}
if MayPublish(c, CalledSubject("mesh.seat.mesh-vault.tool.move-in", ControllerName)) {
t.Error("the controller may move a value in itself: it opens none")
}
if !MayPublish(c, AskMoveSecretSubject("anchor")) {
t.Error("the controller may not ask a machine's node-engine to move a value")
}
// A vault that serves neither, the running one: no grant for either.
older := Declared{Module: "mesh-vault", Holds: []Seat{{Name: "mesh-vault", Serves: []string{"get"}}}}
users, _ = Users(Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{"anchor": {older}}})
for _, u := range users {
p, _ := PermissionsFor(u)
for _, s := range p.Publish {
if strings.Contains(s, "move-in") || strings.Contains(s, "move-asks") {
t.Errorf("%s may publish %s while the vault serves neither", u.Username(), s)
}
}
}
}
+131 -4
View File
@@ -16,6 +16,7 @@ import (
"errors"
"fmt"
"regexp"
"slices"
"sort"
"strings"
"time"
@@ -42,6 +43,10 @@ const (
// Its authority is the union of what the modules it carries would each have had for their
// tools — and nothing of what they consume, because tools are what it runs, not reactions.
KindNodeTools Kind = "node-tools"
// KindProvisioner is a machine's provisioner client (novox/hq ADR 0349 clauses 6 and 11, design 57 P3): a
// service of the module holding NodeProvisionerSeat, outside the tool runner, under an account of its own so
// the vault can tell it from an agent. Its authority is provisionerGrants and nothing else.
KindProvisioner Kind = "provisioner"
)
// RuntimeModule is the module that IS the node's tool runtime (novox/hq ADR 0175). Where it is
@@ -125,11 +130,24 @@ type Principal struct {
// AsksIssued are the tools the controller asks in its own name for a credential a provider's software
// issued, each `<module>.<tool>` (novox/hq ADR 0305): on the subject that names its caller, so the tool
// knows the controller asks. Composed from the records, so the installer's first user list — a controller
// knows the controller asks. The vault's holder is granted the same tools in its own name (issue 723,
// vaultCalls); the controller's grant goes with its path in R1 (issue 682). Composed from the records, so the installer's first user list — a controller
// with no records — carries none, and moves with issue 464 like the rest of the controller's caller-named
// grants.
AsksIssued []string
// AsksTheVault are the vault's verbs the controller asks in its own name (VerbsTheControllerAsksTheVault,
// issue 675), those the vault's holder serves: on the subject that names its caller, so the vault knows the
// controller asks. Composed from the records like AsksIssued, so the installer's first user list carries none.
AsksTheVault []string
// AsksTheDesk says this principal is the vault's holder, which asks the desks' secret prompt in its own name
// (novox/hq ADR 0349 clause 9, issue 723): `node-launcher.secret` on any machine, on the subject that names
// its caller, and nothing else of that seat. With AsksIssued on the same principal, the vault's whole reach
// outside its own seat: it asks each issuing provider's tool in its own name (clause 10). The controller
// keeps its own grant to both until R1 (issue 682), for the modules still filled the old way (clause 17).
AsksTheDesk bool
// State is the local names of the state this principal's module keeps, and Reads the state of
// others it reads as `<module>.<name>` (novox/hq ADR 0201): a bucket each, kept by the owner's
// instances and read by whoever declares it.
@@ -160,6 +178,16 @@ type Principal struct {
// `<module>.<tool>` (novox/hq ADR 0240, to-be 48 §3): asked of the instance on its own machine and
// nowhere else.
Checks []string
// MovesIntoTheVault is a machine principal whose node-engine may ask the vault's `move-in` in its own name (design
// 57's V3, issue 676): set while the vault's holder serves it, so the engine can hand in the values sealed to its
// machine, which nothing else can open.
MovesIntoTheVault bool
// StatesItsKeyToTheVault is a machine principal whose node-engine may ask the vault's `node-key` in its own name
// (issue 786, novox/hq ADR 0372): set while the vault's holder serves it. That one verb, and nothing else new.
StatesItsKeyToTheVault bool
// BoundBuckets are the providers' buckets of entries, `<module>_bound`, a provisioner client reads its own
// machine's keys of (mesh-sdk go/provider, issue 679): every bucket named bound of a module the mesh places.
BoundBuckets []string
// PasswordHash is the bcrypt hash the mesh minted. The plaintext is sealed to the principal
// and never appears here: this file is written to a node's disk and read by a server, and a
@@ -273,6 +301,29 @@ var VerbsTheControllerAsksTheDeliveryOwner = []SeatVerb{{Seat: "mesh-delivery",
{Seat: "mesh-delivery", Verb: "close"}, {Seat: "mesh-delivery", Verb: "release"}, {Seat: "mesh-delivery", Verb: "stop"},
{Seat: "mesh-delivery", Verb: "times"}, {Seat: "mesh-delivery", Verb: "groups"}}
// VerbsTheControllerAsksTheVault are the mesh-vault seat's verbs the controller calls in its own name (novox/hq
// ADR 0349 clauses 8 to 10, design 57's V2, issue 675): it carries a value given at its terminal, typed at a desk
// or issued by a provider's software into the vault, sealed to a key the vault made for the call, and asks a
// rotation. The vault takes these from the controller alone, which it knows from the subject that names its
// caller; granted as Principal.AsksTheVault says.
//
// And `move-asks` (design 57's V3, issue 676): the controller copies its record of asks at a desk, which holds no
// value, into the vault's.
//
// And `get` (novox/hq ADR 0349 §16, issue 755): the controller asks for a core module's own credential sealed to its
// machine's key, which it carries in that machine's declaration and cannot open (mesh-vault's core.go).
var VerbsTheControllerAsksTheVault = []SeatVerb{{Seat: "mesh-vault", Verb: "accept"},
{Seat: "mesh-vault", Verb: "secret-ask"}, {Seat: "mesh-vault", Verb: "issued"}, {Seat: "mesh-vault", Verb: "rotate"},
{Seat: "mesh-vault", Verb: "move-asks"}, {Seat: "mesh-vault", Verb: "get"}}
// VaultMoveIn is the vault's verb a machine's node-engine asks in its own name to hand in a value sealed to its
// machine (design 57's V3, issue 676); granted as Principal.MovesIntoTheVault says.
var VaultMoveIn = SeatVerb{Seat: "mesh-vault", Verb: "move-in"}
// VaultNodeKey is the vault's verb a machine's node-engine asks in its own name to state its sealing key (issue 786,
// novox/hq ADR 0372); granted as Principal.StatesItsKeyToTheVault says.
var VaultNodeKey = SeatVerb{Seat: "mesh-vault", Verb: "node-key"}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
// controller's own `plans`. Named one by one; a node seat's on any machine, a mesh seat's flat.
@@ -336,6 +387,10 @@ func (p Principal) Username() string {
return "node." + p.Node
case KindController:
return "controller"
case KindProvisioner:
// Named for the machine, as a node is: one client per machine, whichever module runs it, and never
// `<node>.<module>`, which is the module's own account or the tool runner's.
return "provisioner." + p.Node
case KindEnrolment:
// Per token, not one shared user. **The inbox is the reason**: with a single `enrolment`
// user every machine enrolling at once could read every other's answer, and an answer
@@ -375,6 +430,11 @@ func AskJudgeDirectoriesSubject(node string) string {
return "mesh.node." + node + ".ask.judge-directories"
}
// AskMoveSecretSubject is where the controller asks one machine's node-engine to move one value sealed to it into
// the vault (design 57's V3, issue 676): a request answered once, signed as a hand-over is (link.MoveSecretContext),
// for the same reason.
func AskMoveSecretSubject(node string) string { return "mesh.node." + node + ".ask.move-secret" }
// inbox is a principal's own reply space. No user is ever granted a bare `_INBOX.>` (design 25
// §4): with one account, inbox privacy is the permission list or it is nothing, so each user's
// inbox is derived from its own identity and its permissions name that prefix and no other.
@@ -598,6 +658,12 @@ func PermissionsFor(p Principal) (Permissions, error) {
pub = append(pub, "$JS.ACK."+DeadLetterNoticesStream+"."+ControllerName+".>", deadLetterPrefix+">",
againPrefix+">")
case KindProvisioner:
var err error
if pub, sub, err = provisionerGrants(p); err != nil {
return Permissions{}, err
}
case KindPerson:
// Tools, and nothing else. Every subject a person may publish is a tool call; a person
// who could publish an event would be able to claim a module said something.
@@ -659,11 +725,24 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And asking it for a fresh search for setuid programs (novox/hq issue 361), answered the same way.
AskSetuidSearchSubject(p.Node),
// And asking it to judge its declared directories, read-only (novox/hq issue 358), answered the same way.
AskJudgeDirectoriesSubject(p.Node)}
AskJudgeDirectoriesSubject(p.Node),
// And asking it to move a value sealed to it into the vault (design 57's V3, issue 676), answered the
// same way.
AskMoveSecretSubject(p.Node)}
// The node-engine witnesses the core builds it places (novox/hq to-be 45 §8, ADR 0236; the
// contract is lease/witness.go): it asks its own machine's node tools PING, and, where the
// machine runs the controller, reads the lease's one key — read, never written.
pub = append(pub, WitnessSubjects(p)...)
// And, while the vault serves it, the vault's move-in in its own name and nobody else's (issue 676): the
// values sealed to this machine, which only it opens, handed in sealed to the vault's key for the call.
if p.MovesIntoTheVault {
pub = append(pub, "mesh.seat."+VaultMoveIn.Seat+"."+CallKind+"."+VaultMoveIn.Verb+"."+CallerToken(p.Username()))
}
// And, while the vault serves it, its node-key in its own name (issue 786): the machine's sealing key, stated
// by the engine that holds it, which the vault takes from no other caller.
if p.StatesItsKeyToTheVault {
pub = append(pub, "mesh.seat."+VaultNodeKey.Seat+"."+CallKind+"."+VaultNodeKey.Verb+"."+CallerToken(p.Username()))
}
case KindModule:
// 1. Its own namespace: it publishes its events there and serves its tools there. Nothing
@@ -874,6 +953,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
return Permissions{}, err
}
pub = append(pub, invoked...)
// And its own machine's provisioner client, for a carried module's config and non-secret bound values,
// naming the module from the bundle that asks (novox/hq ADR 0370 §3, §4): the ask on its own machine's
// subject alone, and what the client says moved, so a module's code's watches are asked again.
pub = append(pub, ProvisionerAskSubject(p.Node))
sub = append(sub, "prov.changed."+p.Node+".>")
// It says what it serves and may ask what answers (novox/hq ADR 0197): the runtime answers
// discovery for each module and seat it carries, and the console it is asks the bus.
// One service per runtime process, named for the runtime: the bus lets a principal answer each
@@ -979,8 +1063,14 @@ func PermissionsFor(p Principal) (Permissions, error) {
// runs, so a change to them waits on a node-engine delivery. It calls and serves on the subjects that name
// no caller meanwhile — a caller-named call to its seat reaches nobody and is asked again on those at once —
// and moves with issue 464.
// The vault's calls in its own name (issue 723), kept apart so the desk's grant is not denied below.
var vaultsOwn []string
if p.Kind != KindController {
pub, sub = callerNamed(p.Username(), pub, sub)
if p.Kind == KindModule {
vaultsOwn = vaultCalls(p)
pub = append(pub, vaultsOwn...)
}
} else {
// **But the tools that answer a software-issued credential** (novox/hq ADR 0305 §2): those answer the
// controller alone, and know it only from the subject's last token, so it asks them in its own name —
@@ -992,6 +1082,13 @@ func PermissionsFor(p Principal) (Permissions, error) {
}
pub = append(pub, "mesh.mod."+module+"."+CallKind+"."+tool+".*."+CallerToken(p.Username()))
}
// **And the vault's intake verbs** (issue 675): the vault takes a value from the controller alone, known
// from the subject's last token — the verbs its holder serves, flat, and nothing else of its seat.
for _, v := range VerbsTheControllerAsksTheVault {
if slices.Contains(p.AsksTheVault, v.Verb) {
pub = append(pub, "mesh.seat."+v.Seat+"."+CallKind+"."+v.Verb+"."+CallerToken(p.Username()))
}
}
}
sort.Strings(pub)
@@ -1006,6 +1103,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
if p.Kind != KindController {
for _, only := range ControllerOnly() {
for _, a := range pub {
if slices.Contains(vaultsOwn, a) {
continue
}
if SubjectsOverlap(a, only) {
deny = append(deny, only)
break
@@ -1022,11 +1122,38 @@ func PermissionsFor(p Principal) (Permissions, error) {
// thing, a hand-over on its own subject (novox/hq issue 356), and answers that: the node is its
// machine's engine, root there already, and it is delivered only its own subjects. What it answers is
// never trusted for being an answer — the controller reads the engine's words and records nothing. A
// person is never asked anything, and is granted nothing here.
AllowResponses: p.Kind == KindModule || p.Kind == KindController || p.Kind == KindNodeTools || p.Kind == KindNode,
// provisioner client is asked by its own machine's tool runner alone, on its own machine's subject (novox/hq
// ADR 0370 §3), and answers that. A person is never asked anything, and is granted nothing here.
AllowResponses: p.Kind == KindModule || p.Kind == KindController || p.Kind == KindNodeTools || p.Kind == KindNode ||
p.Kind == KindProvisioner,
}, nil
}
// vaultCalls are the calls the vault's holder makes in its own name (novox/hq ADR 0349 clauses 9 and 10, issue
// 723): the desks' secret prompt on any machine, and each tool that answers a credential its provider's software
// issued, on any machine — each on the subject whose last token is the vault's own name, which the bus lets it
// alone publish, so the prompt and the tool know the vault asks. Nothing else outside its seat.
func vaultCalls(p Principal) []string {
token := CallerToken(p.Username())
if token == "" {
return nil
}
var out []string
if p.AsksTheDesk {
for _, v := range VerbsTheControllerAsksForASecret {
out = append(out, "mesh.seat."+v.Seat+"."+CallKind+"."+v.Verb+".*."+token)
}
}
for _, key := range p.AsksIssued {
module, tool, ok := strings.Cut(key, ".")
if !ok || module == "" || tool == "" || strings.ContainsAny(key, "*> ") {
continue
}
out = append(out, "mesh.mod."+module+"."+CallKind+"."+tool+".*."+token)
}
return out
}
// seatSubject places a seat's verb under the kind of traffic it is.
//
// **The kind token is load-bearing, not decoration.** A stream is defined by a subject filter, so
+58
View File
@@ -0,0 +1,58 @@
package broker
import "testing"
// Issue 786 (novox/hq ADR 0372): each machine's node-engine may ask the vault's node-key in its own name, the
// subject that names it and no other machine's, once the vault's holder serves it; and nothing else of the vault's
// is granted by it. A vault that does not serve it grants nothing.
func TestANodeEngineMayStateItsSealingKeyToTheVaultInItsOwnNameAlone(t *testing.T) {
vault := Declared{Module: "mesh-vault", Holds: []Seat{{Name: "mesh-vault", Serves: []string{"get", "node-key"}}}}
users, err := Users(Records{Nodes: []string{"anchor", "shanks"}, Assigned: map[string][]Declared{"anchor": {vault}}})
if err != nil {
t.Fatal(err)
}
var anchor Principal
for _, u := range users {
if u.Kind == KindNode && u.Node == "anchor" {
anchor = u
}
}
p, err := PermissionsFor(anchor)
if err != nil {
t.Fatal(err)
}
if !MayPublish(p, CalledSubject("mesh.seat.mesh-vault.tool.node-key", "node.anchor")) {
t.Error("anchor's node-engine may not state its key to the vault in its own name")
}
for _, s := range []string{CalledSubject("mesh.seat.mesh-vault.tool.node-key", "node.shanks"),
CalledSubject("mesh.seat.mesh-vault.tool.node-key", ControllerName),
CalledSubject("mesh.seat.mesh-vault.tool.get", "node.anchor"),
CalledSubject("mesh.seat.mesh-vault.tool.move-in", "node.anchor"),
"mesh.seat.mesh-vault.tool.node-key"} {
if MayPublish(p, s) {
t.Errorf("anchor's node-engine may publish %s", s)
}
}
for _, u := range users {
if u.Kind == KindModule && u.Module == "mesh-vault" {
h, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
if !MaySubscribe(h, CalledSubject("mesh.seat.mesh-vault.tool.node-key", "node.anchor")) {
t.Error("the vault's holder does not hear node-key asked by a node-engine")
}
}
}
older := Declared{Module: "mesh-vault", Holds: []Seat{{Name: "mesh-vault", Serves: []string{"get", "move-in"}}}}
users, _ = Users(Records{Nodes: []string{"anchor"}, Assigned: map[string][]Declared{"anchor": {older}}})
for _, u := range users {
if u.Kind != KindNode {
continue
}
q, _ := PermissionsFor(u)
if MayPublish(q, CalledSubject("mesh.seat.mesh-vault.tool.node-key", "node.anchor")) {
t.Error("a node-engine may state its key while the vault does not serve node-key")
}
}
}
+135
View File
@@ -0,0 +1,135 @@
package broker
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"golang.org/x/crypto/bcrypt"
)
// **On a real server, as composed** (design 57 P6, issue 672, ADR 0362): a machine's provisioner client's word on a
// module's files is kept by the mesh's own stream, newest per module, and the controller reads it by subject; the
// client cannot say it of another machine, and nothing but a machine's own client is granted to say it.
func TestTheProvisionersReportIsKeptAndReadByTheController(t *testing.T) {
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
accounts, err := ComposeAccounts([]Principal{
{Kind: KindController, PasswordHash: string(hash)},
{Kind: KindProvisioner, Node: "g14", Module: "mesh-provisioner-client", PasswordHash: string(hash)},
{Kind: KindProvisioner, Node: "shanks", Module: "mesh-provisioner-client", PasswordHash: string(hash)},
})
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
defer s.Shutdown()
refusals := make(chan error, 16)
connect := func(user string) *nats.Conn {
t.Helper()
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refusals <- err }))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
return nc
}
control := OnConn(connect("controller"))
if err := AssertMeshStreams(control); err != nil {
t.Fatalf("the mesh's streams cannot be asserted: %v", err)
}
g14 := connect("provisioner.g14")
for _, said := range []string{`{"status":"waiting"}`, `{"status":"applied","templates":"sha256:new"}`} {
if err := g14.Publish(ProvisionerAppliedSubject("g14", "dunst"), []byte(said)); err != nil {
t.Fatal(err)
}
}
if err := g14.Flush(); err != nil {
t.Fatal(err)
}
// Another machine's client says nothing of g14: refused by the server, and nothing kept.
shanks := connect("provisioner.shanks")
if err := shanks.Publish(ProvisionerAppliedSubject("g14", "dunst"), []byte(`{"status":"failed"}`)); err != nil {
t.Fatal(err)
}
_ = shanks.Flush()
select {
case err := <-refusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Fatalf("the server said %v, want a permissions violation", err)
}
case <-time.After(2 * time.Second):
t.Fatal("the server did not refuse shanks's client a report on g14")
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
js, err := control.Conn().JetStream(nats.Context(ctx))
if err != nil {
t.Fatal(err)
}
// A core publish is kept by the stream once the server has stored it: read until it has, a few seconds at most.
var got []byte
var said bool
for deadline := time.Now().Add(5 * time.Second); ; {
got, said, err = LastProvApplied(js, "g14", "dunst")
if (err == nil && said && strings.Contains(string(got), "applied")) || time.Now().After(deadline) {
break
}
time.Sleep(20 * time.Millisecond)
}
if err != nil || !said || string(got) != `{"status":"applied","templates":"sha256:new"}` {
t.Fatalf("the controller read %q %v %v, want g14's newest report", got, said, err)
}
if _, said, err := LastProvApplied(js, "shanks", "dunst"); err != nil || said {
t.Fatalf("a module nothing was said of reads as said: %v %v", said, err)
}
all, err := ProvAppliedReports(js)
if err != nil || len(all) != 1 || string(all["prov.applied.g14.dunst"]) != `{"status":"applied","templates":"sha256:new"}` {
t.Fatalf("every report, newest per module: %q %v", all, err)
}
info, err := js.StreamInfo(ProvAppliedStream)
if err != nil || info.State.Msgs != 1 {
t.Fatalf("the stream keeps the newest report per module only: %+v %v", info, err)
}
// The writers table: a machine's own client alone is granted to say it.
if err := CheckWriters(Principal{Kind: KindProvisioner, Node: "g14"}, []string{"prov.applied.g14.>"}); err != nil {
t.Fatalf("g14's client may not say its own: %v", err)
}
for _, p := range []Principal{{Kind: KindProvisioner, Node: "shanks"}, {Kind: KindNode, Node: "g14"},
{Kind: KindModule, Node: "g14", Module: "dunst"}} {
err := CheckWriters(p, []string{"prov.applied.g14.>"})
if err == nil || !strings.Contains(err.Error(), "a module's files placed on a machine") {
t.Errorf("%s may say g14's provisioner's report: %v", p.Username(), err)
}
}
}
+212
View File
@@ -0,0 +1,212 @@
package broker
import (
"errors"
"fmt"
"strings"
"github.com/nats-io/nats.go"
)
// The provisioner client's own bus account (novox/hq ADR 0349 clauses 6 and 11, design 57 package P3).
//
// **Not the tool runner's account.** The client fills a machine's files and asks the vault for the secrets that
// machine's declaration names; the vault hands a secret only to an account of its own, because a call under the
// tool runner's account cannot be told from an agent's on that machine (ADR 0305 §3, §12). So each machine that
// runs the client has a principal of its own, KindProvisioner, named for the machine, and granted what the client
// does and nothing else: read its own machine's keys in each domain's bucket, write its own machine's keys of the
// provisioner server's needs, applied and collections (design 57 P4), read the server's registry, say what it
// applied on its own machine, and ask the vault in its own name.
// NodeProvisionerSeat is the seat the provisioner client holds on every machine (ADR 0349 clause 3).
const NodeProvisionerSeat = "node-provisioner"
// ProvisionerSecret is the own secret of the module running the client that its bus account is delivered in, as
// `broker` is a module's own (novox/hq issue 195): a module that claims NodeProvisionerSeat and declares no own
// secret of this name has nowhere to read an account, and is given none.
const ProvisionerSecret = "provisioner-broker"
// ConfigResolvedBucket is the config domain's resolved values, keyed `<node>.<module>.<key>` (ADR 0348, research
// 051): mesh-config's state `resolved`, written by mesh-config alone.
var ConfigResolvedBucket = BucketName("mesh-config", "resolved")
// ProvisionerReads are the domains' buckets a provisioner client reads its own machine's keys of, each keyed
// `<node>.…` (research 051): placement, the controller's assignments mirror (`<node>.<module>`), whose value carries
// the provisioner's plan of a module handed to it (inventory.Provisioning, design 57 P5); and config,
// mesh-config's resolved values. **Named one by one**, as the controller's asks are: a domain that moves to the
// provisioner adds its bucket here, and a bucket not named is one the client cannot read.
var ProvisionerReads = []string{ConfigResolvedBucket}
// ProvisionerReadsWhole are the buckets a provisioner client reads every key of: the controller's assignments mirror,
// where a provider's consumers are bound to it in their own machines' plans, which the provider machine's client
// reads to fill ${consumers:<provision>} with each one's credential (novox/hq ADR 0373, issue 679). It holds placement
// and plans, never a secret's value; the client writes none of it.
var ProvisionerReadsWhole = []string{AssignmentsBucket}
// BoundState is a providing module's state of entries, one per consumer (mesh-sdk go/provider, issue 678): a
// consumer's machine's client reads its own keys of it, `<node>.<module>.<provision>[.<local>]`, to fill
// ${bound:<provision>:<key>} (issue 679).
const BoundState = "bound"
// ProvisionerServerModule is the module holding the seat mesh-provisioner once for the mesh (ADR 0349 clause 3,
// design 57 P4), whose buckets the clients write their own machine's keys of.
const ProvisionerServerModule = "mesh-provisioner"
// The provisioner server's buckets (ADR 0349 clause 3): its state, created by the controller from its manifest
// like any module's, and written by its own account whole.
var (
// ProvisionerRegistryBucket is which domain answers which kind of placeholder, keyed by the kind; the
// server's alone, read whole by every client.
ProvisionerRegistryBucket = BucketName(ProvisionerServerModule, "registry")
// ProvisionerNeedsBucket is what each module on each machine needs, `<node>.<module>.<kind>.<name>`.
ProvisionerNeedsBucket = BucketName(ProvisionerServerModule, "needs")
// ProvisionerAppliedBucket is what is applied where, `<node>.<module>`.
ProvisionerAppliedBucket = BucketName(ProvisionerServerModule, "applied")
// ProvisionerCollectBucket is the items of every collection, `<node>.<collection>.<contributor>`.
ProvisionerCollectBucket = BucketName(ProvisionerServerModule, "collect")
)
// ProvisionerWrites are the server's buckets a client writes, and reads, its own machine's keys of (`<node>.>`)
// and no other's: the needs it records, what it applied, and the items of the collections on its machine. A
// client cannot write another machine's needs, or say another machine applied a file.
var ProvisionerWrites = []string{ProvisionerNeedsBucket, ProvisionerAppliedBucket, ProvisionerCollectBucket}
// VerbsTheProvisionerAsksTheVault are the vault's verbs a provisioner client asks in its own name (ADR 0349
// clause 11): `get`, a secret its machine's declaration names, answered sealed to the asker's key. On the subject
// that names its caller, so the vault knows which machine's client asks from the bus's word.
var VerbsTheProvisionerAsksTheVault = []SeatVerb{{Seat: "mesh-vault", Verb: "get"}}
// VaultSecretChangedSubject is where the vault says a secret it holds changed (`secret-changed`: which, by whom,
// when, its generation; never a value, ADR 0349 clause 11), as link.EventSubject addresses mesh-vault's event. A
// provisioner client listens to it so a rotated secret reaches its file without a new declaration (design 57 V4):
// it asks the vault again, in its own name, for a secret its machine's declaration names.
const VaultSecretChangedSubject = "mesh.mod.mesh-vault.event.secret-changed"
// ProvisionerAppliedSubject is where a machine's client says what it applied for one module, applied or failed
// with the reason (ADR 0349 clauses 4 and 15): `prov.applied.<node>.<module>`.
func ProvisionerAppliedSubject(node, module string) string {
return "prov.applied." + node + "." + module
}
// ProvisionerAskSubject is where a machine's tool runner asks that machine's provisioner client for a config value
// or a non-secret bound value of a module it carries, naming the module from the bundle that asks (novox/hq ADR
// 0370 §1, §3, §4; issue 685): `prov.ask.<node>`, which the client subscribes and the tool runner's account of the
// same machine alone may publish. The bus proves only that the tool runner asks; the module is the runtime's word,
// so the client answers no secret here (ADR 0370 §2).
func ProvisionerAskSubject(node string) string { return "prov.ask." + node }
// ProvisionerChangedSubject is where a machine's client says that a value of one module it answers moved, so the
// tool runner asks again for the module's code's watches (ADR 0349 §14, ADR 0370 §1): `prov.changed.<node>.<module>`,
// carrying which value and never the value itself, heard by that machine's tool runner alone.
func ProvisionerChangedSubject(node, module string) string {
return "prov.changed." + node + "." + module
}
// provisionerGrants is a provisioner client's whole authority: to publish, and to subscribe.
func provisionerGrants(p Principal) (pub, sub []string, err error) {
if p.Node == "" {
// Refused rather than composed into `$KV.<bucket>..>`, a pattern no machine's keys are under, and a
// name every nameless client would share.
return nil, nil, errors.New("a provisioner client names no machine, so it would have no keys of its own " +
"to read and an account it shared with every other")
}
for _, bucket := range ProvisionerReads {
pub = append(pub, machineKeysRead(bucket, p.Node)...)
}
for _, bucket := range ProvisionerReadsWhole {
pub = append(pub, wholeBucketRead(bucket)...)
}
for _, bucket := range p.BoundBuckets {
module, name, ok := strings.Cut(bucket, "_")
if !ok || name != BoundState || !safeSubject.MatchString(module) || strings.Contains(module, "_") {
return nil, nil, fmt.Errorf("%q is not a provider's bucket of entries, <module>_%s", bucket, BoundState)
}
pub = append(pub, machineKeysRead(bucket, p.Node)...)
}
// The server's buckets (design 57 P4): its own machine's keys of the needs, the applied and the
// collections, read and written, `$KV.<bucket>.<node>.>` and nothing wider; and the registry, read whole,
// which says which domain answers which kind and is the same for every machine.
for _, bucket := range ProvisionerWrites {
pub = append(pub, machineKeysRead(bucket, p.Node)...)
pub = append(pub, "$KV."+bucket+"."+p.Node+".>")
}
pub = append(pub, wholeBucketRead(ProvisionerRegistryBucket)...)
pub = append(pub, "prov.applied."+p.Node+".>")
token := CallerToken(p.Username())
if token == "" {
return nil, nil, fmt.Errorf("%q cannot name a caller", p.Username())
}
for _, v := range VerbsTheProvisionerAsksTheVault {
pub = append(pub, "mesh.seat."+v.Seat+"."+CallKind+"."+v.Verb+"."+token)
}
// What moved of a value it answers its own machine's tool runner (ADR 0370 §1), under its own machine's name.
pub = append(pub, "prov.changed."+p.Node+".>")
// Its own reply space, for the vault's answers and its reads' and watches' deliveries; what the vault says
// changed; and its own machine's tool runner asking it for a module's config and non-secret bound values (ADR
// 0370 §3), which it answers on the ask's reply and nowhere else.
return pub, []string{p.inbox(), VaultSecretChangedSubject, ProvisionerAskSubject(p.Node)}, nil
}
// machineKeysRead is reading every key of a bucket under one machine's name, `<node>.>`, and no other key: a
// direct get of a key's own subject, and a consumer whose filter is under it — the server's create-with-filter
// form, which it holds to the filter the subject names (as stateGrants does for one key, novox/hq ADR 0260).
// Binding, deleting a consumer and flow control cannot be narrowed by key and read no value.
func machineKeysRead(bucket, node string) []string {
stream := "KV_" + bucket
keys := "$KV." + bucket + "." + node + ".>"
return []string{
"$JS.API.STREAM.INFO." + stream,
"$JS.API.DIRECT.GET." + stream + "." + keys,
"$JS.API.CONSUMER.CREATE." + stream + ".*." + keys,
"$JS.API.CONSUMER.DELETE." + stream + ".>",
"$JS.FC." + stream + ".>",
}
}
// wholeBucketRead is reading every key of a bucket and writing none: what stateGrants grants a module for a
// bucket it `reads`.
func wholeBucketRead(bucket string) []string {
stream := "KV_" + bucket
return []string{
"$JS.API.STREAM.INFO." + stream,
"$JS.API.DIRECT.GET." + stream + ".>",
"$JS.API.CONSUMER.CREATE." + stream + ".>",
"$JS.API.CONSUMER.DELETE." + stream + ".>",
"$JS.FC." + stream + ".>",
}
}
// LastProvApplied is what a machine's provisioner client last said of one module's files, as it published it on
// ProvisionerAppliedSubject and ProvAppliedStream keeps it (design 57 P6): false when it has said nothing of it in
// the stream's week.
func LastProvApplied(js nats.JetStreamContext, node, module string) ([]byte, bool, error) {
msg, err := js.GetLastMsg(ProvAppliedStream, ProvisionerAppliedSubject(node, module))
if errors.Is(err, nats.ErrMsgNotFound) {
return nil, false, nil
}
if err != nil {
return nil, false, fmt.Errorf("what %s's provisioner client said of %s cannot be read: %w", node, module, err)
}
return msg.Data, true, nil
}
// ProvAppliedReports is the newest word of every machine's provisioner client on every module it said one of, kept by
// ProvAppliedStream, by subject (`prov.applied.<node>.<module>`).
func ProvAppliedReports(js nats.JetStreamContext) (map[string][]byte, error) {
info, err := js.StreamInfo(ProvAppliedStream, &nats.StreamInfoRequest{SubjectsFilter: ProvAppliedSubjects})
if err != nil {
return nil, fmt.Errorf("the provisioner clients' reports cannot be listed: %w", err)
}
out := map[string][]byte{}
for subject := range info.State.Subjects {
msg, err := js.GetLastMsg(ProvAppliedStream, subject)
if errors.Is(err, nats.ErrMsgNotFound) {
continue // aged out between the two reads
}
if err != nil {
return nil, fmt.Errorf("the provisioner client's report on %s cannot be read: %w", subject, err)
}
out[subject] = msg.Data
}
return out, nil
}
+393
View File
@@ -0,0 +1,393 @@
package broker
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"golang.org/x/crypto/bcrypt"
)
// **On a real server, as composed** (design 57 package P3, issue 669): a machine's provisioner client reads and
// watches its own machine's keys in a domain's bucket, and the server refuses it another machine's keys, a write,
// and a report or a vault ask in another's name.
func TestAServerComposedFromTheGrantsRefusesTheProvisionerClientAnotherMachinesKeys(t *testing.T) {
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
vault := Seat{Name: "mesh-vault", Scope: "mesh", Serves: []string{"get"}}
accounts, err := ComposeAccounts([]Principal{
{Kind: KindController, PasswordHash: string(hash)},
{Kind: KindModule, Node: "novox", Module: "mesh-config", State: []string{"resolved"}, PasswordHash: string(hash)},
{Kind: KindModule, Node: "novox", Module: "mesh-vault", Holds: []Seat{vault}, PasswordHash: string(hash)},
{Kind: KindModule, Node: "novox", Module: "postgres", State: []string{"bound"}, PasswordHash: string(hash)},
{Kind: KindProvisioner, Node: "g14", Module: "mesh-provisioner", BoundBuckets: []string{"postgres_bound"},
PasswordHash: string(hash)},
})
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
defer s.Shutdown()
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
connect := func(user string, refusals chan error) *nats.Conn {
t.Helper()
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) {
if refusals != nil {
refusals <- err
}
}))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
return nc
}
// The controller makes the buckets, as it does from registration; the controller writes its mirror and
// mesh-config writes its own bucket.
control, err := jetstream.New(connect("controller", nil))
if err != nil {
t.Fatal(err)
}
put := func(js jetstream.JetStream, bucket string, values map[string]string) {
t.Helper()
kv, err := js.KeyValue(ctx, bucket)
if err != nil {
t.Fatal(err)
}
for k, v := range values {
if _, err := kv.PutString(ctx, k, v); err != nil {
t.Fatalf("writing %s in %s: %v", k, bucket, err)
}
}
}
for _, bucket := range append(append([]string{"postgres_bound"}, ProvisionerReads...), ProvisionerReadsWhole...) {
if _, err := control.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: bucket}); err != nil {
t.Fatal(err)
}
}
put(control, AssignmentsBucket, map[string]string{"g14.dunst": "{}", "shanks.dunst": "{}"})
provider, err := jetstream.New(connect("novox.postgres", nil))
if err != nil {
t.Fatal(err)
}
put(provider, "postgres_bound", map[string]string{"g14.gitea.postgres-database": "{}",
"shanks.photos.postgres-database": "{}"})
config, err := jetstream.New(connect("novox.mesh-config", nil))
if err != nil {
t.Fatal(err)
}
put(config, ConfigResolvedBucket, map[string]string{"g14.dunst.font-size": "13", "shanks.dunst.font-size": "11"})
refusals := make(chan error, 16)
clientConn := connect("provisioner.g14", refusals)
client, err := jetstream.New(clientConn)
if err != nil {
t.Fatal(err)
}
resolved, err := client.KeyValue(ctx, ConfigResolvedBucket)
if err != nil {
t.Fatalf("the client cannot bind to the config domain's bucket: %v", err)
}
// Its own machine's key, read and watched; the watch delivers the current value first.
if e, err := resolved.Get(ctx, "g14.dunst.font-size"); err != nil || string(e.Value()) != "13" {
t.Fatalf("its own machine's key: %v %v", e, err)
}
w, err := resolved.Watch(ctx, "g14.>")
if err != nil {
t.Fatalf("watching its own machine's keys: %v", err)
}
select {
case e := <-w.Updates():
if e == nil || e.Key() != "g14.dunst.font-size" {
t.Fatalf("the watch delivered %v first", e)
}
case <-time.After(5 * time.Second):
t.Fatal("the watch of its own machine's keys delivered nothing")
}
_ = w.Stop()
assignments, err := client.KeyValue(ctx, AssignmentsBucket)
if err != nil {
t.Fatal(err)
}
if _, err := assignments.Get(ctx, "g14.dunst"); err != nil {
t.Fatalf("its own machine's placement: %v", err)
}
// Another machine's key: refused by the server, read or watched, and every key at once too; and a write of
// a domain's bucket, which is the domain's alone. Each refusal is the server's word, not a missing key.
drain := func() {
for {
select {
case <-refusals:
default:
return
}
}
}
serverRefuses := func(what string, act func(context.Context) error) {
t.Helper()
drain()
short, done := context.WithTimeout(ctx, 2*time.Second)
defer done()
if err := act(short); err == nil {
t.Errorf("the client may %s", what)
}
select {
case err := <-refusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("%s: the server said %v, want a permissions violation", what, err)
}
case <-time.After(time.Second):
t.Errorf("%s: the server did not refuse it", what)
}
}
serverRefuses("read another machine's config", func(c context.Context) error {
_, err := resolved.Get(c, "shanks.dunst.font-size")
return err
})
// Another machine's placement is read, since a provider's consumers are bound to it in their own machines'
// plans (ADR 0373, issue 679), and never written.
if _, err := assignments.Get(ctx, "shanks.dunst"); err != nil {
t.Fatalf("another machine's plan, which binds its consumers to a provider here: %v", err)
}
serverRefuses("write the assignments mirror", func(c context.Context) error {
_, err := assignments.PutString(c, "shanks.dunst", "{}")
return err
})
// A provider's entries: its own machine's consumers' keys alone (issue 679).
entries, err := client.KeyValue(ctx, "postgres_bound")
if err != nil {
t.Fatalf("the client cannot bind to a provider's bucket of entries: %v", err)
}
if _, err := entries.Get(ctx, "g14.gitea.postgres-database"); err != nil {
t.Fatalf("its own machine's consumer's entry: %v", err)
}
serverRefuses("read another machine's consumer's entry", func(c context.Context) error {
_, err := entries.Get(c, "shanks.photos.postgres-database")
return err
})
serverRefuses("write a provider's entry", func(c context.Context) error {
_, err := entries.PutString(c, "g14.gitea.postgres-database", "{}")
return err
})
for _, keys := range []string{"shanks.>", ">"} {
serverRefuses("watch "+keys, func(c context.Context) error {
w, err := resolved.Watch(c, keys)
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
select {
case e := <-w.Updates():
if e != nil {
return nil
}
return errors.New("nothing delivered")
case <-c.Done():
return c.Err()
}
})
}
serverRefuses("write the config domain's bucket", func(c context.Context) error {
_, err := resolved.PutString(c, "g14.dunst.font-size", "40")
return err
})
// What it applied, on its own machine and no other's; the vault asked in its own name and no other's.
refused := func(subject string, want bool) {
t.Helper()
drain()
if err := clientConn.Publish(subject, []byte(`{}`)); err != nil {
t.Fatal(err)
}
_ = clientConn.Flush()
select {
case err := <-refusals:
if !want {
t.Errorf("%s was refused: %v", subject, err)
} else if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("%s: the server said %v, want a permissions violation", subject, err)
}
case <-time.After(500 * time.Millisecond):
if want {
t.Errorf("the server did not refuse %s", subject)
}
}
}
refused("prov.applied.g14.dunst", false)
refused("prov.applied.shanks.dunst", true)
refused("mesh.seat.mesh-vault.call.get.provisioner~g14", false)
refused("mesh.seat.mesh-vault.call.get.g14~node-tools", true)
refused("mesh.seat.mesh-vault.call.accept.provisioner~g14", true)
refused("mesh.mod.dunst.tool.anything", true)
}
// **On a real server, as composed** (novox/hq ADR 0370 §3, issue 685): a machine's tool runner asks its own
// machine's provisioner client and is answered; the server refuses the ask from another machine's tool runner, from
// a module's own account and from the client of another machine, and refuses the client what another machine's
// tool runner would hear.
func TestAServerComposedFromTheGrantsLetsOnlyItsOwnToolRunnerAskTheProvisionerClient(t *testing.T) {
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
accounts, err := ComposeAccounts([]Principal{
{Kind: KindController, PasswordHash: string(hash)},
{Kind: KindNodeTools, Node: "g14", Module: RuntimeModule, PasswordHash: string(hash),
Carries: []Declared{{Module: "claude-code"}}},
{Kind: KindNodeTools, Node: "shanks", Module: RuntimeModule, PasswordHash: string(hash),
Carries: []Declared{{Module: "claude-code"}}},
{Kind: KindModule, Node: "g14", Module: "dunst", PasswordHash: string(hash)},
{Kind: KindProvisioner, Node: "g14", Module: "mesh-provisioner", BoundBuckets: []string{"postgres_bound"},
PasswordHash: string(hash)},
{Kind: KindProvisioner, Node: "shanks", Module: "mesh-provisioner", PasswordHash: string(hash)},
})
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
defer s.Shutdown()
connect := func(user string, refusals chan error) *nats.Conn {
t.Helper()
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) {
if refusals != nil {
refusals <- err
}
}))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
return nc
}
clientRefusals := make(chan error, 8)
client := connect("provisioner.g14", clientRefusals)
asked := make(chan string, 8)
if _, err := client.Subscribe(ProvisionerAskSubject("g14"), func(m *nats.Msg) {
asked <- string(m.Data)
_ = m.Respond([]byte(`{"value":"answered"}`))
}); err != nil {
t.Fatal(err)
}
if err := client.Flush(); err != nil {
t.Fatal(err)
}
select {
case err := <-clientRefusals:
t.Fatalf("the client may not hear its own machine's asks: %v", err)
case <-time.After(300 * time.Millisecond):
}
// Its own machine's tool runner asks and is answered.
runner := connect("g14.node-tools", nil)
reply, err := runner.Request(ProvisionerAskSubject("g14"), []byte(`{"module":"claude-code"}`), 5*time.Second)
if err != nil {
t.Fatalf("g14's tool runner cannot ask g14's client: %v", err)
}
if string(reply.Data) != `{"value":"answered"}` {
t.Fatalf("answered %s", reply.Data)
}
<-asked
// Anybody else's ask is refused by the server, and never reaches the client.
for _, user := range []string{"shanks.node-tools", "g14.dunst", "provisioner.shanks", "controller"} {
refusals := make(chan error, 4)
nc := connect(user, refusals)
if err := nc.Publish(ProvisionerAskSubject("g14"), []byte(`{"module":"claude-code"}`)); err != nil {
t.Fatal(err)
}
_ = nc.Flush()
select {
case err := <-refusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("%s: the server said %v, want a permissions violation", user, err)
}
case <-time.After(time.Second):
t.Errorf("the server did not refuse %s an ask of g14's client", user)
}
select {
case body := <-asked:
t.Errorf("g14's client heard %s from %s", body, user)
case <-time.After(200 * time.Millisecond):
}
}
// What moved is said under its own machine's name, heard by that machine's tool runner; another machine's name
// is refused it.
told := make(chan string, 4)
if _, err := runner.Subscribe("prov.changed.g14.>", func(m *nats.Msg) { told <- m.Subject }); err != nil {
t.Fatal(err)
}
_ = runner.Flush()
if err := client.Publish(ProvisionerChangedSubject("g14", "claude-code"), []byte(`{"kind":"config"}`)); err != nil {
t.Fatal(err)
}
select {
case subject := <-told:
if subject != "prov.changed.g14.claude-code" {
t.Errorf("heard %s", subject)
}
case <-time.After(5 * time.Second):
t.Error("g14's tool runner did not hear what g14's client said moved")
}
if err := client.Publish(ProvisionerChangedSubject("shanks", "claude-code"), []byte(`{}`)); err != nil {
t.Fatal(err)
}
_ = client.Flush()
select {
case err := <-clientRefusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("the server said %v, want a permissions violation", err)
}
case <-time.After(time.Second):
t.Error("the server did not refuse g14's client a word in shanks's name")
}
}
@@ -0,0 +1,217 @@
package broker
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"golang.org/x/crypto/bcrypt"
)
// **On a real server, as composed** (design 57 package P4, issue 670): the provisioner server's buckets are
// written by the server's own account whole, and by each machine's provisioner client for its own machine's keys
// alone (ADR 0349 clause 3). The server refuses a client another machine's needs, applied or items, and the
// registry; and refuses every other account a write, a provider reading the needs included.
func TestOnlyTheServerAndEachClientForItsOwnMachineWriteTheProvisionersBuckets(t *testing.T) {
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
pw := string(hash)
accounts, err := ComposeAccounts([]Principal{
{Kind: KindController, PasswordHash: pw},
{Kind: KindModule, Node: "novox", Module: ProvisionerServerModule,
State: []string{"registry", "needs", "applied", "collect"}, PasswordHash: pw},
// A provider learns its consumers from the needs (ADR 0349 clause 12): it reads them, and writes none.
{Kind: KindModule, Node: "novox", Module: "postgres", Reads: []string{"mesh-provisioner.needs"}, PasswordHash: pw},
{Kind: KindModule, Node: "g14", Module: "dunst", PasswordHash: pw},
{Kind: KindProvisioner, Node: "g14", Module: ProvisionerServerModule, PasswordHash: pw},
{Kind: KindProvisioner, Node: "shanks", Module: ProvisionerServerModule, PasswordHash: pw},
})
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
defer s.Shutdown()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
type account struct {
js jetstream.JetStream
refusals chan error
}
connect := func(user string) account {
t.Helper()
refusals := make(chan error, 16)
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refusals <- err }))
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
return account{js: js, refusals: refusals}
}
bucket := func(a account, name string) jetstream.KeyValue {
t.Helper()
kv, err := a.js.KeyValue(ctx, name)
if err != nil {
t.Fatalf("binding to %s: %v", name, err)
}
return kv
}
// The controller makes the buckets from the server's manifest, as it does for every module's state.
control := connect("controller")
for _, name := range append([]string{ProvisionerRegistryBucket}, ProvisionerWrites...) {
if _, err := control.js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name}); err != nil {
t.Fatal(err)
}
}
// The server writes all four, every machine's keys.
srv := connect("novox." + ProvisionerServerModule)
for name, key := range map[string]string{ProvisionerRegistryBucket: "config", ProvisionerNeedsBucket: "shanks.dunst.config.font-size",
ProvisionerAppliedBucket: "shanks.dunst", ProvisionerCollectBucket: "shanks.zsh-snippets.git"} {
if _, err := bucket(srv, name).PutString(ctx, key, `{}`); err != nil {
t.Fatalf("the server writing %s in %s: %v", key, name, err)
}
}
// A client writes, reads and watches its own machine's keys, and reads the registry whole.
g14 := connect("provisioner.g14")
own := map[string]string{ProvisionerNeedsBucket: "g14.dunst.config.font-size", ProvisionerAppliedBucket: "g14.dunst",
ProvisionerCollectBucket: "g14.zsh-snippets.git"}
for name, key := range own {
kv := bucket(g14, name)
if _, err := kv.PutString(ctx, key, `{}`); err != nil {
t.Fatalf("the client writing its own machine's %s in %s: %v", key, name, err)
}
if e, err := kv.Get(ctx, key); err != nil || string(e.Value()) != `{}` {
t.Fatalf("the client reading back its own %s in %s: %v %v", key, name, e, err)
}
w, err := kv.Watch(ctx, "g14.>")
if err != nil {
t.Fatalf("the client watching its own machine's keys in %s: %v", name, err)
}
select {
case e := <-w.Updates():
if e == nil || e.Key() != key {
t.Fatalf("the watch of %s delivered %v first", name, e)
}
case <-time.After(5 * time.Second):
t.Fatalf("the watch of its own machine's keys in %s delivered nothing", name)
}
_ = w.Stop()
if err := kv.Delete(ctx, key); err != nil {
t.Fatalf("the client deleting its own %s in %s (a module taken off its machine): %v", key, name, err)
}
}
registry := bucket(g14, ProvisionerRegistryBucket)
if e, err := registry.Get(ctx, "config"); err != nil || e == nil {
t.Fatalf("the client reading the registry: %v %v", e, err)
}
// Each refusal is the server's word, not a missing key.
serverRefuses := func(a account, what string, act func(context.Context) error) {
t.Helper()
for drained := false; !drained; {
select {
case <-a.refusals:
default:
drained = true
}
}
short, done := context.WithTimeout(ctx, 2*time.Second)
defer done()
if err := act(short); err == nil {
t.Errorf("%s was let through", what)
}
select {
case err := <-a.refusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("%s: the server said %v, want a permissions violation", what, err)
}
case <-time.After(time.Second):
t.Errorf("%s: the server did not refuse it", what)
}
}
put := func(kv jetstream.KeyValue, key string) func(context.Context) error {
return func(c context.Context) error { _, err := kv.PutString(c, key, `{}`); return err }
}
for name, key := range map[string]string{ProvisionerNeedsBucket: "shanks.dunst.config.font-size",
ProvisionerAppliedBucket: "shanks.dunst", ProvisionerCollectBucket: "shanks.zsh-snippets.git"} {
kv := bucket(g14, name)
serverRefuses(g14, "g14's client writing shanks's "+key+" in "+name, put(kv, key))
serverRefuses(g14, "g14's client reading shanks's "+key+" in "+name, func(c context.Context) error {
_, err := kv.Get(c, key)
return err
})
}
serverRefuses(g14, "g14's client watching every machine's needs", func(c context.Context) error {
w, err := bucket(g14, ProvisionerNeedsBucket).Watch(c, ">")
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
select {
case e := <-w.Updates():
if e != nil {
return nil
}
return errors.New("nothing delivered")
case <-c.Done():
return c.Err()
}
})
serverRefuses(g14, "a client writing the registry", put(registry, "config"))
// Shanks's client is held to shanks's keys the same way: its write of g14's applied is refused.
shanks := connect("provisioner.shanks")
if _, err := bucket(shanks, ProvisionerAppliedBucket).PutString(ctx, "shanks.dunst", `{}`); err != nil {
t.Fatalf("shanks's client writing its own applied: %v", err)
}
serverRefuses(shanks, "shanks's client writing g14's applied", put(bucket(shanks, ProvisionerAppliedBucket), "g14.dunst"))
// A provider reads every machine's needs and writes none; a module on a client's machine writes none.
provider := connect("novox.postgres")
needs := bucket(provider, ProvisionerNeedsBucket)
if e, err := needs.Get(ctx, "shanks.dunst.config.font-size"); err != nil || e == nil {
t.Fatalf("the provider reading the needs: %v %v", e, err)
}
serverRefuses(provider, "a provider writing the needs", put(needs, "shanks.postgres.bound.db"))
dunst := connect("g14.dunst")
for _, name := range append([]string{ProvisionerRegistryBucket}, ProvisionerWrites...) {
serverRefuses(dunst, "a module on g14 writing "+name, func(c context.Context) error {
// Bound by the stream's own subject: the module is granted no bind, so its put is published raw.
_, err := dunst.js.Publish(c, "$KV."+name+".g14.dunst", []byte(`{}`))
return err
})
}
}
+229
View File
@@ -0,0 +1,229 @@
package broker
import (
"slices"
"testing"
)
// The provisioner client's own bus account (novox/hq ADR 0349 clauses 6 and 11, design 57 package P3, issue
// 669): one per machine running it, apart from the machine's tool runner, granted to read its own machine's keys
// in each domain's bucket, to say what it applied on that machine, and to ask the vault in its own name; nothing
// else.
func provisionerRecords() Records {
client := Declared{Module: "mesh-provisioner", ProvisionsNode: true}
return Records{
Nodes: []string{"g14", "shanks", "ace"},
Assigned: map[string][]Declared{
"g14": {client, {Module: RuntimeModule}},
"shanks": {client},
"ace": {{Module: "dunst"}},
},
}
}
// One account per machine that runs the client, named for the machine and apart from its tool runner's: the
// vault tells the client from an agent on that machine by the name the bus proves.
func TestEachMachineRunningTheProvisionerClientHasAnAccountOfItsOwn(t *testing.T) {
users, err := Users(provisionerRecords())
if err != nil {
t.Fatal(err)
}
var got []string
for _, u := range users {
if u.Kind == KindProvisioner {
got = append(got, u.Username())
if u.Module != "mesh-provisioner" {
t.Errorf("%s is delivered to %q, want the module that runs it", u.Username(), u.Module)
}
}
}
if want := []string{"provisioner.g14", "provisioner.shanks"}; !slices.Equal(got, want) {
t.Fatalf("the provisioner clients' accounts are %v, want %v (none for a machine that does not run it)", got, want)
}
for _, u := range users {
if u.Kind == KindNodeTools && u.Username() == "provisioner.g14" {
t.Fatal("the client shares the tool runner's account")
}
}
}
// Its whole authority, subject by subject: a read of its own machine's keys in each domain's bucket, a read and a
// write of its own machine's keys in the provisioner server's needs, applied and collections, a read of the
// server's registry (design 57 P4), a publish of what it applied on its own machine, the vault's `get` in its own
// name, its own inbox and the vault's secret-changed.
func TestTheProvisionerClientIsGrantedItsMachinesKeysAndNothingElse(t *testing.T) {
perms, err := PermissionsFor(Principal{Kind: KindProvisioner, Node: "g14", Module: "mesh-provisioner",
BoundBuckets: []string{"postgres_bound"}})
if err != nil {
t.Fatal(err)
}
var wantPub []string
for _, bucket := range []string{"mesh-config_resolved", "postgres_bound"} {
stream := "KV_" + bucket
wantPub = append(wantPub,
"$JS.API.STREAM.INFO."+stream,
"$JS.API.DIRECT.GET."+stream+".$KV."+bucket+".g14.>",
"$JS.API.CONSUMER.CREATE."+stream+".*.$KV."+bucket+".g14.>",
"$JS.API.CONSUMER.DELETE."+stream+".>",
"$JS.FC."+stream+".>")
}
// The assignments mirror whole, read and never written: a provider's consumers are bound to it in their own
// machines' plans, which its machine's client reads to fill ${consumers:<provision>} (novox/hq ADR 0373, issue
// 679). The mirror holds placement and plans, never a value of a secret.
wantPub = append(wantPub,
"$JS.API.STREAM.INFO.KV_mesh-controller_assignments",
"$JS.API.DIRECT.GET.KV_mesh-controller_assignments.>",
"$JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments.>",
"$JS.API.CONSUMER.DELETE.KV_mesh-controller_assignments.>",
"$JS.FC.KV_mesh-controller_assignments.>")
for _, bucket := range []string{"mesh-provisioner_needs", "mesh-provisioner_applied", "mesh-provisioner_collect"} {
stream := "KV_" + bucket
wantPub = append(wantPub,
"$JS.API.STREAM.INFO."+stream,
"$JS.API.DIRECT.GET."+stream+".$KV."+bucket+".g14.>",
"$JS.API.CONSUMER.CREATE."+stream+".*.$KV."+bucket+".g14.>",
"$JS.API.CONSUMER.DELETE."+stream+".>",
"$JS.FC."+stream+".>",
"$KV."+bucket+".g14.>")
}
wantPub = append(wantPub,
"$JS.API.STREAM.INFO.KV_mesh-provisioner_registry",
"$JS.API.DIRECT.GET.KV_mesh-provisioner_registry.>",
"$JS.API.CONSUMER.CREATE.KV_mesh-provisioner_registry.>",
"$JS.API.CONSUMER.DELETE.KV_mesh-provisioner_registry.>",
"$JS.FC.KV_mesh-provisioner_registry.>")
wantPub = append(wantPub, "prov.applied.g14.>", "mesh.seat.mesh-vault.call.get.provisioner~g14",
"prov.changed.g14.>")
slices.Sort(wantPub)
if !slices.Equal(perms.Publish, wantPub) {
t.Errorf("publish:\n have %q\n want %q", perms.Publish, wantPub)
}
// Its own inbox, what the vault says changed (`secret-changed`: which, by whom, never a value), so a rotated
// secret reaches its file without a new declaration (design 57 V4), and its own machine's tool runner asking
// it for a carried module's config (ADR 0370 §3), which it answers on the ask's reply.
if want := []string{"_INBOX.provisioner.g14.>", "mesh.mod.mesh-vault.event.secret-changed", "prov.ask.g14"}; !slices.Equal(perms.Subscribe, want) {
t.Errorf("subscribe:\n have %q\n want %q", perms.Subscribe, want)
}
if !perms.AllowResponses || len(perms.PublishDeny) > 0 {
t.Errorf("it answers what it is asked and needs no deny: %+v", perms)
}
// No write of any bucket but its own machine's keys of the server's needs, applied and collections: a domain
// writes its own (ADR 0349 clause 3, research 051), and the registry is the server's alone.
for _, p := range perms.Publish {
if !SubjectsOverlap(p, "$KV.>") {
continue
}
if !slices.Contains([]string{"$KV.mesh-provisioner_needs.g14.>", "$KV.mesh-provisioner_applied.g14.>",
"$KV.mesh-provisioner_collect.g14.>"}, p) {
t.Errorf("it may write a bucket: %s", p)
}
}
}
// Each provider's bucket of entries, `<module>_bound` (mesh-sdk go/provider, issue 678), is read by every machine's
// client for that machine's consumers' keys alone (`<node>.>`), to fill ${bound:<provision>:<key>} (issue 679): the
// buckets of every module the mesh places that keeps state named bound, wherever it runs, and no other.
func TestTheProvisionerClientReadsItsMachinesKeysOfEachProvidersBoundBucket(t *testing.T) {
records := provisionerRecords()
records.Assigned["ace"] = append(records.Assigned["ace"],
Declared{Module: "postgres", State: []Bucket{{Module: "postgres", Name: "bound"}, {Module: "postgres", Name: "kept"}}},
Declared{Module: "keycloak", State: []Bucket{{Module: "keycloak", Name: "bound"}}})
records.Assigned["shanks"] = append(records.Assigned["shanks"],
Declared{Module: "postgres", State: []Bucket{{Module: "postgres", Name: "bound"}}})
users, err := Users(records)
if err != nil {
t.Fatal(err)
}
for _, u := range users {
if u.Kind != KindProvisioner {
continue
}
if want := []string{"keycloak_bound", "postgres_bound"}; !slices.Equal(u.BoundBuckets, want) {
t.Errorf("%s reads the bound buckets %v, want %v", u.Username(), u.BoundBuckets, want)
}
perms, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
mine := "$JS.API.DIRECT.GET.KV_postgres_bound.$KV.postgres_bound." + u.Node + ".>"
if !slices.Contains(perms.Publish, mine) {
t.Errorf("%s may not read its machine's keys of postgres_bound", u.Username())
}
for _, p := range perms.Publish {
if SubjectsOverlap(p, "$JS.API.DIRECT.GET.KV_postgres_kept.>") || SubjectsOverlap(p, "$KV.postgres_bound.>") ||
SubjectsOverlap(p, "$JS.API.DIRECT.GET.KV_postgres_bound.$KV.postgres_bound.ace.>") {
t.Errorf("%s is granted %s: another bucket, a write, or another machine's consumers", u.Username(), p)
}
}
}
if _, err := PermissionsFor(Principal{Kind: KindProvisioner, Node: "g14", Module: "mesh-provisioner",
BoundBuckets: []string{"postgres_bound.>"}}); err == nil {
t.Error("a bound bucket that is a pattern composed")
}
}
// A machine with no usable name reaches nothing: its grants would otherwise be a pattern over every machine.
func TestAProvisionerClientNamingNoMachineIsRefused(t *testing.T) {
for _, node := range []string{"", "g14.>", "*"} {
if _, err := PermissionsFor(Principal{Kind: KindProvisioner, Node: node, Module: "mesh-provisioner"}); err == nil {
t.Errorf("a provisioner client on %q composed", node)
}
}
}
// A whole mesh composes with the clients beside every other kind: nobody is granted a second writer's subject.
func TestAMeshWithProvisionerClientsComposes(t *testing.T) {
users, err := Users(provisionerRecords())
if err != nil {
t.Fatal(err)
}
for i := range users {
users[i].PasswordHash = "x"
}
if _, err := ComposeAccounts(users); err != nil {
t.Fatal(err)
}
}
// The client is asked on a subject its own machine's tool runner alone may publish (novox/hq ADR 0370 §3, issue
// 685): across a whole mesh, no principal but g14's tool runner may ask g14's client, none but g14's client hears
// it, and the client's word of what moved is heard by g14's tool runner alone.
func TestTheProvisionerClientIsAskedByItsOwnMachinesToolRunnerAlone(t *testing.T) {
records := provisionerRecords()
records.Assigned["shanks"] = append(records.Assigned["shanks"], Declared{Module: RuntimeModule},
Declared{Module: "dunst"})
records.Assigned["g14"] = append(records.Assigned["g14"], Declared{Module: "claude-code"})
users, err := Users(records)
if err != nil {
t.Fatal(err)
}
users = append(users, Principal{Kind: KindPerson, Module: "jochen", Invokes: []string{"*"}})
var asks, hears, told []string
for _, u := range users {
perms, err := PermissionsFor(u)
if err != nil {
t.Fatalf("%s: %v", u.Username(), err)
}
if slices.ContainsFunc(perms.Publish, func(p string) bool { return SubjectsOverlap(p, ProvisionerAskSubject("g14")) }) {
asks = append(asks, u.Username())
}
if slices.ContainsFunc(perms.Subscribe, func(p string) bool { return SubjectsOverlap(p, ProvisionerAskSubject("g14")) }) {
hears = append(hears, u.Username())
}
if slices.ContainsFunc(perms.Subscribe, func(p string) bool {
return SubjectsOverlap(p, ProvisionerChangedSubject("g14", "claude-code"))
}) {
told = append(told, u.Username())
}
}
if want := []string{"g14.node-tools"}; !slices.Equal(asks, want) {
t.Errorf("g14's client may be asked by %v, want %v alone", asks, want)
}
if want := []string{"provisioner.g14"}; !slices.Equal(hears, want) {
t.Errorf("g14's client's asks are heard by %v, want %v alone", hears, want)
}
if want := []string{"g14.node-tools"}; !slices.Equal(told, want) {
t.Errorf("what g14's client says moved is heard by %v, want %v alone", told, want)
}
}
+7
View File
@@ -34,6 +34,8 @@ func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
states = append(states, link.KeyChecked)
// And a walk kept in a new state, for the delivery it walks (novox/hq ADR 0239).
states = append(states, link.KeyPlanMoved)
// And a key of a record mirror written or removed (novox/hq ADR 0289, design 54 §3; design 57 F3).
states = append(states, link.KeyAssignmentChanged, link.KeyDeclaredConfigChanged, link.KeyPredefinedChanged)
for _, event := range states {
if !slices.Contains(broker.ControllerStates, event) {
t.Errorf("the mesh states %q and its account may not publish it", event)
@@ -42,6 +44,11 @@ func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
if len(broker.ControllerStates) != len(states) {
t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
}
// The record mirrors a module may read are the ones the bus holds (design 54 §5, design 57 F3).
if !slices.Equal(catalogue.ControllerMirrors, broker.ControllerMirrors()) {
t.Errorf("the catalogue lets a module read the controller's mirrors %v and the bus holds %v",
catalogue.ControllerMirrors, broker.ControllerMirrors())
}
// **And the seat says it.** A seat carries the protocol of its role (novox/hq ADR 0129), so the
// facts the control plane states are the seat's `emits` — which is what lets anything else declare
// that it consumes them, and what the subject-agreement check reads to know they have an owner.
+23 -1
View File
@@ -211,9 +211,28 @@ func MeshStreams() []Stream {
"stream's message, which ages out with its stream (ADR 0376), and full it refuses the next copy " +
"rather than drop one it holds",
},
{
Name: ProvAppliedStream,
Subjects: []string{ProvAppliedSubjects},
Retention: RetentionLastPerSubject,
Direct: true,
MaxAge: 7 * 24 * 60 * 60,
Why: "each machine's provisioner client's newest word on each module's files, applied, waiting or " +
"failed with the reason (novox/hq ADR 0349 §4): what a walk's end and a gate wait for (ADR 0362, " +
"ADR 0349 §15), read by subject; a week, as the events are",
},
}
}
// ProvAppliedStream keeps what each machine's provisioner client last said of each module's files, newest per
// subject (novox/hq ADR 0349 §4, ADR 0362, design 57 P6): `prov.applied.<node>.<module>` lies outside `mesh.*`, so
// no other stream captures it, and a report said while no controller listened is still there to be read.
const ProvAppliedStream = "PROV_APPLIED"
// ProvAppliedSubjects is every machine's client's reports: `prov.applied.<node>.<module>`, a module's name dots and
// all (ProvisionerAppliedSubject).
const ProvAppliedSubjects = "prov.applied.>"
// DeliverSubjectFor is where a push consumer's messages land.
//
// **Per consumer, which means per stream as well as per name** (novox/hq 04-ISSUES/146). A push
@@ -326,7 +345,10 @@ var ControllerStates = []string{"applied", "refused", "built-before",
// the pull request's status, and anybody else may read.
"checked",
// And a walk kept in a new state (novox/hq ADR 0239): what the delivery it walks reads its steps from.
"plan-moved"}
"plan-moved",
// And a key of a record mirror written or removed (novox/hq ADR 0289, design 54 §3): the assignments and
// each registered module's declared config (design 57 F3).
"assignment-changed", "declared-config-changed", "predefined-changed"}
// BusAdvisories are what the bus server says about the mesh's own account that the controller
// reads (novox/hq to-be 45 §3, S9): a durable consumer that handed a message over as often as it
+3
View File
@@ -131,6 +131,9 @@ func TestEachStreamCarriesTheRetentionItsShapeNeeds(t *testing.T) {
// kept until somebody acts.
"DEAD_LETTER_NOTICES": RetentionWorkQueue,
"DEAD_LETTERS": RetentionLimits,
// What each machine's provisioner client last said of each module's files (design 57 P6): the newest
// word per module, what a walk's end reads.
"PROV_APPLIED": RetentionLastPerSubject,
}
got := map[string]Retention{}
for _, s := range MeshStreams() {
+2 -2
View File
@@ -24,7 +24,7 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_predefined.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" }
} }
@@ -34,7 +34,7 @@ accounts {
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.INFO.NODES.one", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.move-secret", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
allow_responses: { max: 1, ttl: "1m" }
} }
{ user: "one.nats", password: "$2a$11$bbbbbbbbbbbbbbbbbbbbbb", permissions: {
+96 -2
View File
@@ -2,6 +2,7 @@ package broker
import (
"fmt"
"slices"
"sort"
)
@@ -68,6 +69,10 @@ type Declared struct {
// RunsAs is the account the module runs as in a runtime of its own (novox/hq ADR 0259 §8): it is never
// carried by the machine's runtime, and reaches the bus on its own account.
RunsAs string
// ProvisionsNode says the module runs the machine's provisioner client: it claims NodeProvisionerSeat and
// declares the own secret ProvisionerSecret its account is delivered in (novox/hq ADR 0349 clause 11, design 57
// P3). The client is given a bus account of its own, never the module's or the tool runner's.
ProvisionsNode bool
}
// Records is what composing a user list needs to know about the mesh, and nothing more.
@@ -97,7 +102,7 @@ type Records struct {
// is a mesh that cannot be told anything, and there is no state of the records in which that is
// correct.
func Users(r Records) ([]Principal, error) {
out := []Principal{{Kind: KindController, Uses: asksSeatOf(r), AsksIssued: issuedOf(r)}}
out := []Principal{{Kind: KindController, Uses: asksSeatOf(r), AsksIssued: issuedOf(r), AsksTheVault: vaultVerbsOf(r)}}
for _, node := range sortedCopy(r.Nodes) {
witness := false
@@ -108,7 +113,17 @@ func Users(r Records) ([]Principal, error) {
}
checks = append(checks, d.Checks...)
}
out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness, Checks: checks})
out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness, Checks: checks,
MovesIntoTheVault: servesMoveIn(r), StatesItsKeyToTheVault: servesVerb(r, VaultNodeKey)})
// **And the machine's provisioner client, on an account of its own** (novox/hq ADR 0349 clause 11): one
// per machine that runs it, delivered to the module running it.
for _, d := range r.Assigned[node] {
if d.ProvisionsNode {
out = append(out, Principal{Kind: KindProvisioner, Node: node, Module: d.Module,
BoundBuckets: boundBucketsOf(r)})
break
}
}
// **Where the runtime is assigned, the machine gets one runtime principal in place of the
// runtime module's own** (novox/hq ADR 0175, to-be 38). It carries every module on the
// node: its serving grants are the union of theirs. Every other module keeps its own
@@ -141,6 +156,12 @@ func Users(r Records) ([]Principal, error) {
PerMachine: perMachineNames(d.State), KeyedReads: d.KeyedReads,
Logs: logNames(d.Logs), Streams: d.Streams, Hears: ComposedHears(d),
})
// **The vault's holder asks the desks and the issuing providers itself** (issue 723), under the
// account its own runtime runs on (ADR 0259 §8's runs-as): the one module granted it.
if holdsSeat(d, VaultSeat) {
out[len(out)-1].AsksTheDesk = true
out[len(out)-1].AsksIssued = issuedOf(r)
}
}
if runtimeHere {
var carried []Declared
@@ -254,3 +275,76 @@ func asksSeatOf(r Records) []Seat {
}
return nil
}
// VaultSeat is the vault's seat: its holder alone asks the desks and the issuing providers in its own name.
const VaultSeat = "mesh-vault"
func holdsSeat(d Declared, seat string) bool {
for _, s := range d.Holds {
if s.Name == seat {
return true
}
}
return false
}
// vaultVerbsOf are the vault's intake verbs its holder serves, as the records say (issue 675): what the controller
// is granted to ask it in its own name. None while no holder serves them.
func vaultVerbsOf(r Records) []string {
seen := map[string]bool{}
var out []string
for _, node := range sortedCopy(r.Nodes) {
for _, d := range r.Assigned[node] {
for _, s := range d.Holds {
for _, v := range VerbsTheControllerAsksTheVault {
if s.Name == v.Seat && slices.Contains(s.Serves, v.Verb) && !seen[v.Verb] {
seen[v.Verb] = true
out = append(out, v.Verb)
}
}
}
}
}
sort.Strings(out)
return out
}
// servesMoveIn says the vault's holder serves move-in, as the records say (issue 676): what lets each machine's
// node-engine ask it in its own name.
func servesMoveIn(r Records) bool { return servesVerb(r, VaultMoveIn) }
// servesVerb says a holder of verb's seat serves it, as the records say.
func servesVerb(r Records, verb SeatVerb) bool {
for _, node := range r.Nodes {
for _, d := range r.Assigned[node] {
for _, s := range d.Holds {
if s.Name == verb.Seat && slices.Contains(s.Serves, verb.Verb) {
return true
}
}
}
}
return false
}
// boundBucketsOf is every provider's bucket of entries, `<module>_bound` (mesh-sdk go/provider, issue 678): the state
// named bound of every module placed anywhere, once each, in name order. A consumer on any machine may be bound to
// it, so each machine's provisioner client reads its own keys of each (issue 679).
func boundBucketsOf(r Records) []string {
seen := map[string]bool{}
for _, mods := range r.Assigned {
for _, d := range mods {
for _, b := range d.State {
if b.Name == BoundState {
seen[b.Bucket()] = true
}
}
}
}
out := make([]string, 0, len(seen))
for b := range seen {
out = append(out, b)
}
sort.Strings(out)
return out
}
+227
View File
@@ -0,0 +1,227 @@
package broker
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"golang.org/x/crypto/bcrypt"
)
// vaultRecords is a mesh of two machines: anchor, a desk whose runtime carries the desk's prompt (rofi holding
// node-launcher) beside an issuing provider (plex) and a module with a tool of its own; and novox, where the vault
// runs in a runtime of its own on its own account (ADR 0259 §8's runs-as).
func vaultRecords() Records {
launcher := Seat{Name: "node-launcher", Scope: "node", Serves: []string{"menu", "secret"}}
vault := Seat{Name: VaultSeat, Scope: "mesh", Serves: []string{"get", "accept", "secret-ask", "issued", "rotate"}}
return Records{Nodes: []string{"anchor", "novox"}, Assigned: map[string][]Declared{
"anchor": {{Module: RuntimeModule}, {Module: "rofi", Holds: []Seat{launcher}, NoAccount: true},
{Module: "plex", Serves: []string{"plex_issued_credential", "plex_status"},
Issued: []string{"plex_issued_credential"}, NoAccount: true},
{Module: "other", Serves: []string{"tool"}, NoAccount: true}},
"novox": {{Module: RuntimeModule}, {Module: "mesh-vault", Holds: []Seat{vault}, RunsAs: "mesh-vault"}},
}}
}
// The vault's holder may ask the desk's prompt and each issuing tool in its own name, and nothing else outside its
// seat; no other module and no runtime is granted it (issue 723).
func TestTheVaultAsksTheDeskAndTheIssuingToolsInItsOwnNameAlone(t *testing.T) {
users, err := Users(vaultRecords())
if err != nil {
t.Fatal(err)
}
perms := map[string]Permissions{}
for _, u := range users {
p, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
perms[u.Username()] = p
}
vault, ok := perms["novox.mesh-vault"]
if !ok {
t.Fatal("the vault has no account of its own")
}
token := CallerToken("novox.mesh-vault")
for _, s := range []string{
"mesh.seat.node-launcher.call.secret.anchor." + token,
"mesh.mod.plex.call.plex_issued_credential.anchor." + token,
} {
if !MayPublish(vault, s) {
t.Errorf("the vault may not ask %s", s)
}
}
for _, s := range []string{
"mesh.seat.node-launcher.tool.secret.anchor",
"mesh.seat.node-launcher.call.secret.anchor.controller",
"mesh.seat.node-launcher.call.menu.anchor." + token,
"mesh.mod.rofi.call.node-launcher.secret.anchor." + token,
"mesh.mod.plex.call.plex_status.anchor." + token,
"mesh.mod.plex.tool.plex_issued_credential.anchor",
"mesh.mod.other.call.tool.anchor." + token,
"mesh.mod.other.tool.tool.anchor",
"mesh.seat.mesh-controller.call.status." + token,
} {
if MayPublish(vault, s) {
t.Errorf("the vault may publish %s", s)
}
}
// Outside its own namespace and its own seat, the vault publishes nothing but those two calls, its inbox's
// answers and the stream API's reads of what it keeps.
for _, s := range vault.Publish {
switch {
case strings.HasPrefix(s, "mesh.mod.mesh-vault."), strings.HasPrefix(s, "mesh.seat."+VaultSeat+"."),
s == "mesh.seat.node-launcher.call.secret.*."+token,
s == "mesh.mod.plex.call.plex_issued_credential.*."+token,
strings.HasPrefix(s, "$JS."), strings.HasPrefix(s, "$KV."), strings.HasPrefix(s, "_INBOX."),
strings.HasPrefix(s, "mesh.discover."), strings.HasPrefix(s, "$SRV."):
default:
t.Errorf("the vault is granted %s, outside its seat", s)
}
}
// Nobody else is granted the vault's calls; the controller keeps today's path until R1 (issue 682).
for name, p := range perms {
if name == "novox.mesh-vault" {
continue
}
for _, s := range []string{"mesh.seat.node-launcher.call.secret.anchor." + token,
"mesh.mod.plex.call.plex_issued_credential.anchor." + token} {
if MayPublish(p, s) {
t.Errorf("%s may call in the vault's name: %s", name, s)
}
}
}
controller := perms["controller"]
for _, s := range []string{"mesh.seat.node-launcher.tool.secret.anchor",
"mesh.mod.plex.call.plex_issued_credential.anchor.controller"} {
if !MayPublish(controller, s) {
t.Errorf("the controller lost today's path %s before R1", s)
}
}
if MayPublish(perms["anchor.node-tools"], "mesh.seat.node-launcher.call.secret.anchor."+CallerToken("anchor.node-tools")) {
t.Error("a machine's runtime may ask the desk's prompt")
}
}
// **On a real server, as composed** (issue 723): the vault's account asks the desk's prompt and an issuing tool
// and is answered, naming itself; the server refuses it any other call; and the controller is still answered on
// today's path. That only sealed values cross is the vault's test (mesh-catalog, mesh-vault intake_test.go).
func TestOnARealBusTheVaultsAccountIsAnsweredByTheDeskAndTheProviderAndNothingElse(t *testing.T) {
users, err := Users(vaultRecords())
if err != nil {
t.Fatal(err)
}
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
for i := range users {
users[i].PasswordHash = string(hash)
}
accounts, err := ComposeAccounts(users)
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
defer s.Shutdown()
// anchor's runtime serves the desk's prompt and plex's issuing tool, on every subject it is granted, and says
// which subject it heard.
holder, err := nats.Connect(s.ClientURL(), nats.UserInfo("anchor.node-tools", "pw"))
if err != nil {
t.Fatal(err)
}
defer holder.Close()
heard := make(chan string, 8)
answer := func(m *nats.Msg) {
heard <- m.Subject
_ = m.Respond([]byte(`{"result":{"sealed":"c2VhbGVkLXRvLXRoZS1jYWxs"}}`))
}
for _, subject := range []string{"mesh.seat.node-launcher.call.secret.anchor.*",
"mesh.seat.node-launcher.tool.secret.anchor", "mesh.mod.plex.call.plex_issued_credential.anchor.*",
"mesh.mod.other.call.tool.anchor.*", "mesh.seat.node-launcher.call.menu.anchor.*"} {
sub, err := holder.Subscribe(subject, answer)
if err != nil {
t.Fatal(err)
}
_ = holder.Flush()
if !sub.IsValid() {
t.Fatalf("the desk's runtime may not hear %s", subject)
}
}
refusals := make(chan error, 16)
vault, err := nats.Connect(s.ClientURL(), nats.UserInfo("novox.mesh-vault", "pw"),
nats.CustomInboxPrefix("_INBOX.novox.mesh-vault"),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refusals <- err }))
if err != nil {
t.Fatalf("the vault's own account cannot connect: %v", err)
}
defer vault.Close()
token := CallerToken("novox.mesh-vault")
for _, subject := range []string{"mesh.seat.node-launcher.call.secret.anchor." + token,
"mesh.mod.plex.call.plex_issued_credential.anchor." + token} {
if _, err := vault.Request(subject, []byte(`{"seal_to":"a2V5LW9mLXRoZS1jYWxs"}`), 3*time.Second); err != nil {
t.Fatalf("the vault's call %s was not answered: %v", subject, err)
}
if got := <-heard; got != subject {
t.Fatalf("the holder heard %s, want %s", got, subject)
}
}
for _, subject := range []string{"mesh.seat.node-launcher.tool.secret.anchor",
"mesh.seat.node-launcher.call.secret.anchor.controller",
"mesh.seat.node-launcher.call.menu.anchor." + token,
"mesh.mod.other.call.tool.anchor." + token} {
if err := vault.Publish(subject, []byte(`{}`)); err != nil {
t.Fatal(err)
}
_ = vault.Flush()
select {
case err := <-refusals:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("%s: the server said %v, want a permissions violation", subject, err)
}
case <-time.After(3 * time.Second):
t.Errorf("the server did not refuse the vault %s", subject)
}
}
select {
case got := <-heard:
t.Errorf("a call the vault may not make reached the holder: %s", got)
case <-time.After(200 * time.Millisecond):
}
// The controller is still answered on today's path, for the modules filled the old way (until R1).
controller, err := nats.Connect(s.ClientURL(), nats.UserInfo("controller", "pw"),
nats.CustomInboxPrefix("_INBOX.controller"))
if err != nil {
t.Fatal(err)
}
defer controller.Close()
if _, err := controller.Request("mesh.seat.node-launcher.tool.secret.anchor", []byte(`{}`), 3*time.Second); err != nil {
t.Errorf("the controller's call to the desk on today's path: %v", err)
}
<-heard
}
+27
View File
@@ -77,6 +77,13 @@ func ownModule(p Principal, pattern string) bool {
return false
}
// ownProvisioner is a machine's provisioner client saying a report about its own machine: the third token of
// `prov.applied.<node>.…` is its machine (novox/hq ADR 0349 §4, design 57 P3).
func ownProvisioner(p Principal, pattern string) bool {
tokens := strings.Split(pattern, ".")
return p.Kind == KindProvisioner && len(tokens) > 3 && tokens[2] == p.Node
}
// kvOf is a bucket's write subjects.
func kvOf(bucket string) []string { return []string{"$KV." + bucket + ".>"} }
@@ -158,6 +165,26 @@ var WritersTable = []WriterRow{
{State: "a message a consumer gave up on", Writer: "controller", KeptIn: "the bus, the stream " + DeadLettersStream,
Others: "read, delivered again or dropped through the controller's dead-letters verb",
Subjects: []string{deadLetterPrefix + ">", againPrefix + ">"}, Writes: isController},
// The record mirrors (novox/hq ADR 0289 §9, design 54 §2): rows of this table, so a grant that lets
// anything else write them is refused at composition. Built so far: the assignments and each registered
// module's declared config (design 57 F3), and the list of predefined keys kept (design 57 K3).
{State: "the assignments mirror", Writer: "controller (lease holder), from its store, only on a change",
KeptIn: "key-value " + AssignmentsBucket, Others: "read through `reads` as mesh-controller." + MirrorAssignments,
Subjects: kvOf(AssignmentsBucket), Writes: isController},
{State: "the declared-config mirror", Writer: "controller (lease holder), from its store, only on a change",
KeptIn: "key-value " + DeclaredConfigBucket, Others: "read through `reads` as mesh-controller." + MirrorDeclaredConfig,
Subjects: kvOf(DeclaredConfigBucket), Writes: isController},
{State: "the predefined-keys mirror", Writer: "controller (lease holder), from its store, only on a change",
KeptIn: "key-value " + PredefinedBucket, Others: "read through `reads` as mesh-controller." + MirrorPredefined,
Subjects: kvOf(PredefinedBucket), Writes: isController},
// What a machine's provisioner client placed of a module's files (novox/hq ADR 0349 §4, design 57 P6): its own
// machine's word alone, which a walk's end and a gate wait for (ADR 0362, ADR 0349 §15). Kept newest per module
// in the stream ProvAppliedStream, and by the client in the provisioner server's `applied` bucket.
{State: "a module's files placed on a machine", Writer: "that machine's provisioner client",
KeptIn: "the bus, the stream " + ProvAppliedStream + ", newest per module; and key-value " +
ProvisionerAppliedBucket,
Others: "the controller reads it at a walk's end and a gate; a failure is that machine's failed apply",
Subjects: []string{"prov.applied.*.>"}, Writes: ownProvisioner},
}
// CheckWriters refuses a grant that lets a principal publish on a subject the writers table gives
+6
View File
@@ -34,6 +34,12 @@ var designRows = []string{
"the operator-channel's open messages",
"the facts snapshot",
"a message a consumer gave up on",
// The record mirrors, rows of to-be 45 §1's table by design 54 §2 (ADR 0289 §9).
"the assignments mirror",
"the declared-config mirror",
"the predefined-keys mirror",
// The provisioner client's word on what it placed (novox/hq ADR 0349 §4, design 57 P6).
"a module's files placed on a machine",
}
func TestTheWritersTableIsTheDesigns(t *testing.T) {
+64 -3
View File
@@ -108,6 +108,17 @@ type Result struct {
// recipe and the package it compiles; for any other, nothing is said and the whole of what the build
// sees stays its source, as before.
Sources []BuildSource
// Predefined is the list of predefined keys at the root of the repository at the commit built (novox/hq ADR
// 0348 §7), which the manifest was judged against; nil for a repository that holds none.
Predefined *Predefined
}
// Predefined is the list of predefined keys a build read: its text, canonical (catalogue.PredefinedList), and
// the generation of the commit built — how many commits it stands on, which orders two builds of one trunk.
type Predefined struct {
Keys []byte
Generation int
}
// BuildSource is the build source a build read in one repository (novox/hq ADR 0267).
@@ -265,14 +276,38 @@ func build(ctx context.Context, run Runner, publish Publisher,
"%s has no %s at %s, so there is nothing saying what it is: %w",
repository, ManifestName, describe(path), err)
}
manifest, err := catalogue.ParseManifest(raw)
// **The list of predefined keys at the commit built** (novox/hq ADR 0348 §7): predefined.json at the
// repository's root, read strictly; a list that is wrong fails the build, and the manifest is judged against
// the list of its own tree, never against one the mesh published from another commit.
predefined, err := catalogue.ReadPredefined(tree)
if err != nil {
say("predefined", "INVALID: %v", err)
return Result{}, err
}
manifest, err := catalogue.ParseManifestWith(raw, predefined)
if err != nil {
say("manifest", "INVALID: %v", err)
return Result{}, err
}
var listed *Predefined
if predefined != nil {
text, err := predefined.Canonical()
if err != nil {
return Result{}, err
}
listed = &Predefined{Keys: text, Generation: generationOf(ctx, run, tree, commit)}
say("predefined", "%d key(s) at %s, generation %d", len(predefined), short(commit), listed.Generation)
}
// What is judged only where a definition is written (novox/hq ADR 0326 §3, issue 574): a build is
// where a changed definition is first read, so it is refused here before anything is made of it.
if problems := catalogue.WrittenSettingsProblems(manifest); len(problems) > 0 {
// Judged with each template's text in its file, as written in the tree (novox/hq ADR 0349 §2): a template the
// module's directory lacks is refused here, before anything is built.
judged, err := catalogue.WithTemplatesFrom(manifest, within)
if err != nil {
say("manifest", "INVALID: %v", err)
return Result{}, err
}
if problems := catalogue.WrittenSettingsProblems(judged); len(problems) > 0 {
err := fmt.Errorf("%s", strings.Join(problems, "; "))
say("manifest", "INVALID: %v", err)
return Result{}, err
@@ -288,6 +323,11 @@ func build(ctx context.Context, run Runner, publish Publisher,
if src.tree, err = gitTree(ctx, run, tree, path); err != nil {
src.notPinned("its tree could not be named: " + err.Error())
}
// A module falling back to a predefined key was judged against the list, so a change to the list is a change
// to it: the planner rebuilds it, and its build publishes the list (ADR 0348 §7).
if predefined != nil && manifest.NamesPredefined() {
src.rootHas(catalogue.PredefinedFile)
}
// A build-time credential, written into the build context as .npmrc, but ONLY for a module that
// asks for it: a `package` artifact (which publishes), or an image whose Dockerfile COPYs .npmrc.
@@ -419,6 +459,11 @@ func build(ctx context.Context, run Runner, publish Publisher,
}
}
// Each template's text, read back from the archive the build made or took (novox/hq ADR 0349 §2).
if built, err = withTemplateTexts(manifest, within, built); err != nil {
say("templates", "FAILED: %v", err)
return Result{}, err
}
resolved, err := manifest.Resolve(built)
if err != nil {
say("resolve", "FAILED: %v", err)
@@ -448,7 +493,23 @@ func build(ctx context.Context, run Runner, publish Publisher,
return Result{Manifest: resolved, Commit: commit, Built: built,
Against: against(within, manifest, stoodOn), Read: readBy(manifest), Source: fingerprint,
Trunk: trunk, OnTrunk: onTrunk, Branches: branches, Sources: sources,
ReuseKey: key.Key, ReuseParts: key.Parts, NoKey: key.Why, Reused: reused, Why: builtWhy}, nil
ReuseKey: key.Key, ReuseParts: key.Parts, NoKey: key.Why, Reused: reused, Why: builtWhy,
Predefined: listed}, nil
}
// generationOf is how many commits the commit built stands on, itself included: of two commits on one trunk the
// later has more, so the controller keeps the list of the newer one whatever order it hears them in (ADR 0348
// §7). 0 when git could not say.
func generationOf(ctx context.Context, run Runner, tree, commit string) int {
out, err := run(ctx, tree, "git", "rev-list", "--count", commit)
if err != nil {
return 0
}
n, err := strconv.Atoi(strings.TrimSpace(out))
if err != nil || n < 0 {
return 0
}
return n
}
// takeACheckBuild is the walk's look for a check build of its key (novox/hq ADR 0350 §2, §3): the key taken by
+2
View File
@@ -83,6 +83,8 @@ func (r *recorded) run(_ context.Context, dir, name string, args ...string) (str
return r.tree + "\n", nil
case name == "git" && len(args) > 0 && args[0] == "rev-parse":
return "c0ffeec0ffeec0ffeec0ffeec0ffeec0ffeec0ff\n", nil
case name == "git" && len(args) > 1 && args[0] == "rev-list" && args[1] == "--count":
return "42\n", nil
}
return "", nil
}
+106
View File
@@ -0,0 +1,106 @@
package builder
import (
"context"
"slices"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A build reads the list of predefined keys at the root of the repository at the commit it builds (novox/hq ADR
// 0348 §7, design 57 package K2, issue 661): it refuses a list that is wrong, judges the manifest against it, and
// carries it with the commit's generation for the controller to publish.
const theFirstKeys = `{
"font-size": {"type": "number", "description": "the size text is shown in, in points", "minimum": 6, "maximum": 72},
"agent-managed-settings": {"type": "object", "description": "the managed settings of every agent session, a JSON object", "trusted": true}
}`
// fallingBack is a module of the catalogue whose font size falls back to the predefined key named.
func fallingBack(key, kind string) string {
def := "10"
if kind == "string" {
def = `"10"`
}
return `{"module":"dunst","version":"1",
"build":{"artifacts":[{"name":"look","kind":"archive","from":"files"}]},
"config":{"font-size":{"type":"` + kind + `","description":"the notification's font size, in points",
"default":` + def + `,"predefined":"` + key + `"}},
"resources":[
{"id":"theme","type":"archive","path":"/opt/dunst","artifact":"look"},
{"id":"rc","type":"file","path":"/etc/xdg/dunst/dunstrc","content":"font = Inter ${config:font-size}\n"}]}`
}
func buildInTheCatalogue(t *testing.T, manifest string, list *string) (Result, error) {
t.Helper()
r := &recorded{contents: map[string]string{"modules/dunst/" + ManifestName: manifest,
"modules/dunst/files/theme.conf": "dark", "README.md": "the catalogue"}}
if list != nil {
r.contents[catalogue.PredefinedFile] = *list
}
return Build(context.Background(), onTrunk{recorded: r}.run, r, "https://forge.invalid/mesh-catalog.git",
"modules/dunst", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
}
func TestABuildNamingAPredefinedKeyIsJudgedAgainstItsTreesListAndCarriesIt(t *testing.T) {
list := theFirstKeys
got, err := buildInTheCatalogue(t, fallingBack("font-size", "number"), &list)
if err != nil {
t.Fatalf("a module falling back to font-size did not build: %v", err)
}
if got.Predefined == nil || got.Predefined.Generation != 42 {
t.Fatalf("the list was not carried with its generation: %+v", got.Predefined)
}
carried, err := catalogue.ParsePredefined(got.Predefined.Keys)
if err != nil || len(carried) != 2 || carried["font-size"].Type != catalogue.ConfigNumber {
t.Fatalf("the list carried is %s (%v)", got.Predefined.Keys, err)
}
// A change to the list is a change to the module that was judged against it.
if len(got.Sources) == 0 || !slices.Contains(got.Sources[0].Paths, catalogue.PredefinedFile) {
t.Fatalf("its build source leaves the list out: %+v", got.Sources)
}
for _, c := range []struct{ manifest, says string }{
{fallingBack("font-scale", "number"), `"font-scale", which is not on the list`},
{fallingBack("font-size", "string"), "which is a number, and this value is a string"},
} {
if _, err := buildInTheCatalogue(t, c.manifest, &list); err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("wanted the build refused saying %q, got %v", c.says, err)
}
}
}
func TestABuildRefusesAWrongList(t *testing.T) {
wrong := `{"font-size": {"type": "number", "description": "points", "unit": "pt"}}`
if _, err := buildInTheCatalogue(t, fallingBack("font-size", "number"), &wrong); err == nil ||
!strings.Contains(err.Error(), `unknown field "unit"`) {
t.Fatalf("a wrong list was built with: %v", err)
}
}
// A repository without the list builds as before, carrying none; a value naming a key there is refused.
func TestABuildWithoutAListCarriesNone(t *testing.T) {
plain := strings.Replace(fallingBack("font-size", "number"), `,"predefined":"font-size"`, "", 1)
got, err := buildInTheCatalogue(t, plain, nil)
if err != nil {
t.Fatal(err)
}
if got.Predefined != nil {
t.Fatalf("a list was carried from a tree holding none: %+v", got.Predefined)
}
if slices.Contains(got.Sources[0].Paths, catalogue.PredefinedFile) {
t.Fatalf("the build source names a list: %+v", got.Sources)
}
if _, err := buildInTheCatalogue(t, fallingBack("font-size", "number"), nil); err == nil ||
!strings.Contains(err.Error(), "is not present") {
t.Fatalf("a key named where no list is: %v", err)
}
// A list present and a module naming no key: carried, and not the module's build source.
list := theFirstKeys
got, err = buildInTheCatalogue(t, plain, &list)
if err != nil || got.Predefined == nil || slices.Contains(got.Sources[0].Paths, catalogue.PredefinedFile) {
t.Fatalf("a module naming no key: %+v %+v %v", got.Predefined, got.Sources, err)
}
}
+14 -2
View File
@@ -74,8 +74,14 @@ const reusedModule = `{"module":"meshboard","version":"1",
// aModuleAt is a repository holding the module under mod/, whose directory git names as tree, on its trunk.
func aModuleAt(t *testing.T, tree string) (*recorded, Runner, string) {
t.Helper()
r := &recorded{tree: tree, contents: map[string]string{"mod/" + ManifestName: reusedModule,
"mod/Dockerfile": "FROM scratch", "mod/files/theme.conf": "dark"}}
return aModuleWith(t, tree, map[string]string{"mod/" + ManifestName: reusedModule,
"mod/Dockerfile": "FROM scratch", "mod/files/theme.conf": "dark"})
}
// aModuleWith is aModuleAt with the repository's files given.
func aModuleWith(t *testing.T, tree string, contents map[string]string) (*recorded, Runner, string) {
t.Helper()
r := &recorded{tree: tree, contents: contents}
run := func(ctx context.Context, dir, name string, args ...string) (string, error) {
if name == "git" && len(args) > 0 && args[0] == "symbolic-ref" {
r.ran = append(r.ran, name+" "+strings.Join(args, " "))
@@ -101,6 +107,12 @@ var aPull = PullRef{Owner: "novox", Repo: "meshboard", Number: 7}
func aCheckBuilt(t *testing.T, scratch *aScratch, tree string) CheckBuilt {
t.Helper()
r, run, workspace := aModuleAt(t, tree)
return aCheckBuiltBy(t, scratch, r, run, workspace)
}
// aCheckBuiltBy is aCheckBuilt of the repository given.
func aCheckBuiltBy(t *testing.T, scratch *aScratch, r *recorded, run Runner, workspace string) CheckBuilt {
t.Helper()
spec := CheckSpec{ID: "build-1", Owner: aPull.Owner, Repo: aPull.Repo, Number: aPull.Number,
Builds: []CheckBuildSpec{{Module: "meshboard", Repository: "https://forge.invalid/meshboard.git", Path: "mod",
Ref: "main"}}}
+20 -1
View File
@@ -58,6 +58,9 @@ type sourceInputs struct {
// whole when an artifact's is not known (novox/hq ADR 0267).
own map[string]bool
ownIsAll bool
// root is the module's build source in its own repository outside its directory, relative to the
// repository's root: the list of predefined keys a manifest naming one was judged against (ADR 0348 §7).
root map[string]bool
// read is, per context (repository and ref), the build source read there; nil for one read whole.
read map[string]map[string]bool
readAs map[string]catalogue.ArtifactContext
@@ -86,6 +89,9 @@ func (s *sourceInputs) copy() *sourceInputs {
for k, v := range s.own {
c.own[k] = v
}
for k, v := range s.root {
c.root[k] = v
}
for k, v := range s.read {
if v == nil {
c.read[k] = nil
@@ -112,7 +118,7 @@ func (s *sourceInputs) contextOffHead() {
func newSourceInputs(module string) *sourceInputs {
return &sourceInputs{module: module, contexts: map[string]string{}, toolchains: map[string]string{},
own: map[string]bool{}, read: map[string]map[string]bool{}, readAs: map[string]catalogue.ArtifactContext{},
own: map[string]bool{}, root: map[string]bool{}, read: map[string]map[string]bool{}, readAs: map[string]catalogue.ArtifactContext{},
contextsAtHead: true}
}
@@ -126,6 +132,16 @@ func (s *sourceInputs) ownHas(entries ...string) {
}
}
// rootHas adds entries, relative to the repository's root, to the module's build source in its own repository.
func (s *sourceInputs) rootHas(entries ...string) {
if s == nil {
return
}
for _, e := range entries {
s.root[e] = true
}
}
// ownWhole says an artifact's build source in the module's own repository is not known: the module's
// whole directory is its source, as it was before.
func (s *sourceInputs) ownWhole() {
@@ -178,6 +194,9 @@ func (s *sourceInputs) buildSources() []BuildSource {
for e := range s.own {
own = append(own, withPrefix(s.prefix, e))
}
for e := range s.root {
own = append(own, e)
}
sort.Strings(own)
out = append(out, BuildSource{Paths: compactSorted(own)})
}
+95
View File
@@ -0,0 +1,95 @@
package builder
import (
"archive/tar"
"bytes"
"compress/gzip"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"path/filepath"
"github.com/novox/mesh-controller/internal/catalogue"
)
// withTemplateTexts is what was built with each template's text, for a module whose files name templates (novox/hq ADR
// 0349 §2): its templates archive — the one built from its templates directory, files/ unless its manifest names
// another (issue 731) — gains the text of every file in it, read back from the packed bytes, by its path inside the
// archive. Resolve reads them into the files' content.
//
// **Read from the archive, proven by its digest.** The module's templates directory is packed again, as the build packs
// it, and the bytes are read only when they hash to the digest the build made or took: a fresh build's own archive, or
// a check's build a walk copied by digest without building (novox/hq ADR 0350 §2). Packing is deterministic, so one
// source packs to one digest; a tree that packs to another is not the archive, and its text is refused.
func withTemplateTexts(manifest catalogue.Manifest, within string, built []catalogue.Built) ([]catalogue.Built, error) {
if !manifest.Templated() {
return built, nil
}
name := catalogue.TemplatesArtifact(manifest)
at := -1
for i, b := range built {
if b.Name == name {
at = i
}
}
if name == "" || at < 0 {
return nil, fmt.Errorf("%s names templates and built no archive of %s/ to read them from",
manifest.Module, manifest.TemplatesDir())
}
body, err := pack(filepath.Join(within, filepath.FromSlash(manifest.TemplatesDir())))
if err != nil {
return nil, fmt.Errorf("%s: packing %s/ to read its templates back failed: %w", manifest.Module,
manifest.TemplatesDir(), err)
}
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
if built[at].Digest != digest {
return nil, fmt.Errorf("%s: %s/ packs to the digest %s, and the archive %q built or taken is %s: the "+
"templates are read only from the archive the mesh records", manifest.Module, manifest.TemplatesDir(),
digest, name, orNone(built[at].Digest))
}
texts, err := unpackTexts(body)
if err != nil {
return nil, fmt.Errorf("%s: reading the archive %q back failed: %w", manifest.Module, name, err)
}
out := append([]catalogue.Built(nil), built...)
out[at].Texts = texts
return out, nil
}
// unpackTexts is every file of a packed archive (pack), by its path inside it.
func unpackTexts(body []byte) (map[string]string, error) {
zipped, err := gzip.NewReader(bytes.NewReader(body))
if err != nil {
return nil, err
}
defer zipped.Close()
reader := tar.NewReader(zipped)
texts := map[string]string{}
for {
header, err := reader.Next()
if errors.Is(err, io.EOF) {
return texts, nil
}
if err != nil {
return nil, err
}
if header.Typeflag != tar.TypeReg {
continue
}
text, err := io.ReadAll(reader)
if err != nil {
return nil, err
}
texts[header.Name] = string(text)
}
}
func orNone(digest string) string {
if digest == "" {
return "no digest"
}
return digest
}
+181
View File
@@ -0,0 +1,181 @@
package builder
import (
"context"
"crypto/sha256"
"encoding/hex"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A module's files kept as template files in its repository (novox/hq ADR 0349 §2, design 57 package T1, issue
// 654): the build packs files/ into the module's templates archive and reads each template's text back from the
// packed bytes into the file's content, so the manifest the mesh holds is the one it would hold for the text inline.
const templatedModule = `{"module":"meshboard","version":"1",
"build":{"artifacts":[
{"name":"server","kind":"image","from":"Dockerfile"},
{"name":"files","kind":"archive","from":"files"}]},
"resources":[
{"id":"svc","type":"container","name":"meshboard","artifact":"server"},
{"id":"conf","type":"file","path":"/etc/meshboard/meshboard.conf","template":"files/meshboard.conf"},
{"id":"deeper","type":"file","path":"/etc/meshboard/deeper.conf","template":"files/a/deeper.conf"}]}`
var templatedFiles = map[string]string{
"Dockerfile": "FROM scratch", "files/meshboard.conf": "listen = 8080\n\"quoted\" é\n",
"files/a/deeper.conf": "deeper\n", "files/unused.conf": "packed, named by nothing",
}
func contentOf(t *testing.T, m catalogue.Manifest, id string) string {
t.Helper()
for _, r := range m.Resources {
if r["id"] == id {
if _, has := r[catalogue.TemplateField]; has {
t.Fatalf("%s still names its template in the built manifest: %v", id, r)
}
text, ok := r["content"].(string)
if !ok {
t.Fatalf("%s carries no content: %v", id, r)
}
return text
}
}
t.Fatalf("no resource %s in %v", id, m.Resources)
return ""
}
func TestABuildReadsEachTemplateBackFromItsArchive(t *testing.T) {
r, workspace := aRepository(t, templatedModule, templatedFiles)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if text := contentOf(t, got.Manifest, "conf"); text != templatedFiles["files/meshboard.conf"] {
t.Fatalf("the template's text is %q, not the file's", text)
}
if text := contentOf(t, got.Manifest, "deeper"); text != templatedFiles["files/a/deeper.conf"] {
t.Fatalf("the template's text is %q, not the file's", text)
}
if r.archives["meshboard/files"] == "" {
t.Fatalf("the templates archive was not published: %v", r.archives)
}
if problem := catalogue.HeldTemplateProblem(got.Manifest); problem != "" {
t.Fatal(problem)
}
}
func TestABuildWhoseTemplateIsMissingFailsNamingIt(t *testing.T) {
files := map[string]string{}
for k, v := range templatedFiles {
files[k] = v
}
delete(files, "files/a/deeper.conf")
r, workspace := aRepository(t, templatedModule, files)
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err == nil || !strings.Contains(err.Error(), "files/a/deeper.conf") {
t.Fatalf("a build whose template is missing did not fail naming it: %v", err)
}
}
// A walk that takes its check's build copies the archive by digest and builds nothing: the templates are read back
// from the tree packed again, which is the archive only when it packs to the digest taken.
func TestAWalkTakingAChecksBuildReadsTheTemplatesBackToo(t *testing.T) {
withVersion(t, "builder-1")
contents := map[string]string{"mod/" + ManifestName: templatedModule}
for k, v := range templatedFiles {
contents["mod/"+k] = v
}
scratch := newScratch()
r, run, workspace := aModuleWith(t, "tree-one", contents)
checked := aCheckBuiltBy(t, scratch, r, run, workspace)
r, run, workspace = aModuleWith(t, "tree-one", contents)
got, err := BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: named(checked)},
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if got.Reused == nil || len(r.archives) > 0 {
t.Fatalf("the walk built again: reused %+v, why %q, published %v", got.Reused, got.Why, r.archives)
}
if text := contentOf(t, got.Manifest, "conf"); text != templatedFiles["files/meshboard.conf"] {
t.Fatalf("the template's text is %q, not the file's", text)
}
}
// The tree packed again must be the archive taken; text read from anything else is not the archive's.
func TestTemplateTextsAreRefusedWhenTheTreeDoesNotPackToTheDigestTaken(t *testing.T) {
m, err := catalogue.ParseManifest([]byte(templatedModule))
if err != nil {
t.Fatal(err)
}
within := writeTree(t, t.TempDir(), templatedFiles)
built := []catalogue.Built{{Name: "files", Kind: catalogue.ArtifactArchive, Digest: "sha256:" + strings.Repeat("0", 64)}}
if _, err := withTemplateTexts(m, within, built); err == nil || !strings.Contains(err.Error(), "digest") {
t.Fatalf("text was read from a tree that does not pack to the archive taken: %v", err)
}
}
// writeTree writes files under dir and answers dir.
func writeTree(t *testing.T, dir string, files map[string]string) string {
t.Helper()
for name, body := range files {
full := filepath.Join(dir, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
return dir
}
// A module whose files/ is an archive unpacked onto the machine keeps its templates in a directory it names (issue
// 731): the build reads each template back from the archive packed from that directory, and files/ is packed as it
// was, its templates nowhere in it.
const unpackingModule = `{"module":"meshboard","version":"1",
"templates":"templates",
"build":{"artifacts":[
{"name":"scripts","kind":"archive","from":"files"},
{"name":"templates","kind":"archive","from":"templates"}]},
"resources":[
{"id":"scripts","type":"archive","path":"/usr/local/lib/meshboard","artifact":"scripts"},
{"id":"conf","type":"file","path":"/etc/meshboard/meshboard.conf","template":"templates/meshboard.conf"}]}`
func TestABuildReadsEachTemplateBackFromTheTemplatesDirectoryItsManifestNames(t *testing.T) {
files := map[string]string{"files/bin/meshboard-hook": "#!/bin/sh\n", "templates/meshboard.conf": "listen = 8080\n"}
r, workspace := aRepository(t, unpackingModule, files)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if text := contentOf(t, got.Manifest, "conf"); text != files["templates/meshboard.conf"] {
t.Fatalf("the template's text is %q, not the file's", text)
}
if got.Manifest.Templates != "" {
t.Fatalf("the built manifest still names its templates directory: %q", got.Manifest.Templates)
}
// files/ packs exactly as it would with no templates beside it.
alone, err := pack(writeTree(t, t.TempDir(), map[string]string{"bin/meshboard-hook": "#!/bin/sh\n"}))
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(alone)
packed := ""
for _, b := range got.Built {
if b.Name == "scripts" {
packed = b.Digest
}
}
if packed != "sha256:"+hex.EncodeToString(sum[:]) {
t.Fatalf("files/ packed to %q, not to what it packs to alone", packed)
}
}
+7 -7
View File
@@ -56,7 +56,7 @@ func anchorRendering(adopted bool) Rendering {
Generators: map[string]Generator{"overlay": hub{}},
Ports: map[string]map[int]int{"distribution": {5000: 5000}, "hello-web": {8080: 20001}},
Settings: SettingsBy{"distribution": {{From: "node anchor",
Values: map[string]any{ExposeSetting: map[string]any{"5000": FromEverywhere}}}}},
Placement: map[string]any{ExposeSetting: map[string]any{"5000": FromEverywhere}}}}},
Mesh: []string{"10.42.0.1"},
Foundation: []int{5671},
// What the machine reported faces outside, which every rule in the filter is written
@@ -310,14 +310,14 @@ func TestAGivenPortIsUsedEverywhereThePortIs(t *testing.T) {
func TestAGivenPortIsTheNodesAndReachesSomething(t *testing.T) {
store := anAdoptedAnchor().Modules[1]
node := func(v any) []Layer {
return []Layer{{From: "anchor", Values: map[string]any{PortsSetting: v}}}
return []Layer{{From: "anchor", Placement: map[string]any{PortsSetting: v}}}
}
if got, err := GivenPorts(store, node(map[string]any{"5432": float64(5433)})); err != nil ||
got[5432] != 5433 {
t.Fatalf("a node's given port was not read: %v %v", got, err)
}
if _, err := GivenPorts(store, []Layer{{From: MeshWideLayer,
Values: map[string]any{PortsSetting: map[string]any{"5432": float64(5433)}}}}); err == nil {
Placement: map[string]any{PortsSetting: map[string]any{"5432": float64(5433)}}}}); err == nil {
t.Fatal("a port given for the whole mesh was accepted")
}
if _, err := GivenPorts(store, node(map[string]any{"6000": float64(6001)})); err == nil {
@@ -409,7 +409,7 @@ func TestAGivenPortIsRefusedForAPortNoContainerPublishes(t *testing.T) {
onTheMachine := Manifest{Module: "daemon",
Listens: []Listening{{Port: 9000, From: FromMesh}}, Guards: []int{9000}}
layers := []Layer{{From: "node anchor",
Values: map[string]any{PortsSetting: map[string]any{"9000": float64(9100)}}}}
Placement: map[string]any{PortsSetting: map[string]any{"9000": float64(9100)}}}}
_, err := GivenPorts(onTheMachine, layers)
if err == nil || !strings.Contains(err.Error(), "does not publish") {
t.Fatalf("a port no container publishes was given: %v", err)
@@ -421,7 +421,7 @@ func TestAGivenPortIsRefusedForAPortNoContainerPublishes(t *testing.T) {
func TestAGuardedPortOpenedToEveryoneIsNotGuarded(t *testing.T) {
with := anchorRendering(true)
with.Settings["postgres"] = []Layer{{From: "node anchor",
Values: map[string]any{ExposeSetting: map[string]any{"5432": FromEverywhere}}}}
Placement: map[string]any{ExposeSetting: map[string]any{"5432": FromEverywhere}}}}
composed, err := anAdoptedAnchor().Compose(with)
if err != nil {
t.Fatal(err)
@@ -477,7 +477,7 @@ func portsAsThePlanWould(m Manifest, given map[int]int) map[int]int {
func TestAGivenPortNamesEitherEndOfWhatTheModulePublishes(t *testing.T) {
forge := aForge()
node := func(v any) []Layer {
return []Layer{{From: "node anchor", Values: map[string]any{PortsSetting: v}}}
return []Layer{{From: "node anchor", Placement: map[string]any{PortsSetting: v}}}
}
// The machine side — the number this module says it listens on, and the one the predecessor
@@ -573,7 +573,7 @@ func TestAGivenPortNamesEitherEndOfWhatTheModulePublishes(t *testing.T) {
func TestAGivenMachineSideReachesTheFilterTheOpeningAndTheConsumer(t *testing.T) {
forge := aForge()
given, err := GivenPorts(forge, []Layer{{From: "node anchor",
Values: map[string]any{PortsSetting: map[string]any{"2222": float64(222)}}}})
Placement: map[string]any{PortsSetting: map[string]any{"2222": float64(222)}}}})
if err != nil {
t.Fatalf("the machine side of a mapping cannot be given a port: %v", err)
}
+53
View File
@@ -0,0 +1,53 @@
package catalogue
import (
"fmt"
"regexp"
)
// What the machine's modules back up is placement (design 57 L3, issue 794). The holder of node-backup names it as
// `${backup:lines}` (what to run and which paths to keep) and `${backup:data}` (every item with its class and
// protection): the text `${contribution:node-backup:backup}` and `${contribution:node-backup:data}` render, from the
// same function, derived from each placed module's declared data (novox/hq ADR 0233). A handed holder has the text in
// its plan's placement for its client to fill; one not handed has the controller fill it where it stands.
// BackupNamespace is the placeholder kind of the machine's backup lines and data.
const BackupNamespace = "backup"
// ofBackup is where a holder places them: ${backup:<key>}. Loose inside the braces, so a misspelt key is found and
// refused rather than written out as text.
var ofBackup = regexp.MustCompile(`\$\{backup:([^}]*)\}`)
// backupKinds is the contribution kind each key of the backup placement renders.
var backupKinds = map[string]string{"lines": BackupKindBackup, "data": BackupKindData}
// backupPlacer is the machine's modules the backup placement is composed over, as the old contributions are.
type backupPlacer struct {
modules []Manifest
with Rendering
facts map[string]string
caps map[string]bool
unplaced *[]string
}
// placed is the text of one key for a holder, or what is wrong with naming it.
func (b backupPlacer) placed(holder Manifest, key string, item bool) (string, string) {
kind, known := backupKinds[key]
switch {
case item:
return "", "an item is kept per item; the machine's backup lines are placed only in a file"
case !known:
return "", fmt.Sprintf("the backup placement has `lines` and `data`, not %q", key)
case !holder.ClaimsSeat(BackupSeat):
return "", fmt.Sprintf("%s does not claim %s; the machine's backup lines are placed by its holder alone "+
"(novox/hq ADR 0212)", holder.Module, BackupSeat)
}
text, left, err := seatContributions(b.modules, holder, BackupSeat+":"+kind, b.with, b.facts, b.caps)
if b.unplaced != nil {
*b.unplaced = append(*b.unplaced, left...)
}
if err != nil {
return "", err.Error()
}
return text, ""
}

Some files were not shown because too many files have changed in this diff Show More