Compare commits

...
Author SHA1 Message Date
mesh-admin 221fe5a60d Merge pull request '57/K1: A module declares its config with type, description and validation (issue 660)' (#320) from pkg/57-k1-config-declaration into feat/57-config-and-provisioning 2026-10-14 05:25:35 +00:00
mesh-adminandClaude Opus 5.5 9aa0332694 Refuse an undeclared takes-config key, and say an unset required value in the config form's words
mesh/merge-gate head 9aa033269495 merged into trunk 60d390adeddc — pass: the change touches no module of the mesh's graph
mesh/repo-check head 9aa033269495 merged into trunk 60d390adeddc — pass: its merge-check.sh passed
mesh/delivery delivered
A key a mergeable file took had no type, validation or trusted mark, and the config verb could never set it (ADR 0348 §4). A required config value nothing sets was refused naming ${setting:} and settings set, words the module does not use. Review of mesh-controller#320 (issue 660).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:17:26 +00:00
mesh-admin 60d390aded Merge pull request '57/F2b: A settings layer no longer replaces what a provider serves (issue 696)' (#321) from pkg/57-f2b-served-keys into feat/57-config-and-provisioning 2026-10-14 05:16:03 +00:00
mesh-adminandClaude Opus 5.5 c5fe7d4795 Stop a settings layer replacing what a provider serves (issue 696)
mesh/merge-gate head c5fe7d4795d7 merged into trunk dc36536e199e — pass: the change touches no module of the mesh's graph
mesh/repo-check head c5fe7d4795d7 merged into trunk dc36536e199e — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group pkg/57-f2b-served-keys delivered: every member is delivered
What a provider serves is its definition's (ADR 0348 clause 11): a layer
fills only the keys it asks for as ${setting:<key>}. Settle replaced any
served key a layer named, so a setting for one of a provider's files could
change what every consumer was told. influxdb on ace was the one live layer
relying on it and now serves its settings (mesh-catalog, merged first), so
grafana's data source composes byte for byte as before. UnusedSettings no
longer counts a literal served key as a place a setting lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 05:03:08 +00:00
mesh-adminandClaude Opus 5.5 ae071e2b59 Prove each config refusal and the byte-identical composition of a module moved to config (issue 660)
mesh/merge-gate out of date: judged against trunk dc36536e199e, the trunk is now 60d390adeddc; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk dc36536e199e, the trunk is now 60d390adeddc; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
One registration refusal test per rule of design 57 package K1, the forms kept apart and written back as read, and dunst rewritten from settings to config composing to the same bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:39:13 +00:00
mesh-adminandClaude Opus 5.5 233956ef09 Let a module declare its config with type, description and validation (issue 660)
The config form's declaration and its registration refusals, each by name with what to do (ADR 0348, design 57 package K1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:38:04 +00:00
mesh-adminandClaude Opus 5.5 2f40ae5f76 Read the config form beside the settings form in a manifest (part of issue 660)
The manifest's hooks for the config form: read into the fields the settings form fills, written back as read, and judged by ConfigProblems; the rules are in config.go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:36:45 +00:00
mesh-admin dc36536e19 Merge pull request '57/V1: The vault's seat promises get (issue 674)' (#317) from pkg/57-v1-vault-in-go into feat/57-config-and-provisioning 2026-10-14 01:19:23 +00:00
mesh-admin 71362c196d Merge pull request '57/F2: Placement leaves the settings layers, held beside the assignment (issue 651)' (#316) from pkg/57-f2-placement-facts into feat/57-config-and-provisioning 2026-10-14 01:09:55 +00:00
mesh-admin 617881c309 Merge pull request '57/T1: A file resource may name a template file in its module's repository (issue 654)' (#313) from pkg/57-t1-template-files into feat/57-config-and-provisioning 2026-10-14 01:00:47 +00:00
mesh-adminandClaude Opus 5.5 661918f4e4 Promise get on the vault's seat, so the Go vault can serve it (issue 674)
mesh/merge-gate head 661918f4e4e0 merged into trunk 71362c196dbc — pass: the change touches no module of the mesh's graph
mesh/delivery-group group pkg/57-v1-vault-in-go delivering: 1 of 2 delivered
mesh/repo-check head 661918f4e4e0 merged into trunk 71362c196dbc — pass: its merge-check.sh passed
mesh/delivery delivered
The vault hands a secret only on request and reply, sealed to the asker's key
(hq ADR 0349 §11). The seat set is the controller's, so the verb is promised
here; optional until the vault that runs serves it (design 33 §7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:51:59 +00:00
mesh-adminandClaude Opus 5.5 def3ccefe3 Refuse templates read through a symlink anywhere under files/
mesh/delivery delivered
mesh/merge-gate head def3ccefe3c8 merged into trunk bfc79c64d989 — pass: the change touches no module of the mesh's graph
mesh/repo-check head def3ccefe3c8 merged into trunk bfc79c64d989 — pass: its merge-check.sh passed
A symlinked files/ or a symlinked directory under it let module check, the merge
gate and module add read a template from outside the repository tree, and a
symlink no template names passed them while the build's packing refuses it
(review of mesh-controller#313). Every entry under files/ must now be a plain
directory or file, and templates are read through an os.Root of the module's
directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:51:48 +00:00
mesh-adminandClaude Opus 5.5 912d76df25 Move every layer's placement out with migration 0097, and prove no machine is told anything new (issue 651)
mesh/delivery delivered
mesh/merge-gate head 912d76df25f8 merged into trunk 617881c3093d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 912d76df25f8 merged into trunk 617881c3093d — pass: its merge-check.sh passed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:45:58 +00:00
mesh-adminandClaude Opus 5.5 3048dccd0e Keep the declarations the controller composed before placement moved (issue 651)
Written by the integration branch's controller at bfc79c6 from the seed of
TestADeclarationIsTheSameAfterPlacementMoves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:44:51 +00:00
mesh-adminandClaude Opus 5.5 f0af09cfc7 Test placement read only from the placement, and raise a store before a migration (issue 651)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:44:20 +00:00
mesh-adminandClaude Opus 5.5 85b85a3fa9 Say a layer whole where the mesh reports it, and give tests their placement (issue 651)
The facts snapshot, the take preview and the settings history read a layer as it
was set; tests that place give the layer through LayerOf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:43:40 +00:00
mesh-adminandClaude Opus 5.5 02b542a059 Keep a layer's placement beside it, and say the layer whole (issue 651)
The settings verb's path splits a layer it is given into its config values and
its placement, so what a person types still works; a layer reads back and goes
into the history whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:42:37 +00:00
mesh-adminandClaude Opus 5.5 7530c2aeac Read placement from a layer's Placement, never from its config values (issue 651)
ADR 0348 clause 11: places, accesses and kept networks are placement, not config.
Today's mergeable files still take them as said, so nothing composes differently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:41:06 +00:00
mesh-adminandClaude Opus 5.5 f45e0a1de5 Let a file resource name a template file in its module's repository: composition, check and gate tests
mesh/merge-gate head f45e0a1de508 merged into trunk bfc79c64d989 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f45e0a1de508 merged into trunk bfc79c64d989 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:59:22 +00:00
mesh-adminandClaude Opus 5.5 276f6755ab Let a file resource name a template file in its module's repository: the catalogue's reading of templates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:58:16 +00:00
mesh-adminandClaude Opus 5.5 1ce4ab8c2a Let a file resource name a template file in its module's repository: the templates and their tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:57:33 +00:00
mesh-adminandClaude Opus 5.5 d43fbc202c Let a file resource name a template file in its module's repository
A file's text may live in files/<name> beside the manifest instead of inline,
so modules can move their files out of module.json (ADR 0349 clause 2, issue
654). The build packs files/ into the module's declared archive and reads each
template back from the packed bytes, proven by the digest, into content: the
manifest the mesh holds, composition and the node-engine stay as they are.
module check, the merge gate and module add read templates from the tree beside
the manifest, so a change to a template alone is judged. A file naming both
template and content, a template outside files/, a missing one, and a held
manifest that still names one are refused by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:56:43 +00:00
mesh-admin bfc79c64d9 Open the integration branch of release 57 (config and provisioning)
So the packages of hq design 57 meet here, reviewed one by one, and reach main as one delivery group.
2026-10-13 23:39:22 +00:00
47 changed files with 3411 additions and 126 deletions
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+3 -2
View File
@@ -424,8 +424,9 @@ func comparisonOf(module string, c comparison, opts takeOptions) (preview string
}
// And its settings on this machine, composed against its definition (rule 1, rule 6).
for _, layer := range c.layers {
keys := make([]string, 0, len(layer.Values))
for k := range layer.Values {
said := layer.Whole()
keys := make([]string, 0, len(said))
for k := range said {
keys = append(keys, k)
}
sort.Strings(keys)
+5
View File
@@ -55,6 +55,11 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
continue
}
m, err := catalogue.ParseManifest(raw)
if err == nil {
// Each template's text read from the tree beside the manifest, and judged as inline content is
// (novox/hq ADR 0349 §2): a missing template is refused by name.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
+2 -2
View File
@@ -348,11 +348,11 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
if layer.From == catalogue.MeshWideLayer {
if !meshWide[module] {
meshWide[module] = true
f.Settings = append(f.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Values)})
f.Settings = append(f.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Whole())})
}
continue
}
m.Settings = append(m.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Values)})
m.Settings = append(m.Settings, snapshot.Settings{Module: module, Values: scrub.Values(layer.Whole())})
}
}
m.Declaration = declarationFacts(ctx, open, n.Name, gens, gensErr, scrub)
+5
View File
@@ -576,6 +576,11 @@ func shelfWithChange(base map[string]catalogue.Manifest, sources map[string]inve
return err
}
m, err := catalogue.ParseManifest(raw)
if err == nil {
// Each template's text from the tree beside the manifest, as the build reads it back from the archive it
// packs (novox/hq ADR 0349 §2): so a change to a template alone is a change to its module.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
if err != nil {
if inScope(dir) {
failures = append(failures, fmt.Sprintf("%s: the controller judging this (%s) refuses it — %s",
+15 -1
View File
@@ -33,7 +33,21 @@ import (
func aMesh(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t) // raises the store, migrates it, and points the environment at it
licences.ForTest(t) // planning reaches this one too, by name and never by connection
return aMeshOnTheStore(t)
}
// aMeshBefore is aMesh on a store migrated only below migration `number`, with the function that runs the
// rest: for a test of what a migration that moves data does to what the mesh composes (novox/hq issue 651).
func aMeshBefore(t *testing.T, number int) (*stores, func()) {
t.Helper()
_, migrate := inventory.ForTestBefore(t, number)
return aMeshOnTheStore(t), migrate
}
// aMeshOnTheStore is the mesh aMesh makes, on the store the environment points at.
func aMeshOnTheStore(t *testing.T) *stores {
t.Helper()
licences.ForTest(t) // planning reaches this one too, by name and never by connection
open, err := openStores(t.Context())
if err != nil {
+9
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"net"
"os"
"path/filepath"
"sort"
"strings"
"time"
@@ -127,6 +128,10 @@ func moduleCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
// A manifest handed over by hand takes each template's text from the tree beside it (novox/hq ADR 0349 §2).
if m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(positionals[0])); err != nil {
return err
}
from, err := whereItComesFrom(*repo, *ref, *commit, *path, *self)
if err != nil {
return err
@@ -537,6 +542,10 @@ func settingsCommand(ctx context.Context, args []string) error {
if p.SetBy != "" {
by = "; " + p.SetBy
}
if p.Why != "" {
// A layer whose placement moved out of it (migration 0097): still the layer, as it stands.
by += "; " + p.Why
}
fmt.Printf("%s on %s, until %s (%s%s):\n %s\n", positionals[0], where,
p.ReplacedAt.Local().Format("2006-01-02 15:04:05"), p.ReplacedBy, by, shown)
}
+113
View File
@@ -0,0 +1,113 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"sort"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// **Placement leaves the settings layers and no machine is told anything new** (novox/hq ADR 0348 clause
// 11, issue 651). A mesh is seeded as the store holds it before migration 0097 — `places`, `accesses` and a
// kept network among the settings, at the mesh's level and a machine's, one layer of placement only — then
// migrated, and every machine's declaration is composed. What each machine is told of the seeded modules is
// byte for byte what the controller before this change composed from the same seed: the golden files were
// written by the integration branch's controller at bfc79c6 running this same test (MESH_WRITE_GOLDEN=1),
// where there was no migration 0097 to run.
func TestADeclarationIsTheSameAfterPlacementMoves(t *testing.T) {
open, migrate := aMeshBefore(t, 97)
ctx := t.Context()
inv := open.inventory
image := "registry.invalid:5000/library@sha256:" + strings.Repeat("a", 64)
register(t, open, catalogue.Manifest{Module: "library", Version: "1",
Accesses: []catalogue.Access{{ID: "series"}, {ID: "downloads", Mode: catalogue.AccessRead}},
Resources: []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "server", "type": "container", "image": image,
"volumes": []any{"${dir:config}:/config", "${access:series}:/series", "${access:downloads}:/downloads:ro"}},
// Today's form: a mergeable file naming no keys takes every key of the layer (ADR 0326 §3).
{"id": "runtime-config", "type": "file", "path": "${dir:config}/runtime.json", "mode": "0640",
"content": `{"log":"info"}`, "merge": "json"},
}})
register(t, open, catalogue.Manifest{Module: "found", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "image": image}}})
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
for _, l := range []struct{ node, module, values string }{
{"", "library", `{"accesses": {"downloads": "/storage/downloads"}, "log": "warn"}`},
{"anchor", "library", `{"places": {"config": {"path": "/var/lib/library/config", "owner": "1001:2000"}},
"accesses": {"series": "/storage/media/series"}, "workers": 4}`},
{"laptop", "library", `{"accesses": {"series": "/home/media/series"}}`},
{"anchor", "found", `{"networks": {"server": ["predecessor_default"]}}`},
} {
if l.node == "" {
inv.ExecForTest(t, `insert into settings (node, module, values) values (null, $1, $2)`, l.module, l.values)
continue
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.node, l.module, l.values)
}
migrate()
// Assigned once the store is migrated: an assignment reads the layers, as this controller holds them.
for _, a := range [][]string{{"anchor", "library", "found"}, {"laptop", "library"}} {
if _, err := assign(ctx, open, a[0], a[1:]...); err != nil {
t.Fatalf("assign %v: %v", a, err)
}
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatal(err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatal(err)
}
if len(composed.LeftOut) > 0 {
t.Fatalf("%s leaves out %v", node, composed.LeftOut)
}
var told []map[string]any
for _, r := range composed.Resources {
if owner := composed.Owner[r["id"].(string)]; owner == "library" || owner == "found" {
told = append(told, r)
}
}
sort.Slice(told, func(i, j int) bool { return told[i]["id"].(string) < told[j]["id"].(string) })
got, err := json.MarshalIndent(told, "", " ")
if err != nil {
t.Fatal(err)
}
golden := filepath.Join("testdata", "placement-moved", node+".json")
if os.Getenv("MESH_WRITE_GOLDEN") != "" {
if err := os.MkdirAll(filepath.Dir(golden), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(golden, append(got, '\n'), 0o644); err != nil {
t.Fatal(err)
}
continue
}
want, err := os.ReadFile(golden)
if err != nil {
t.Fatal(err)
}
if string(want) != string(got)+"\n" {
t.Errorf("%s is told something else after placement moved:\n%s\nwas:\n%s", node, got, want)
}
}
}
+204
View File
@@ -0,0 +1,204 @@
package main
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
)
// A file's text kept as a template file in its module's repository (novox/hq ADR 0349 §2, design 57 package T1,
// issue 654): `module check` and the merge gate read it from the tree beside the manifest, so a change to a
// template alone is judged as a change to its module.
const templatedNotifier = `{"module":"notifier","version":"1",
"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]},
"resources":[{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644","template":"files/notifier.conf"}]}`
// inlineNotifier is the same module as the mesh holds it: built, its text inline.
const inlineNotifier = `{"module":"notifier","version":"1",
"resources":[{"id":"conf","type":"file","path":"/etc/notifier/notifier.conf","mode":"0644","content":"volume = 3\n"}]}`
// aTemplatedModule writes a module's directory: its manifest and its files/.
func aTemplatedModule(t *testing.T, dir, manifest string, files map[string]string) string {
t.Helper()
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, "module.json")
if err := os.WriteFile(path, []byte(manifest), 0o644); err != nil {
t.Fatal(err)
}
for name, text := range files {
full := filepath.Join(dir, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
return path
}
func TestModuleCheckReadsTemplatesFromTheTreeBesideTheManifest(t *testing.T) {
good := aTemplatedModule(t, t.TempDir(), templatedNotifier, map[string]string{"files/notifier.conf": "volume = 3\n"})
var out bytes.Buffer
if err := moduleCheck([]string{good}, &out); err != nil || !strings.Contains(out.String(), "notifier: ok") {
t.Fatalf("a module with its text in files/ was refused: %v\n%s", err, out.String())
}
// A change to the template alone, to text the controller refuses inline, is refused the same.
bad := aTemplatedModule(t, t.TempDir(), templatedNotifier,
map[string]string{"files/notifier.conf": "volume = 3\nstore = ${dir:nowhere}\n"})
out.Reset()
if err := moduleCheck([]string{bad}, &out); err == nil || !strings.Contains(out.String(), "nowhere") {
t.Fatalf("a template whose text is refused inline passed:\n%s", out.String())
}
// A symlink under files/ that no template names: the build's packing refuses it, so the check does too.
linked := t.TempDir()
aTemplatedModule(t, linked, templatedNotifier, map[string]string{"files/notifier.conf": "volume = 3\n"})
if err := os.Symlink("/etc/hostname", filepath.Join(linked, "files", "elsewhere")); err != nil {
t.Fatal(err)
}
out.Reset()
if err := moduleCheck([]string{filepath.Join(linked, "module.json")}, &out); err == nil ||
!strings.Contains(out.String(), "files/elsewhere is a symlink") {
t.Fatalf("a symlink under files/ passed the check:\n%s", out.String())
}
// And a template the tree lacks, by name.
missing := aTemplatedModule(t, t.TempDir(), templatedNotifier, nil)
out.Reset()
if err := moduleCheck([]string{missing}, &out); err == nil || !strings.Contains(out.String(), "files/notifier.conf") {
t.Fatalf("a missing template was not refused by name:\n%s", out.String())
}
}
// templatedMesh is catalogueMesh with the notifier on the laptop, registered as built: its text inline.
func templatedMesh(t *testing.T) (snapshot.Facts, map[string]string) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
manifests := map[string]string{
"lemurs": `{"module":"lemurs","version":"1","capabilities":["systemd"],
"resources":[{"id":"service","type":"service","unit":"lemurs.service","state":"running","boot":"enabled"}]}`,
"notifier": inlineNotifier,
}
for name, raw := range manifests {
m, err := catalogue.ParseManifest([]byte(raw))
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if err := open.inventory.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog",
Path: "modules/" + name, BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", name); err != nil {
t.Fatalf("assign %s: %v", name, err)
}
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
return f, manifests
}
// aTemplatedTree is the catalogue's checkout with the notifier's text in files/.
func aTemplatedTree(t *testing.T, manifests map[string]string, text string, has bool) string {
t.Helper()
tree := aTree(t, withEdit(manifests, "notifier", ""))
files := map[string]string{}
if has {
files["files/notifier.conf"] = text
}
aTemplatedModule(t, filepath.Join(tree, "modules", "notifier"), templatedNotifier, files)
return tree
}
const notifierTemplate = "modules/notifier/files/notifier.conf"
func TestTheMergeGateJudgesAChangeToATemplateAlone(t *testing.T) {
f, manifests := templatedMesh(t)
// The module moved to its template, the same text: nothing changes on any machine.
v := gateJudged(t, f, aTemplatedTree(t, manifests, "volume = 3\n", true), "modules/notifier/module.json",
notifierTemplate)
if v.Verdict == "fail" || v.Modules["notifier"] != "" {
t.Fatalf("the module moved to its template with the same text reads as a change (%v):\n%s", v.Modules, v.Report())
}
// The template alone changed: its module changed, and composes.
v = gateJudged(t, f, aTemplatedTree(t, manifests, "volume = 4\n", true), notifierTemplate)
if v.Verdict == "fail" || v.Modules["notifier"] != "changed" {
t.Fatalf("a change to the template alone is not a change to its module (%v):\n%s", v.Modules, v.Report())
}
if v.Width == nil || strings.Join(v.Width.Modules, ",") != "notifier" {
t.Fatalf("a change to the template rebuilds %+v, not its module", v.Width)
}
// The template alone changed to text the controller refuses: the pull request fails, naming the module.
v = gateJudged(t, f, aTemplatedTree(t, manifests, "store = ${dir:nowhere}\n", true), notifierTemplate)
if v.Verdict != "fail" || !strings.Contains(v.Report(), "notifier") {
t.Fatalf("a template the controller refuses passed the gate:\n%s", v.Report())
}
// A symlinked files/ fails: nothing is read from outside the tree.
tree := aTree(t, withEdit(manifests, "notifier", ""))
elsewhere := t.TempDir()
if err := os.WriteFile(filepath.Join(elsewhere, "notifier.conf"), []byte("volume = 3\n"), 0o644); err != nil {
t.Fatal(err)
}
aTemplatedModule(t, filepath.Join(tree, "modules", "notifier"), templatedNotifier, nil)
if err := os.Symlink(elsewhere, filepath.Join(tree, "modules", "notifier", "files")); err != nil {
t.Fatal(err)
}
v = gateJudged(t, f, tree, notifierTemplate)
if v.Verdict != "fail" || !strings.Contains(v.Report(), "files/ is a symlink") {
t.Fatalf("a symlinked files/ passed the gate:\n%s", v.Report())
}
// And the template removed from the tree fails, naming it.
v = gateJudged(t, f, aTemplatedTree(t, manifests, "", false), notifierTemplate)
if v.Verdict != "fail" || !strings.Contains(v.Report(), "files/notifier.conf") {
t.Fatalf("a missing template passed the gate:\n%s", v.Report())
}
}
// What the gate composes for the module in its template form is what the mesh composes for it inline.
func TestTheMergeGateResolvesATemplatedModuleToItsInlineTwin(t *testing.T) {
dir := t.TempDir()
path := aTemplatedModule(t, dir, templatedNotifier, map[string]string{"files/notifier.conf": "volume = 3\n"})
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
m, err = catalogue.WithTemplatesFrom(m, dir)
if err != nil {
t.Fatal(err)
}
resolved, err := standInBuild(m)
if err != nil {
t.Fatal(err)
}
inline, err := catalogue.ParseManifest([]byte(inlineNotifier))
if err != nil {
t.Fatal(err)
}
if builtAlike(resolved) != builtAlike(inline) {
got, _ := json.Marshal(resolved)
t.Fatalf("the templated module resolves to %s, not its inline twin", got)
}
}
@@ -0,0 +1,46 @@
[
{
"id": "found.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"networks": [
"predecessor_default"
],
"type": "container"
},
{
"id": "library.access-storage/downloads",
"mode": "read",
"path": "/storage/downloads",
"type": "access"
},
{
"id": "library.access-storage/media/series",
"mode": "read",
"path": "/storage/media/series",
"type": "access"
},
{
"id": "library.config",
"mode": "0750",
"owner": "1001:2000",
"path": "/var/lib/library/config",
"type": "directory"
},
{
"content": "{\n \"accesses\": {\n \"downloads\": \"/storage/downloads\",\n \"series\": \"/storage/media/series\"\n },\n \"log\": \"warn\",\n \"places\": {\n \"config\": {\n \"owner\": \"1001:2000\",\n \"path\": \"/var/lib/library/config\"\n }\n },\n \"workers\": 4\n}\n",
"id": "library.runtime-config",
"mode": "0640",
"path": "/var/lib/library/config/runtime.json",
"type": "file"
},
{
"id": "library.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"type": "container",
"volumes": [
"/var/lib/library/config:/config",
"/storage/media/series:/series",
"/storage/downloads:/downloads:ro"
]
}
]
@@ -0,0 +1,37 @@
[
{
"id": "library.access-home/media/series",
"mode": "read",
"path": "/home/media/series",
"type": "access"
},
{
"id": "library.access-storage/downloads",
"mode": "read",
"path": "/storage/downloads",
"type": "access"
},
{
"id": "library.config",
"mode": "0750",
"path": "/var/lib/library/config",
"type": "directory"
},
{
"content": "{\n \"accesses\": {\n \"downloads\": \"/storage/downloads\",\n \"series\": \"/home/media/series\"\n },\n \"log\": \"warn\"\n}\n",
"id": "library.runtime-config",
"mode": "0640",
"path": "/var/lib/library/config/runtime.json",
"type": "file"
},
{
"id": "library.server",
"image": "registry.invalid:5000/library@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"type": "container",
"volumes": [
"/var/lib/library/config:/config",
"/home/media/series:/series",
"/storage/downloads:/downloads:ro"
]
}
]
+13 -1
View File
@@ -268,7 +268,14 @@ func build(ctx context.Context, run Runner, publish Publisher,
}
// What is judged only where a definition is written (novox/hq ADR 0326 §3, issue 574): a build is
// where a changed definition is first read, so it is refused here before anything is made of it.
if problems := catalogue.WrittenSettingsProblems(manifest); len(problems) > 0 {
// Judged with each template's text in its file, as written in the tree (novox/hq ADR 0349 §2): a template the
// module's directory lacks is refused here, before anything is built.
judged, err := catalogue.WithTemplatesFrom(manifest, within)
if err != nil {
say("manifest", "INVALID: %v", err)
return Result{}, err
}
if problems := catalogue.WrittenSettingsProblems(judged); len(problems) > 0 {
err := fmt.Errorf("%s", strings.Join(problems, "; "))
say("manifest", "INVALID: %v", err)
return Result{}, err
@@ -410,6 +417,11 @@ func build(ctx context.Context, run Runner, publish Publisher,
}
}
// Each template's text, read back from the archive the build made or took (novox/hq ADR 0349 §2).
if built, err = withTemplateTexts(manifest, within, built); err != nil {
say("templates", "FAILED: %v", err)
return Result{}, err
}
resolved, err := manifest.Resolve(built)
if err != nil {
say("resolve", "FAILED: %v", err)
+14 -2
View File
@@ -73,8 +73,14 @@ const reusedModule = `{"module":"meshboard","version":"1",
// aModuleAt is a repository holding the module under mod/, whose directory git names as tree, on its trunk.
func aModuleAt(t *testing.T, tree string) (*recorded, Runner, string) {
t.Helper()
r := &recorded{tree: tree, contents: map[string]string{"mod/" + ManifestName: reusedModule,
"mod/Dockerfile": "FROM scratch", "mod/files/theme.conf": "dark"}}
return aModuleWith(t, tree, map[string]string{"mod/" + ManifestName: reusedModule,
"mod/Dockerfile": "FROM scratch", "mod/files/theme.conf": "dark"})
}
// aModuleWith is aModuleAt with the repository's files given.
func aModuleWith(t *testing.T, tree string, contents map[string]string) (*recorded, Runner, string) {
t.Helper()
r := &recorded{tree: tree, contents: contents}
run := func(ctx context.Context, dir, name string, args ...string) (string, error) {
if name == "git" && len(args) > 0 && args[0] == "symbolic-ref" {
r.ran = append(r.ran, name+" "+strings.Join(args, " "))
@@ -100,6 +106,12 @@ var aPull = PullRef{Owner: "novox", Repo: "meshboard", Number: 7}
func aCheckBuilt(t *testing.T, scratch *aScratch, tree string) CheckBuilt {
t.Helper()
r, run, workspace := aModuleAt(t, tree)
return aCheckBuiltBy(t, scratch, r, run, workspace)
}
// aCheckBuiltBy is aCheckBuilt of the repository given.
func aCheckBuiltBy(t *testing.T, scratch *aScratch, r *recorded, run Runner, workspace string) CheckBuilt {
t.Helper()
spec := CheckSpec{ID: "build-1", Owner: aPull.Owner, Repo: aPull.Repo, Number: aPull.Number,
Builds: []CheckBuildSpec{{Module: "meshboard", Repository: "https://forge.invalid/meshboard.git", Path: "mod",
Ref: "main"}}}
+94
View File
@@ -0,0 +1,94 @@
package builder
import (
"archive/tar"
"bytes"
"compress/gzip"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"path/filepath"
"github.com/novox/mesh-controller/internal/catalogue"
)
// withTemplateTexts is what was built with each template's text, for a module whose files name templates
// (novox/hq ADR 0349 §2): its templates archive gains the text of every file in it, read back from the packed
// bytes, by its path inside the archive. Resolve reads them into the files' content.
//
// **Read from the archive, proven by its digest.** The module's files/ is packed again, as the build packs it,
// and the bytes are read only when they hash to the digest the build made or took: a fresh build's own archive,
// or a check's build a walk copied by digest without building (novox/hq ADR 0350 §2). Packing is deterministic,
// so one source packs to one digest; a tree that packs to another is not the archive, and its text is refused.
func withTemplateTexts(manifest catalogue.Manifest, within string, built []catalogue.Built) ([]catalogue.Built, error) {
if !manifest.Templated() {
return built, nil
}
name := catalogue.TemplatesArtifact(manifest)
at := -1
for i, b := range built {
if b.Name == name {
at = i
}
}
if name == "" || at < 0 {
return nil, fmt.Errorf("%s names templates and built no archive of %s/ to read them from",
manifest.Module, catalogue.TemplatesDir)
}
body, err := pack(filepath.Join(within, catalogue.TemplatesDir))
if err != nil {
return nil, fmt.Errorf("%s: packing %s/ to read its templates back failed: %w", manifest.Module,
catalogue.TemplatesDir, err)
}
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
if built[at].Digest != digest {
return nil, fmt.Errorf("%s: %s/ packs to the digest %s, and the archive %q built or taken is %s: the "+
"templates are read only from the archive the mesh records", manifest.Module, catalogue.TemplatesDir,
digest, name, orNone(built[at].Digest))
}
texts, err := unpackTexts(body)
if err != nil {
return nil, fmt.Errorf("%s: reading the archive %q back failed: %w", manifest.Module, name, err)
}
out := append([]catalogue.Built(nil), built...)
out[at].Texts = texts
return out, nil
}
// unpackTexts is every file of a packed archive (pack), by its path inside it.
func unpackTexts(body []byte) (map[string]string, error) {
zipped, err := gzip.NewReader(bytes.NewReader(body))
if err != nil {
return nil, err
}
defer zipped.Close()
reader := tar.NewReader(zipped)
texts := map[string]string{}
for {
header, err := reader.Next()
if errors.Is(err, io.EOF) {
return texts, nil
}
if err != nil {
return nil, err
}
if header.Typeflag != tar.TypeReg {
continue
}
text, err := io.ReadAll(reader)
if err != nil {
return nil, err
}
texts[header.Name] = string(text)
}
}
func orNone(digest string) string {
if digest == "" {
return "no digest"
}
return digest
}
+136
View File
@@ -0,0 +1,136 @@
package builder
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A module's files kept as template files in its repository (novox/hq ADR 0349 §2, design 57 package T1, issue
// 654): the build packs files/ into the module's templates archive and reads each template's text back from the
// packed bytes into the file's content, so the manifest the mesh holds is the one it would hold for the text inline.
const templatedModule = `{"module":"meshboard","version":"1",
"build":{"artifacts":[
{"name":"server","kind":"image","from":"Dockerfile"},
{"name":"files","kind":"archive","from":"files"}]},
"resources":[
{"id":"svc","type":"container","name":"meshboard","artifact":"server"},
{"id":"conf","type":"file","path":"/etc/meshboard/meshboard.conf","template":"files/meshboard.conf"},
{"id":"deeper","type":"file","path":"/etc/meshboard/deeper.conf","template":"files/a/deeper.conf"}]}`
var templatedFiles = map[string]string{
"Dockerfile": "FROM scratch", "files/meshboard.conf": "listen = 8080\n\"quoted\" é\n",
"files/a/deeper.conf": "deeper\n", "files/unused.conf": "packed, named by nothing",
}
func contentOf(t *testing.T, m catalogue.Manifest, id string) string {
t.Helper()
for _, r := range m.Resources {
if r["id"] == id {
if _, has := r[catalogue.TemplateField]; has {
t.Fatalf("%s still names its template in the built manifest: %v", id, r)
}
text, ok := r["content"].(string)
if !ok {
t.Fatalf("%s carries no content: %v", id, r)
}
return text
}
}
t.Fatalf("no resource %s in %v", id, m.Resources)
return ""
}
func TestABuildReadsEachTemplateBackFromItsArchive(t *testing.T) {
r, workspace := aRepository(t, templatedModule, templatedFiles)
got, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if text := contentOf(t, got.Manifest, "conf"); text != templatedFiles["files/meshboard.conf"] {
t.Fatalf("the template's text is %q, not the file's", text)
}
if text := contentOf(t, got.Manifest, "deeper"); text != templatedFiles["files/a/deeper.conf"] {
t.Fatalf("the template's text is %q, not the file's", text)
}
if r.archives["meshboard/files"] == "" {
t.Fatalf("the templates archive was not published: %v", r.archives)
}
if problem := catalogue.HeldTemplateProblem(got.Manifest); problem != "" {
t.Fatal(problem)
}
}
func TestABuildWhoseTemplateIsMissingFailsNamingIt(t *testing.T) {
files := map[string]string{}
for k, v := range templatedFiles {
files[k] = v
}
delete(files, "files/a/deeper.conf")
r, workspace := aRepository(t, templatedModule, files)
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/meshboard.git", "", "", workspace, nil,
Npmrc{}, GitCredential{}, nil)
if err == nil || !strings.Contains(err.Error(), "files/a/deeper.conf") {
t.Fatalf("a build whose template is missing did not fail naming it: %v", err)
}
}
// A walk that takes its check's build copies the archive by digest and builds nothing: the templates are read back
// from the tree packed again, which is the archive only when it packs to the digest taken.
func TestAWalkTakingAChecksBuildReadsTheTemplatesBackToo(t *testing.T) {
withVersion(t, "builder-1")
contents := map[string]string{"mod/" + ManifestName: templatedModule}
for k, v := range templatedFiles {
contents["mod/"+k] = v
}
scratch := newScratch()
r, run, workspace := aModuleWith(t, "tree-one", contents)
checked := aCheckBuiltBy(t, scratch, r, run, workspace)
r, run, workspace = aModuleWith(t, "tree-one", contents)
got, err := BuildReusing(t.Context(), run, r, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: named(checked)},
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if got.Reused == nil || len(r.archives) > 0 {
t.Fatalf("the walk built again: reused %+v, why %q, published %v", got.Reused, got.Why, r.archives)
}
if text := contentOf(t, got.Manifest, "conf"); text != templatedFiles["files/meshboard.conf"] {
t.Fatalf("the template's text is %q, not the file's", text)
}
}
// The tree packed again must be the archive taken; text read from anything else is not the archive's.
func TestTemplateTextsAreRefusedWhenTheTreeDoesNotPackToTheDigestTaken(t *testing.T) {
m, err := catalogue.ParseManifest([]byte(templatedModule))
if err != nil {
t.Fatal(err)
}
within := writeTree(t, t.TempDir(), templatedFiles)
built := []catalogue.Built{{Name: "files", Kind: catalogue.ArtifactArchive, Digest: "sha256:" + strings.Repeat("0", 64)}}
if _, err := withTemplateTexts(m, within, built); err == nil || !strings.Contains(err.Error(), "digest") {
t.Fatalf("text was read from a tree that does not pack to the archive taken: %v", err)
}
}
// writeTree writes files under dir and answers dir.
func writeTree(t *testing.T, dir string, files map[string]string) string {
t.Helper()
for name, body := range files {
full := filepath.Join(dir, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
return dir
}
+24
View File
@@ -35,6 +35,10 @@ type Built struct {
// Launchers are, for a bundle in an interpreted language, the executable the build wrote beside
// each entrypoint, by entrypoint (novox/hq ADR 0193): what the node's runtime starts to serve it.
Launchers map[string]string
// Texts are, for the archive a module packs its templates into (TemplatesArtifact), each file's text as the
// build read it back from the packed archive, by its path inside it (novox/hq ADR 0349 §2). Never carried on
// the wire or kept: Resolve reads them into the files' content, which is what the mesh holds.
Texts map[string]string `json:"-"`
}
// Resolve fills a manifest's resources in from what was built.
@@ -71,6 +75,26 @@ func (m Manifest) Resolve(built []Built) (Manifest, error) {
m.Module, strings.Join(missing, " and "), oneOrOther(len(missing)))
}
// **Each template's text, as the build read it back from its templates archive** (novox/hq ADR 0349 §2): into
// the file's content, so what the mesh holds and composes is what the same text inline would be.
if m.Templated() {
name := TemplatesArtifact(m)
texts := by[name].Texts
if name == "" || texts == nil {
return Manifest{}, fmt.Errorf("%s names templates, and the build carried no text read back from the "+
"archive it packs %s/ into (%q): the builder that built it does not read templates",
m.Module, TemplatesDir, name)
}
filled, err := fillTemplates(m.Module, m.Resources, func(template string) (string, bool) {
t, ok := texts[strings.TrimPrefix(template, TemplatesDir+"/")]
return t, ok
}, fmt.Sprintf("the archive %q", name))
if err != nil {
return Manifest{}, err
}
m.Resources = filled
}
out := m
out.Build = nil
out.Resources = nil
+12 -3
View File
@@ -51,6 +51,15 @@ func stepCA() Manifest {
}
}
// stepCAWithItsRoot is the authority whose root the operator supplies: the served key asks for it as
// `${setting:root}` (ADR 0155), the only way a setting reaches what a provider serves since a layer
// stopped replacing a served key (novox/hq issue 696; before it, the empty `root` was replaced whole).
func stepCAWithItsRoot() Manifest {
m := stepCA()
m.Serves["acme-ca"]["root"] = "${setting:root}"
return m
}
// routeProxy issues from that authority, so it must be told the root to verify it with.
func routeProxy() Manifest {
return Manifest{
@@ -78,7 +87,7 @@ func routeProxy() Manifest {
// The consequence is silent and total: route-proxy wrote an empty CA bundle, fell back to the
// system trust store, could not verify the internal authority, and no certificate was ever issued.
func TestACoLocatedProvidersServedValuesReachItsConsumer(t *testing.T) {
r, err := Resolve(shelf(stepCA(), routeProxy()),
r, err := Resolve(shelf(stepCAWithItsRoot(), routeProxy()),
[]string{"step-ca", "route-proxy"}, reachable(), World{})
if err != nil {
t.Fatal(err)
@@ -106,7 +115,7 @@ func TestACoLocatedProvidersServedValuesReachItsConsumer(t *testing.T) {
// And the binding file says the same thing, for a consumer that reads the binding rather than a
// substituted placeholder. The two are one fact and must not be able to disagree.
func TestACoLocatedConsumersBindingCarriesTheSettledValues(t *testing.T) {
r, err := Resolve(shelf(stepCA(), routeProxy()),
r, err := Resolve(shelf(stepCAWithItsRoot(), routeProxy()),
[]string{"step-ca", "route-proxy"}, reachable(), World{})
if err != nil {
t.Fatal(err)
@@ -152,7 +161,7 @@ func TestACoLocatedProviderIsSettledWhicheverWayItWasPulledIn(t *testing.T) {
{"step-ca", "route-proxy"},
{"route-proxy", "step-ca"},
} {
r, err := Resolve(shelf(stepCA(), routeProxy()), assigned, reachable(), World{})
r, err := Resolve(shelf(stepCAWithItsRoot(), routeProxy()), assigned, reachable(), World{})
if err != nil {
t.Fatal(err)
}
+469
View File
@@ -0,0 +1,469 @@
package catalogue
import (
"encoding/json"
"fmt"
"regexp"
"slices"
"strings"
)
// A module declares its config (novox/hq ADR 0348 §1, §4, §5, §6, §8, §11; design 57 package K1, issue 660).
//
// **One concept, two spellings for a while.** "Config" replaces "settings": the manifest says `config` for
// `settings`, `takes-config` for `takes-settings` and `${config:<key>}` for `${setting:<key>}`. The old names stay
// accepted as aliases until every module has moved (ADR 0348 §1), and a module uses one form or the other, never
// both. The config form is read into the very fields the settings form fills — Settings, a mergeable file's
// takes-settings and `${setting:<key>}` — so composition has one path and `${config:<key>}` fills exactly as
// `${setting:<key>}` does: a module rewritten from settings to config composes byte-identically. It is written
// back in the form it was read in, so what the mesh holds is what its author wrote.
//
// **A moved module is held to the whole declaration.** Under `config` every value has a type (number, string or
// boolean; an object or a list only when trusted), a description including its unit, its validation (enum,
// minimum and maximum, pattern), whether it is trusted, optionally the predefined key it falls back to, and either
// a default or `required`; it lands in a file or is read at runtime (`runtime: true`); every mergeable file names
// the keys it takes; a `reload-on` or `restart-on` names a file the module declares. A module not yet moved keeps
// today's form and today's rules (SettingProblems, TakenSettingsProblems) until the catalogue's list of such
// modules is empty (ADR 0348 §11).
//
// **A required value nothing sets** is the operator's own: it composes as an operator's setting nobody gave does
// today, refused by name at composition (UnsetSettingError). Who reports it once mesh-config answers values is the
// provisioner's (ADR 0349), not this file's.
// ConfigField is the manifest key a module declares its config values under.
const ConfigField = "config"
// TakesConfigField is the key a mergeable file names the config values it takes by.
const TakesConfigField = "takes-config"
// SettingsField is the old name of ConfigField, read as its alias.
const SettingsField = "settings"
// configRef is how a file asks for a config value: ${config:<key>}, the same keys as settingRef.
var configRef = regexp.MustCompile(`\$\{config:([a-z0-9][a-z0-9_.-]*)\}`)
// takesConfigKey and takesSettingsKey are the two spellings as a key of a JSON object. A string value is never
// followed by a colon, and a quote inside a string is escaped, so these match only keys.
var (
takesConfigKey = regexp.MustCompile(`"` + TakesConfigField + `"(\s*):`)
takesSettingsKey = regexp.MustCompile(`"` + TakesSettingsField + `"(\s*):`)
)
// The types a config value may have (ADR 0348 §4).
const (
ConfigNumber = "number"
ConfigString = "string"
ConfigBoolean = "boolean"
ConfigObject = "object"
ConfigList = "list"
)
// accountFacts are what the mesh places for a module's account, never config values (ADR 0348 §11).
var accountFacts = map[string]bool{"puid": true, "pgid": true}
// PredefinedKey is one key of the catalogue's list of predefined keys, as far as a module's value falling back
// to it is judged here: its type, and whether it is trusted (ADR 0348 §7, §8).
type PredefinedKey struct {
Type string
Trusted bool
}
// predefinedKeys is the list of predefined keys, nil until it is present. The list (`predefined.json` at the
// catalogue's root) arrives with design 57's package K2 (issue 661); until then a value naming one is refused
// saying the list is not yet present.
var predefinedKeys map[string]PredefinedKey
// configForms is which of the two forms a manifest was written in, by the words of each it uses.
type configForms struct {
config, settings []string
}
// DeclaresConfig is whether the manifest is written in the config form: `config`, `takes-config` or
// `${config:<key>}` (ADR 0348 §1, §11).
func (m Manifest) DeclaresConfig() bool { return len(m.forms.config) > 0 }
// toSettingsForm is raw with the config form's placeholders and takes-config read as the settings form's, and
// which words of each form it used. `config` itself is a top-level key, renamed where the keys are read
// (UnmarshalJSON).
func toSettingsForm(raw []byte) ([]byte, configForms) {
var f configForms
if configRef.Match(raw) {
f.config = append(f.config, "${config:…}")
}
if takesConfigKey.Match(raw) {
f.config = append(f.config, TakesConfigField)
}
if settingRef.Match(raw) {
f.settings = append(f.settings, "${setting:…}")
}
if takesSettingsKey.Match(raw) {
f.settings = append(f.settings, TakesSettingsField)
}
if len(f.config) == 0 {
return raw, f
}
raw = configRef.ReplaceAllFunc(raw, func(ref []byte) []byte {
return []byte("${setting:" + string(configRef.FindSubmatch(ref)[1]) + "}")
})
raw = takesConfigKey.ReplaceAllFunc(raw, func(key []byte) []byte {
return []byte(`"` + TakesSettingsField + `"` + string(takesConfigKey.FindSubmatch(key)[1]) + ":")
})
return raw, f
}
// toConfigForm is a manifest written in the settings form's words, as JSON, in the config form's: what
// toSettingsForm and UnmarshalJSON read, written back.
func toConfigForm(raw []byte) ([]byte, error) {
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return nil, err
}
if settings, has := keys[SettingsField]; has {
keys[ConfigField] = settings
delete(keys, SettingsField)
var err error
if raw, err = json.Marshal(keys); err != nil {
return nil, err
}
}
raw = settingRef.ReplaceAllFunc(raw, func(ref []byte) []byte {
return []byte("${config:" + string(settingRef.FindSubmatch(ref)[1]) + "}")
})
raw = takesSettingsKey.ReplaceAllFunc(raw, func(key []byte) []byte {
return []byte(`"` + TakesConfigField + `"` + string(takesSettingsKey.FindSubmatch(key)[1]) + ":")
})
return raw, nil
}
// inConfigWords is a refusal said of a module in the config form in its words: `${config:<key>}` and
// takes-config, where the rules shared with the settings form say `${setting:<key>}` and takes-settings.
func inConfigWords(problem string) string {
problem = settingRef.ReplaceAllStringFunc(problem, func(ref string) string {
return "${config:" + settingRef.FindStringSubmatch(ref)[1] + "}"
})
return strings.ReplaceAll(problem, TakesSettingsField, TakesConfigField)
}
// MixedFormProblems is a manifest using both forms, said with the words of each (ADR 0348 §1).
func MixedFormProblems(m Manifest) []string {
if len(m.forms.config) == 0 || len(m.forms.settings) == 0 {
return nil
}
return []string{fmt.Sprintf("%s mixes the config form (%s) with the settings form (%s): a module uses one form "+
"or the other, never both; write %s, %s and ${config:<key>} throughout (novox/hq ADR 0348 §1)",
m.Module, strings.Join(m.forms.config, ", "), strings.Join(m.forms.settings, ", "),
ConfigField, TakesConfigField)}
}
// configDeclaration is the config form's declaration as written: nothing of the settings form's, and nothing
// unsaid.
type configDeclaration struct {
Type string `json:"type,omitempty"`
Description string `json:"description,omitempty"`
Enum []any `json:"enum,omitempty"`
Minimum *float64 `json:"minimum,omitempty"`
Maximum *float64 `json:"maximum,omitempty"`
Pattern string `json:"pattern,omitempty"`
Trusted bool `json:"trusted,omitempty"`
Predefined string `json:"predefined,omitempty"`
Default any `json:"default,omitempty"`
Required bool `json:"required,omitempty"`
Runtime bool `json:"runtime,omitempty"`
}
// MarshalJSON writes a settings declaration exactly as before, and a config declaration without the settings
// form's kind and why, and without a default it does not have.
func (d SettingDeclaration) MarshalJSON() ([]byte, error) {
type plain SettingDeclaration
if d.Kind != "" || d.Why != "" {
return json.Marshal(plain(d))
}
return json.Marshal(configDeclaration{Type: d.Type, Description: d.Description, Enum: d.Enum,
Minimum: d.Minimum, Maximum: d.Maximum, Pattern: d.Pattern, Trusted: d.Trusted, Predefined: d.Predefined,
Default: d.Default, Required: d.Required, Runtime: d.Runtime})
}
// configShaped is whether a declaration says anything of the config form's.
func configShaped(d SettingDeclaration) bool {
return d.Type != "" || d.Description != "" || len(d.Enum) > 0 || d.Minimum != nil || d.Maximum != nil ||
d.Pattern != "" || d.Trusted || d.Predefined != "" || d.Required || d.Runtime
}
// TypedSettingProblems is a declaration under `settings` that says what only `config` says: a value with a type,
// a description and its validation is declared in the config form (ADR 0348 §1, §4).
func TypedSettingProblems(m Manifest) []string {
var problems []string
for _, key := range sortedSettingKeys(m.Settings) {
if configShaped(m.Settings[key]) {
problems = append(problems, fmt.Sprintf("%s's setting %q says its type, description or validation "+
"under %s: a value declared with those is declared under config, with ${config:<key>} and "+
"takes-config, never under %s (novox/hq ADR 0348 §1, §4)", m.Module, key, SettingsField, SettingsField))
}
}
return problems
}
// ConfigProblems is every way a module in the config form declares its config wrongly, each said by name with
// what to do (ADR 0348 How it is checked, design 57 package K1).
func ConfigProblems(m Manifest) []string {
if !m.DeclaresConfig() {
return nil
}
var problems []string
placed := configPlaced(m)
for _, key := range sortedSettingKeys(m.Settings) {
d := m.Settings[key]
say := func(format string, args ...any) {
problems = append(problems, fmt.Sprintf("%s's config value %q: ", m.Module, key)+fmt.Sprintf(format, args...))
}
if !settingRef.MatchString("${setting:" + key + "}") {
say("not a usable key: lower-case letters, digits, dots, dashes and underscores")
continue
}
if meshWords[key] || accountFacts[key] {
say("%q is placement, which the mesh decides, never a config value (novox/hq ADR 0348 §11)", key)
continue
}
if d.Kind != "" || d.Why != "" {
say("kind and why are the settings form's; under config a value says its type and its description " +
"(novox/hq ADR 0348 §1, §4)")
}
switch d.Type {
case ConfigNumber, ConfigString, ConfigBoolean:
case ConfigObject, ConfigList:
if !d.Trusted {
say("is %s and not trusted: an object or a list decides more than one value can say, so it is "+
"trusted, \"trusted\": true, or split into numbers, strings and booleans (novox/hq ADR 0348 §4)", d.Type)
}
case "":
say("no type: say \"type\": number, string or boolean, or object or list when trusted (novox/hq ADR 0348 §4)")
default:
say("type %q is none of number, string, boolean, object and list (novox/hq ADR 0348 §4)", d.Type)
}
if strings.TrimSpace(d.Description) == "" {
say("no description: say in a sentence what the value is, including its unit (novox/hq ADR 0348 §4)")
}
switch {
case d.Required && d.Default != nil:
say("both a default and required: a required value is the operator's own and has no default; give " +
"one or the other (novox/hq ADR 0348 §4)")
case !d.Required && d.Default == nil:
say("neither a default nor required: give a default, or say \"required\": true for a value that is " +
"the operator's own (novox/hq ADR 0348 §4)")
}
for _, wrong := range validationProblems(d) {
say("%s", wrong)
}
if word := operatorsWord(key); word != "" && d.Default != nil {
say("a key naming %q is the operator's value: declare it \"required\": true, with no default "+
"(novox/hq ADR 0348 §4, ADR 0112)", word)
}
if d.Predefined != "" {
p, listed := predefinedKeys[d.Predefined]
switch {
case predefinedKeys == nil:
say("falls back to the predefined key %q, and the list of predefined keys (the catalogue's "+
"predefined.json, design 57 package K2) is not yet present: no value can fall back to one "+
"until it is; leave \"predefined\" out for now (novox/hq ADR 0348 §7)", d.Predefined)
case !listed:
say("falls back to %q, which is not on the list of predefined keys: name a key on it, or leave "+
"\"predefined\" out (novox/hq ADR 0348 §7)", d.Predefined)
default:
if p.Type != d.Type {
say("falls back to the predefined key %q, which is a %s, and this value is a %s: a value falls "+
"back only to a key of its own type (novox/hq ADR 0348 §4)", d.Predefined, p.Type, d.Type)
}
if p.Trusted && !d.Trusted {
say("falls back to the trusted predefined key %q and is not trusted itself: a value that falls "+
"back to a trusted key is trusted, \"trusted\": true (novox/hq ADR 0348 §8)", d.Predefined)
}
}
}
// A template's text is not here yet. Resolve fills it from the build, WithTemplatesFrom from the tree; neither
// judges again itself, but WithTemplatesFrom parses its result again, and a resolved build is parsed again
// where it is registered, and this is judged then.
if !placed[key] && !d.Runtime && !m.Templated() {
say("placed nowhere: no file says ${config:%s} or takes it under %s, and it is not marked "+
"\"runtime\": true for a value the module's code reads itself (novox/hq ADR 0348 §5)", key, TakesConfigField)
}
}
used := settingKeysUsedBy(m)
for _, key := range sortedKeys(used) {
if _, declared := m.Settings[key]; !declared {
problems = append(problems, fmt.Sprintf("%s says ${config:%s} and declares no such value under %s: "+
"declare %q with its type, description and a default or required (novox/hq ADR 0348 §4, §5)",
m.Module, key, ConfigField, key))
}
}
problems = append(problems, configTakesProblems(m)...)
problems = append(problems, reflectsProblems(m)...)
return problems
}
// configPlaced is every value a file, a unit, a directory's owner, a contribution or a served fact asks for, and
// every value a mergeable file takes.
func configPlaced(m Manifest) map[string]bool {
placed := settingKeysUsedBy(m)
for _, r := range m.Resources {
for _, k := range stringsOf(r[TakesSettingsField]) {
placed[k] = true
}
}
return placed
}
// validationProblems is what is wrong with a declaration's validation, and its default against it.
func validationProblems(d SettingDeclaration) []string {
var problems []string
isNumber := d.Type == ConfigNumber
if len(d.Enum) > 0 && d.Type != ConfigNumber && d.Type != ConfigString {
problems = append(problems, fmt.Sprintf("an enum is for a number or a string, and this is a %s", d.Type))
}
if (d.Minimum != nil || d.Maximum != nil) && !isNumber {
problems = append(problems, fmt.Sprintf("a minimum or maximum is for a number, and this is a %s", d.Type))
}
if d.Minimum != nil && d.Maximum != nil && *d.Minimum > *d.Maximum {
problems = append(problems, fmt.Sprintf("its minimum %v is above its maximum %v", *d.Minimum, *d.Maximum))
}
var pattern *regexp.Regexp
if d.Pattern != "" {
if d.Type != ConfigString {
problems = append(problems, fmt.Sprintf("a pattern is for a string, and this is a %s", d.Type))
}
var err error
if pattern, err = regexp.Compile(`^(?:` + d.Pattern + `)$`); err != nil {
problems = append(problems, fmt.Sprintf("its pattern %q is not a regular expression: %v", d.Pattern, err))
}
}
for _, v := range d.Enum {
if !ofType(v, d.Type) {
problems = append(problems, fmt.Sprintf("its enum holds %v, which is not a %s", v, d.Type))
}
}
if d.Default == nil {
return problems
}
if !ofType(d.Default, d.Type) {
if d.Type != "" {
problems = append(problems, fmt.Sprintf("its default %v is not a %s", d.Default, d.Type))
}
return problems
}
if len(d.Enum) > 0 && !slices.ContainsFunc(d.Enum, func(v any) bool { return v == d.Default }) {
problems = append(problems, fmt.Sprintf("its default %v is not one of %v", d.Default, d.Enum))
}
if n, ok := d.Default.(float64); ok {
if d.Minimum != nil && n < *d.Minimum {
problems = append(problems, fmt.Sprintf("its default %v is below its minimum %v", n, *d.Minimum))
}
if d.Maximum != nil && n > *d.Maximum {
problems = append(problems, fmt.Sprintf("its default %v is above its maximum %v", n, *d.Maximum))
}
}
if s, ok := d.Default.(string); ok && pattern != nil && !pattern.MatchString(s) {
problems = append(problems, fmt.Sprintf("its default %q does not match its pattern %q", s, d.Pattern))
}
return problems
}
// ofType is whether a value read from JSON is of a config type.
func ofType(v any, kind string) bool {
switch v.(type) {
case float64:
return kind == ConfigNumber
case string:
return kind == ConfigString
case bool:
return kind == ConfigBoolean
case map[string]any:
return kind == ConfigObject
case []any:
return kind == ConfigList
}
return false
}
// configTakesProblems is what ADR 0348 §11 adds for a module in the config form: an account fact in
// takes-config (the mesh's own words are refused for both forms by TakenSettingsProblems), and a mergeable file
// naming no keys where none of the module's does (where one does, TakenSettingsProblems names every one that
// does not).
func configTakesProblems(m Manifest) []string {
var problems, silent []string
naming := false
for _, r := range m.Resources {
if how, _ := r["merge"].(string); how == "" {
continue
}
if _, said := r[TakesSettingsField]; !said {
silent = append(silent, fmt.Sprint(r["id"]))
continue
}
naming = true
for _, k := range stringsOf(r[TakesSettingsField]) {
_, declared := m.Settings[k]
switch {
case accountFacts[k]:
problems = append(problems, fmt.Sprintf("%s's %v names %q in %s, and %q is an account fact the "+
"mesh places, never a config value (novox/hq ADR 0348 §11)", m.Module, r["id"], k,
TakesConfigField, k))
case meshWords[k]:
// Refused for both forms by TakenSettingsProblems, in its words.
case !declared:
problems = append(problems, fmt.Sprintf("%s's %v names %q in %s and declares no such value under "+
"%s: a value a file takes is declared like every other, with its type, description and a "+
"default or required, or the config verb could never set it (novox/hq ADR 0348 §4, §11)",
m.Module, r["id"], k, TakesConfigField, ConfigField))
}
}
}
if !naming {
for _, id := range silent {
problems = append(problems, fmt.Sprintf("%s's mergeable %s names no %s: a module that declares config "+
"names the keys each mergeable file takes, an empty list for none (novox/hq ADR 0348 §11)",
m.Module, id, TakesConfigField))
}
}
return problems
}
// meshMadeIDs are the resources the mesh composes into a module's own set, which its `restart-on` may name
// without declaring them; meshMadePrefixes are those named after what they are for.
var (
meshMadeIDs = []string{ComposedJailsID(), FilteringID(), CertificateID(), AuthorityID(), BusUsersID()}
meshMadePrefixes = []string{NeedID(""), SecretID(""), strings.TrimSuffix(GrantID("", ""), "-"), BoundID(""), ReceivedID(""), AccessID("/")}
)
// reflectsProblems is a `reload-on` or `restart-on` naming a file the module does not declare (ADR 0348 §6): not
// one of its own resources, not one the mesh composes for it, and not another module's, which is named
// `<module>.<id>`.
func reflectsProblems(m Manifest) []string {
own := map[string]bool{}
for _, r := range m.Resources {
own[fmt.Sprint(r["id"])] = true
}
meshMade := func(id string) bool {
if slices.Contains(meshMadeIDs, id) {
return true
}
for _, p := range meshMadePrefixes {
if strings.HasPrefix(id, p) && len(id) > len(p) {
return true
}
}
return false
}
var problems []string
for _, r := range m.Resources {
for _, field := range []string{"reload-on", "restart-on"} {
for _, id := range stringsOf(r[field]) {
if strings.Contains(id, ".") || own[id] || meshMade(id) {
continue
}
problems = append(problems, fmt.Sprintf("%s's %v names %q in its %s, and %s declares no such "+
"file: name a file of its own, or another module's as <module>.<id> (novox/hq ADR 0348 §6)",
m.Module, r["id"], id, field, m.Module))
}
}
}
return problems
}
+453
View File
@@ -0,0 +1,453 @@
package catalogue
import (
"bytes"
"encoding/json"
"errors"
"strings"
"testing"
)
// A module declares its config (novox/hq ADR 0348 §1, §4, §5, §6, §8, §11; design 57 package K1, issue 660):
// one registration refusal per rule of design 57's K1, the old names read as aliases, and a module rewritten
// from settings to config composing byte-identically.
// inConfigForm is a module in the config form that registers: every value typed, described, defaulted or
// required, placed in a file or read at runtime, and its mergeable file naming its keys.
func inConfigForm() map[string]any {
return map[string]any{
"module": "notifier",
"version": "1",
"config": map[string]any{
"font-size": map[string]any{"type": "number", "description": "the notification's font size, in points",
"minimum": 6, "maximum": 40, "default": 10},
"width": map[string]any{"type": "number", "description": "the notification's width, in pixels",
"default": 250},
"position": map[string]any{"type": "string", "description": "the corner it shows in, a word",
"enum": []any{"top-right", "top-left"}, "default": "top-right"},
"theme": map[string]any{"type": "string", "description": "the colour theme, a lower-case name",
"pattern": "[a-z]+", "default": "dark"},
"history": map[string]any{"type": "number", "description": "how many notifications it keeps, a count",
"default": 20, "runtime": true},
"sender": map[string]any{"type": "string", "description": "the address it mails a digest from",
"required": true},
},
"resources": []any{
map[string]any{"id": "configuration", "type": "file", "path": "/x/notifierrc",
"content": "font = Inter ${config:font-size}\nwidth = ${config:width}\nfrom = ${config:sender}\n"},
map[string]any{"id": "rules", "type": "file", "path": "/x/rules.json", "merge": "json",
"takes-config": []any{"position", "theme"}, "content": "{\"position\": \"top-right\"}\n"},
map[string]any{"id": "notifier", "type": "service", "unit": "notifier.service",
"reload-on": []any{"configuration"}, "restart-on": []any{"rules"}},
},
}
}
// declaration is one value of inConfigForm's config, to change in place.
func declaration(m map[string]any, key string) map[string]any {
return m["config"].(map[string]any)[key].(map[string]any)
}
func resourceOf(m map[string]any, id string) map[string]any {
for _, r := range m["resources"].([]any) {
if r.(map[string]any)["id"] == id {
return r.(map[string]any)
}
}
return nil
}
func parseRaw(t *testing.T, m map[string]any) (Manifest, error) {
t.Helper()
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
return ParseManifest(raw)
}
// refused is that the manifest is refused with every one of these words.
func refused(t *testing.T, m map[string]any, words ...string) {
t.Helper()
_, err := parseRaw(t, m)
if err == nil {
t.Fatalf("registered; wanted a refusal saying %q", words)
}
for _, w := range words {
if !strings.Contains(err.Error(), w) {
t.Fatalf("refused without %q:\n%v", w, err)
}
}
}
func TestAModuleInTheConfigFormRegisters(t *testing.T) {
m, err := parseRaw(t, inConfigForm())
if err != nil {
t.Fatal(err)
}
if !m.DeclaresConfig() {
t.Fatal("a module declaring config is not read as in the config form")
}
d := m.Settings["font-size"]
if d.Type != "number" || d.Description == "" || d.Minimum == nil || *d.Minimum != 6 || d.Default != float64(10) {
t.Fatalf("font-size read as %+v", d)
}
if s := m.Settings["sender"]; !s.Required || s.Default != nil {
t.Fatalf("sender read as %+v", s)
}
}
// Rule 1: a value without type or description.
func TestConfigRefusesAValueWithoutTypeOrDescription(t *testing.T) {
m := inConfigForm()
delete(declaration(m, "width"), "type")
refused(t, m, `notifier's config value "width"`, "no type")
m = inConfigForm()
declaration(m, "width")["type"] = "colour"
refused(t, m, `"width"`, `type "colour"`)
m = inConfigForm()
delete(declaration(m, "width"), "description")
refused(t, m, `"width"`, "no description")
}
// Rule 2: a value with neither a default nor required.
func TestConfigRefusesAValueWithNeitherDefaultNorRequired(t *testing.T) {
m := inConfigForm()
delete(declaration(m, "width"), "default")
refused(t, m, `"width"`, "neither a default nor required")
m = inConfigForm()
declaration(m, "sender")["default"] = "noreply@example.org"
refused(t, m, `"sender"`, "both a default and required")
}
// Rule 3: a default failing its validation.
func TestConfigRefusesADefaultFailingItsValidation(t *testing.T) {
for _, c := range []struct {
key string
value any
said string
}{
{"font-size", 4, "below its minimum"},
{"font-size", 41, "above its maximum"},
{"font-size", "10", "not a number"},
{"position", "bottom", "not one of"},
{"theme", "Dark", "does not match"},
} {
m := inConfigForm()
declaration(m, c.key)["default"] = c.value
refused(t, m, `"`+c.key+`"`, c.said)
}
}
// Rule 4: an object or a list not trusted.
func TestConfigRefusesAnObjectOrListNotTrusted(t *testing.T) {
for _, kind := range []string{"object", "list"} {
m := inConfigForm()
def := any(map[string]any{"a": 1})
if kind == "list" {
def = []any{"a"}
}
m["config"].(map[string]any)["rules"] = map[string]any{"type": kind, "description": "the rules, as " + kind,
"default": def, "runtime": true}
refused(t, m, `"rules"`, "is "+kind+" and not trusted")
m["config"].(map[string]any)["rules"].(map[string]any)["trusted"] = true
if _, err := parseRaw(t, m); err != nil {
t.Fatalf("a trusted %s is refused: %v", kind, err)
}
}
}
// withPredefined is the list of predefined keys as package K2 will give it, for the length of a test.
func withPredefined(t *testing.T, keys map[string]PredefinedKey) {
t.Helper()
was := predefinedKeys
predefinedKeys = keys
t.Cleanup(func() { predefinedKeys = was })
}
// Rule 5: a predefined key not on the list, or of another type. Until the list is present (package K2), a value
// naming one is refused saying so.
func TestConfigRefusesAPredefinedKeyNotOnTheListOrOfAnotherType(t *testing.T) {
m := inConfigForm()
declaration(m, "font-size")["predefined"] = "font-size"
refused(t, m, `"font-size"`, "the list of predefined keys", "not yet present")
withPredefined(t, map[string]PredefinedKey{"font-size": {Type: "number"}, "accent": {Type: "string"}})
if _, err := parseRaw(t, m); err != nil {
t.Fatalf("a value falling back to a listed key of its type is refused: %v", err)
}
m = inConfigForm()
declaration(m, "font-size")["predefined"] = "font-scale"
refused(t, m, `"font-size"`, `"font-scale"`, "not on the list")
m = inConfigForm()
declaration(m, "font-size")["predefined"] = "accent"
refused(t, m, `"font-size"`, `"accent"`, "is a string")
}
// Rule 6: an untrusted value falling back to a trusted predefined key.
func TestConfigRefusesAnUntrustedValueFallingBackToATrustedKey(t *testing.T) {
m := inConfigForm()
declaration(m, "theme")["predefined"] = "theme"
refused(t, m, `"theme"`, "not yet present")
withPredefined(t, map[string]PredefinedKey{"theme": {Type: "string", Trusted: true}})
refused(t, m, `"theme"`, "trusted", "is not")
declaration(m, "theme")["trusted"] = true
if _, err := parseRaw(t, m); err != nil {
t.Fatalf("a trusted value falling back to a trusted key is refused: %v", err)
}
}
// Rule 7: a value neither placed in a template nor read at runtime.
func TestConfigRefusesAValueNeitherPlacedNorReadAtRuntime(t *testing.T) {
m := inConfigForm()
delete(declaration(m, "history"), "runtime")
refused(t, m, `"history"`, "${config:history}", `"runtime": true`)
// Taken by a mergeable file is placed.
m = inConfigForm()
if _, err := parseRaw(t, m); err != nil {
t.Fatal(err)
}
}
// Rule 8: a placeholder naming an undeclared value.
func TestConfigRefusesAPlaceholderNamingAnUndeclaredValue(t *testing.T) {
m := inConfigForm()
resourceOf(m, "configuration")["content"] = "font = ${config:font-size}\nwidth = ${config:width}\n" +
"from = ${config:sender}\ncolour = ${config:colour}\n"
refused(t, m, "${config:colour}", "declares no such value")
}
// Rule 9: a reload-on or restart-on naming a file the module does not declare.
func TestConfigRefusesAReloadOrRestartOnNamingAnUndeclaredFile(t *testing.T) {
m := inConfigForm()
resourceOf(m, "notifier")["reload-on"] = []any{"configuration", "dunstrc"}
refused(t, m, "reload-on", `"dunstrc"`, "declares no such")
m = inConfigForm()
resourceOf(m, "notifier")["restart-on"] = []any{"rule"}
refused(t, m, "restart-on", `"rule"`)
// Another module's file, and one the mesh makes for the module, are not the module's to declare.
m = inConfigForm()
resourceOf(m, "notifier")["restart-on"] = []any{"rules", "resolver.fact-zones", "secret-mail", "composed-jails"}
if _, err := parseRaw(t, m); err != nil {
t.Fatalf("refused: %v", err)
}
}
// Rule 10: a moved module's mergeable file not naming its keys under takes-config.
func TestConfigRefusesAMergeableFileNotNamingItsKeys(t *testing.T) {
m := inConfigForm()
delete(resourceOf(m, "rules"), "takes-config")
// position and theme are then placed nowhere; say they are read at runtime so only the file's rule speaks.
declaration(m, "position")["runtime"] = true
declaration(m, "theme")["runtime"] = true
refused(t, m, "rules", "names no takes-config")
}
// Rule 11: a mesh word, puid or pgid in takes-config.
func TestConfigRefusesAMeshWordOrAnAccountFactInTakesConfig(t *testing.T) {
for _, word := range []string{"ports", "expose", "reach", "endpoints", "places", "accesses", "networks", "puid", "pgid"} {
m := inConfigForm()
resourceOf(m, "rules")["takes-config"] = []any{"position", "theme", word}
refused(t, m, `"`+word+`"`, "takes-config")
}
}
// A module uses one form or the other, never both.
func TestAManifestMixingTheConfigAndSettingsFormsIsRefused(t *testing.T) {
m := inConfigForm()
resourceOf(m, "configuration")["content"] = "font = ${config:font-size}\nwidth = ${setting:width}\nfrom = ${config:sender}\n"
refused(t, m, "mixes", "${setting:", "config")
m = inConfigForm()
delete(resourceOf(m, "rules"), "takes-config")
resourceOf(m, "rules")["takes-settings"] = []any{"position", "theme"}
refused(t, m, "mixes", "takes-settings")
m = inConfigForm()
m["settings"] = map[string]any{"colour": map[string]any{"kind": "preference", "default": "red", "why": "red"}}
refused(t, m, "mixes", "settings")
}
// The settings form keeps its own declaration; the config form's has no kind or why.
func TestEachFormKeepsItsOwnDeclaration(t *testing.T) {
m := inConfigForm()
declaration(m, "width")["why"] = "fits a title"
refused(t, m, `"width"`, "why")
err := parsed(t, Manifest{Module: "notifier",
Settings: map[string]SettingDeclaration{"width": {Kind: KindPreference, Default: float64(250),
Why: "fits a title", Type: "number"}},
Resources: []map[string]any{{"id": "c", "type": "file", "path": "/x", "content": "${setting:width}"}}})
if err == nil || !strings.Contains(err.Error(), "under config") {
t.Fatalf("a typed value under settings: %v", err)
}
}
// What the mesh holds is what was written: a module in the config form is written back in it, and read again
// the same.
func TestAConfigManifestIsWrittenBackInItsOwnForm(t *testing.T) {
m, err := parseRaw(t, inConfigForm())
if err != nil {
t.Fatal(err)
}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
for _, said := range []string{`"config":`, `${config:font-size}`, `"takes-config":`, `"required":true`} {
if !strings.Contains(string(raw), said) {
t.Fatalf("written back without %s: %s", said, raw)
}
}
for _, unsaid := range []string{`"settings":`, `${setting:`, `"takes-settings"`, `"kind"`, `"why"`, `"default":null`} {
if strings.Contains(string(raw), unsaid) {
t.Fatalf("written back with %s: %s", unsaid, raw)
}
}
again, err := ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
twice, _ := json.Marshal(again)
if !bytes.Equal(raw, twice) {
t.Fatalf("read again it is written\n%s\nwhere it was\n%s", twice, raw)
}
}
// A module in the settings form is written back byte for byte as before.
func TestASettingsManifestIsWrittenBackAsBefore(t *testing.T) {
d := SettingDeclaration{Kind: KindPreference, Default: float64(10), Why: "readable"}
raw, err := json.Marshal(d)
if err != nil {
t.Fatal(err)
}
if string(raw) != `{"kind":"preference","default":10,"why":"readable"}` {
t.Fatalf("written as %s", raw)
}
}
// settingsDunst is the dunst module as it is written today; configDunst the same module rewritten from settings to
// config.
const settingsDunst = `{"module": "dunst", "version": "1",
"settings": {
"font-size": {"kind": "preference", "default": 10, "why": "10 points of Inter reads well on a screen of about 100 DPI"},
"width": {"kind": "preference", "default": 250, "why": "250 pixels fits a title of about forty characters at 10 points"}
},
"resources": [
{"id": "package", "type": "package", "package": "dunst"},
{"id": "configuration", "type": "file", "path": "/home/me/.config/dunst/dunstrc", "mode": "0644",
"content": "[global]\n font = Inter ${setting:font-size}\n width = ${setting:width}\n"},
{"id": "rules", "type": "file", "path": "/home/me/.config/dunst/rules.json", "merge": "json",
"takes-settings": [], "content": "{\"urgency\": \"low\"}\n"},
{"id": "notifier", "type": "service", "unit": "dunst.service", "reload-on": ["configuration"]}
]}`
const configDunst = `{"module": "dunst", "version": "1",
"config": {
"font-size": {"type": "number", "description": "the font size of a notification, in points", "minimum": 6,
"maximum": 40, "default": 10},
"width": {"type": "number", "description": "the width of a notification, in pixels", "minimum": 100,
"default": 250}
},
"resources": [
{"id": "package", "type": "package", "package": "dunst"},
{"id": "configuration", "type": "file", "path": "/home/me/.config/dunst/dunstrc", "mode": "0644",
"content": "[global]\n font = Inter ${config:font-size}\n width = ${config:width}\n"},
{"id": "rules", "type": "file", "path": "/home/me/.config/dunst/rules.json", "merge": "json",
"takes-config": [], "content": "{\"urgency\": \"low\"}\n"},
{"id": "notifier", "type": "service", "unit": "dunst.service", "reload-on": ["configuration"]}
]}`
// ${config:<key>} fills exactly as ${setting:<key>} does: the module rewritten from settings to config composes
// to the same bytes, with no layer, with the node's, and with the mesh's under the node's.
func TestAModuleRewrittenFromSettingsToConfigComposesByteIdentically(t *testing.T) {
composed := func(manifest string, layers []Layer) []byte {
t.Helper()
m, err := ParseManifest([]byte(manifest))
if err != nil {
t.Fatal(err)
}
r := anAdoptedAnchor()
r.Modules = append(r.Modules, m)
with := anchorRendering(false)
if layers != nil {
with.Settings["dunst"] = layers
}
c, err := r.Compose(with)
if err != nil {
t.Fatal(err)
}
if why, left := c.LeftOut["dunst"]; left {
t.Fatalf("dunst was left out: %s", why)
}
raw, err := json.Marshal(c)
if err != nil {
t.Fatal(err)
}
return raw
}
for _, layers := range [][]Layer{
nil,
{{From: "anchor", Values: map[string]any{"font-size": float64(13)}}},
{{From: MeshWideLayer, Values: map[string]any{"width": float64(300)}},
{From: "anchor", Values: map[string]any{"font-size": float64(12), "width": float64(320)}}},
} {
was, is := composed(settingsDunst, layers), composed(configDunst, layers)
if !bytes.Equal(was, is) {
t.Fatalf("composed differently with %v:\nsettings: %s\nconfig: %s", layers, was, is)
}
if !strings.Contains(string(is), "font = Inter 1") || strings.Contains(string(is), "${config:") {
t.Fatalf("not filled: %s", is)
}
}
}
// A key a mergeable file takes is a config value like every other: one not declared under config is refused by
// name (ADR 0348 §4, review of mesh-controller#320).
func TestConfigRefusesATakesConfigKeyNotDeclared(t *testing.T) {
m := inConfigForm()
resourceOf(m, "rules")["takes-config"] = []any{"position", "theme", "urgency"}
refused(t, m, `"urgency"`, "takes-config", "declares no such value under config")
}
// A required value nothing sets is refused at composition in the config form's words: ${config:<key>} and the
// config verb, never `settings set` (review of mesh-controller#320). The settings form's words are unchanged.
func TestAnUnsetRequiredValueIsRefusedInTheConfigFormsWords(t *testing.T) {
m, err := parseRaw(t, inConfigForm())
if err != nil {
t.Fatal(err)
}
r := anAdoptedAnchor()
r.Modules = append(r.Modules, m)
composed, err := r.Compose(anchorRendering(false))
if err != nil {
t.Fatal(err)
}
why, left := composed.LeftOut["notifier"]
if !left {
t.Fatal("a module whose required value nothing sets was not left out")
}
for _, said := range []string{"${config:sender}", `required config value "sender"`, "config verb"} {
if !strings.Contains(why, said) {
t.Fatalf("left out without %q: %s", said, why)
}
}
for _, unsaid := range []string{"${setting:", "settings set"} {
if strings.Contains(why, unsaid) {
t.Fatalf("left out in the settings form's words (%q): %s", unsaid, why)
}
}
var unset *UnsetSettingError
if err := unsetInConfigWords(settingInto(map[string]any{"type": "file", "content": "${setting:sender}"}, nil,
"notifier"), m); !errors.As(err, &unset) || unset.Setting != "sender" {
t.Fatalf("not typed as a value nobody gave: %v", err)
}
settingsForm := notifier()
settingsForm.Resources[0]["content"] = "from = ${setting:domain}\n"
err = unsetInConfigWords(settingInto(settingsForm.Resources[0], nil, "notifier"), settingsForm)
if err == nil || !strings.Contains(err.Error(), "`settings set notifier <file>`") {
t.Fatalf("the settings form's refusal changed: %v", err)
}
}
+1 -1
View File
@@ -173,7 +173,7 @@ func TestTheBackupHoldersLinesAreDerivedFromTheData(t *testing.T) {
{"id":"meta","path":"${dir:meta}","class":"rebuildable"}]},
"resources":[{"id":"meta","type":"directory","mode":"0700"}]}`)
facts := map[string]string{"account-home": "/home/op"}
with := Rendering{Settings: SettingsBy{"media": {{From: "node", Values: map[string]any{AccessesSetting: map[string]any{"films": "/tank/films"}}}}}}
with := Rendering{Settings: SettingsBy{"media": {LayerOf("node", map[string]any{AccessesSetting: map[string]any{"films": "/tank/films"}})}}}
backup, _, err := seatContributions([]Manifest{pg, agent, media}, Manifest{}, BackupSeat+":"+BackupKindBackup, with, facts, nil)
if err != nil {
t.Fatal(err)
+1 -1
View File
@@ -1003,7 +1003,7 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// **An operator's value, from the assignment** (novox/hq ADR 0112, ADR 0155): what a
// definition may not carry because it is true of one installation only. Filled from
// the same layers a mergeable file takes, and refused when no layer set it.
if err := settingInto(copied, WithDefaults(m, with.Settings[m.Module]), m.Module); err != nil {
if err := unsetInConfigWords(settingInto(copied, WithDefaults(m, with.Settings[m.Module]), m.Module), m); err != nil {
return nil, err
}
// **Placed before anything reads a path.** A pathless directory receives the path
@@ -165,16 +165,19 @@ func TestTwoConsumersOfOneProviderGetTheirOwnDerivation(t *testing.T) {
}
}
// An operator may still set what the provider serves, and the mesh still derives the rest: the
// setting is laid on first, then the consumer's half is filled.
// An operator may still set what the provider serves where its definition asks (${setting:…}), and
// the mesh still derives the rest: the setting is filled first, then the consumer's half. (Until
// novox/hq issue 696 the layer replaced the served bucket whole; a layer replaces no served key now.)
func TestASettingComposesWithADerivedValue(t *testing.T) {
r, err := Resolve(shelf(store(), files()), []string{"store", "files"}, reachable(), World{})
prefixed := store()
prefixed.Serves["s3-bucket"]["bucket"] = "${setting:bucket-prefix}${consumer:as:dns}"
r, err := Resolve(shelf(prefixed, files()), []string{"store", "files"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{
Settings: SettingsBy{"store": {{From: "the operator",
Values: map[string]any{"bucket": "team-${consumer:as:dns}"}}}},
Values: map[string]any{"bucket-prefix": "team-"}}}},
Grants: []Grant{{Provision: "s3-bucket", Consumer: "workstation", From: "files", Slug: "files",
Values: map[string]any{}, Sealed: "c2VhbGVk"}},
})
+49 -3
View File
@@ -530,6 +530,11 @@ type Manifest struct {
// stays refused by name until a layer sets it. The mesh's layer, then the node's, override it.
Settings map[string]SettingDeclaration `json:"settings,omitempty"`
// forms is which of the two forms its config is written in, `config` or its alias `settings` (config.go,
// novox/hq ADR 0348 §1): the config form is read into Settings, a mergeable file's takes-settings and
// `${setting:<key>}`, and written back as it was read.
forms configForms
// unknown is the first key this manifest has that this controller does not know, at any depth, when
// it was read from the store (novox/hq ADR 0262): a manifest a newer controller registered.
// ParseManifest refuses it; the store's catalogue still loads, and the module is left out of every
@@ -1321,10 +1326,25 @@ type manifestFields Manifest
// strict read there failed the whole catalogue, and with it every plan and every send, the moment a
// controller was rolled back. UnknownField says what was set aside.
func (m *Manifest) UnmarshalJSON(raw []byte) error {
// The config form read into the settings form's fields, so both compose by one path (config.go).
raw, forms := toSettingsForm(raw)
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return err
}
if settings, has := keys[SettingsField]; has && string(settings) != "null" {
forms.settings = append(forms.settings, SettingsField)
}
if config, has := keys[ConfigField]; has {
if string(config) != "null" {
forms.config = append(forms.config, ConfigField)
}
if _, both := keys[SettingsField]; !both {
keys[SettingsField] = config
}
// Beside `settings` it is refused by name (MixedFormProblems), never merged into it.
delete(keys, ConfigField)
}
plain := map[string]string{}
many := map[string]map[string]string{}
if secrets, ok := keys["secrets"]; ok && string(secrets) != "null" {
@@ -1453,6 +1473,7 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
*m = Manifest(fields)
m.unknown = unknown
m.forms = forms
m.OptionalRequires = optionalRequires
if len(plain) > 0 {
m.Secrets = plain
@@ -1470,8 +1491,16 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
// MarshalJSON writes `secrets` and `contributes` back in the shape they were read: single values,
// and objects of local names.
// and objects of local names; and its config in the form it was read in (config.go).
func (m Manifest) MarshalJSON() ([]byte, error) {
raw, err := m.marshalFields()
if err != nil || !m.DeclaresConfig() {
return raw, err
}
return toConfigForm(raw)
}
func (m Manifest) marshalFields() ([]byte, error) {
raw, err := json.Marshal(manifestFields(m))
if err != nil {
return nil, err
@@ -1783,8 +1812,14 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, StateProblems(m)...)
// And what it may call its logs, and how large it may ask them to grow (logs.go, novox/hq ADR 0297).
problems = append(problems, LogProblems(m)...)
// And the defaults it gives its settings (setting_defaults.go, novox/hq ADR 0262).
problems = append(problems, SettingProblems(m)...)
// And the defaults it gives its settings (setting_defaults.go, novox/hq ADR 0262), or, written in the
// config form, the whole declaration of each value (config.go, novox/hq ADR 0348).
if m.DeclaresConfig() {
problems = append(problems, ConfigProblems(m)...)
} else {
problems = append(problems, SettingProblems(m)...)
problems = append(problems, TypedSettingProblems(m)...)
}
// And what it requires optionally, and the files it delivers only while bound (novox/hq ADR 0330).
problems = append(problems, OptionalProblems(m)...)
// And which setting keys each mergeable file takes (setting_takes.go, novox/hq ADR 0326).
@@ -1853,6 +1888,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
problems = append(problems, m.Build.problems(m.Module)...)
// A file's text in a template file of the module's repository (novox/hq ADR 0349 §2): only a file, never beside
// content, only under files/, and packed by the module's build.
problems = append(problems, templateProblems(m)...)
problems = append(problems, undeliveredBundles(m)...)
// **What provides the artifact store cannot be delivered through it** (novox/hq 04-ISSUES/029).
//
@@ -2338,6 +2376,14 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, healthProblems(m)...)
// And whether a container is busy (novox/hq ADR 0335), likewise.
problems = append(problems, busyProblems(m)...)
if m.DeclaresConfig() {
// Said in the words it was written in: the rules shared with the settings form say ${setting:<key>}
// and takes-settings.
for i, p := range problems {
problems[i] = inConfigWords(p)
}
problems = append(problems, MixedFormProblems(m)...)
}
if len(problems) > 0 {
sort.Strings(problems)
return Manifest{}, fmt.Errorf("this manifest cannot be used:\n - %s",
+65 -2
View File
@@ -34,6 +34,69 @@ const PlacesSetting = "places"
// AccessesSetting is the settings key that says where a module's accesses are on this node, by id.
const AccessesSetting = "accesses"
// **Placement is not config** (novox/hq ADR 0348 clause 11, issue 651). `places`, `accesses` and
// `networks` are facts of an assignment the controller decides, held beside the assignment and apart
// from its settings: a Layer carries them in Placement, never in Values, and only Places, AccessPlaces
// and KeptNetworks read them. A person still says them as keys of a layer through `settings`; LayerOf
// splits such a layer, so what they type is routed to the placement and nothing they do changes.
// placementKeys are the keys of a layer that are placement, not config.
var placementKeys = map[string]bool{PlacesSetting: true, AccessesSetting: true, NetworksSetting: true}
// SplitPlacement divides a layer as a person says it into its config values and its placement. Each is
// nil when it holds nothing; the values themselves are not copied.
func SplitPlacement(values map[string]any) (config, placement map[string]any) {
for key, value := range values {
if placementKeys[key] {
if placement == nil {
placement = map[string]any{}
}
placement[key] = value
continue
}
if config == nil {
config = map[string]any{}
}
config[key] = value
}
return config, placement
}
// LayerOf is a layer as a person says it — config values and placement in one map — split into the two.
func LayerOf(from string, values map[string]any) Layer {
config, placement := SplitPlacement(values)
if config == nil {
config = map[string]any{}
}
return Layer{From: from, Values: config, Placement: placement}
}
// Whole is the layer as a person says it: its values and its placement in one map, a new map.
func (l Layer) Whole() map[string]any {
out := make(map[string]any, len(l.Values)+len(l.Placement))
for k, v := range l.Values {
out[k] = v
}
for k, v := range l.Placement {
out[k] = v
}
return out
}
// asSaid is the layers with placement folded back into the values, for what reads a layer whole as it
// did while placement was kept in it: today's mergeable file, which takes every key (ADR 0326 §3), and
// the judgement that no value holds a line end.
func asSaid(layers []Layer) []Layer {
out := make([]Layer, len(layers))
for i, l := range layers {
out[i] = l
if len(l.Placement) > 0 {
out[i] = Layer{From: l.From, Values: l.Whole(), Default: l.Default}
}
}
return out
}
// Placement is what an assignment says about one of a module's directories.
type Placement struct {
// Path is where the directory is on this machine. Absolute.
@@ -208,7 +271,7 @@ func Places(m Manifest, layers []Layer) (map[string]Placement, error) {
}
out := map[string]Placement{}
for _, layer := range layers {
raw, ok := layer.Values[PlacesSetting]
raw, ok := layer.Placement[PlacesSetting]
if !ok {
continue
}
@@ -278,7 +341,7 @@ func AccessPlaces(m Manifest, layers []Layer) (map[string]string, error) {
}
out := map[string]string{}
for _, layer := range layers {
raw, ok := layer.Values[AccessesSetting]
raw, ok := layer.Placement[AccessesSetting]
if !ok {
continue
}
+41 -1
View File
@@ -23,7 +23,7 @@ func placeable() Manifest {
}
func placedBy(values map[string]any) Rendering {
return Rendering{Settings: SettingsBy{"arr": {{From: "node anchor", Values: values}}}}
return Rendering{Settings: SettingsBy{"arr": {LayerOf("node anchor", values)}}}
}
func TestAnAssignmentPlacesDirectoriesAndAccesses(t *testing.T) {
@@ -179,3 +179,43 @@ func TestAnUnknownAccessReferenceIsRefusedAtParse(t *testing.T) {
t.Fatalf("an access with no id and no path was accepted: %v", err)
}
}
// **Placement is read from the placement, never from a layer's values** (novox/hq ADR 0348 clause 11, issue
// 651). A `places` or `accesses` among the config values places nothing; the same in Placement does. A
// mergeable file of today's form still takes every key of the layer as it is said, placement too, so it
// composes as it did while placement was kept in the layer.
func TestPlacementIsReadFromThePlacementOnly(t *testing.T) {
m := placeable()
said := map[string]any{
PlacesSetting: map[string]any{"config": "/services/arr/config"},
AccessesSetting: map[string]any{"series": "/storage/media/series", "spool": "/storage/downloads"},
"log": "warn",
}
among := []Layer{{From: "anchor", Values: said}}
if placed, err := Places(m, among); err != nil || placed != nil {
t.Fatalf("a place among the config values placed %v (%v)", placed, err)
}
if _, _, err := accessesFor(m, among); err == nil {
t.Fatal("an access among the config values placed it")
}
beside := []Layer{LayerOf("anchor", said)}
if beside[0].Values["log"] != "warn" || beside[0].Values[PlacesSetting] != nil || beside[0].Placement[PlacesSetting] == nil {
t.Fatalf("LayerOf split %+v", beside[0])
}
if placed, err := Places(m, beside); err != nil || placed["config"].Path != "/services/arr/config" {
t.Fatalf("the placement did not place: %v (%v)", placed, err)
}
file := map[string]any{"id": "runtime", "type": "file", "path": "/x", "content": `{"a":1}`, "merge": "json"}
whole, err := ApplySettings(file, among)
if err != nil {
t.Fatal(err)
}
split, err := ApplySettings(file, beside)
if err != nil {
t.Fatal(err)
}
if whole["content"] != split["content"] || !strings.Contains(split["content"].(string), `"places"`) {
t.Fatalf("today's mergeable file composes %q, was %q", split["content"], whole["content"])
}
}
+13 -1
View File
@@ -128,7 +128,19 @@ var defaultSeats = append([]Seat{
// The vault: the controller seals every minted credential with what it provides, which is the
// test for a seat of the mesh's own (novox/hq ADR 0161) — a second provider of `secret` is a
// second claimant, refused by name, rather than a candidate for a pin.
{Name: "mesh-vault", Scope: ScopeMesh, Delivers: "secret", Decision: "novox/hq ADR 0161"},
//
// Its one verb, `get`, hands a secret on request and reply, sealed to the key the asker sends and only
// to an asker the vault allows (novox/hq ADR 0349 §11, issue 674). Optional until the vault that runs
// serves it (design 33 §7).
{Name: "mesh-vault", Scope: ScopeMesh, Delivers: "secret", Decision: "novox/hq ADR 0161",
Serves: []Verb{
{Name: "get", Optional: true, Description: "One secret, sealed to the public key the asker sends (the " +
"sealed box of ADR 0206), with its generation; answered only to an asker the vault allows, as the " +
"bus proved it. Never the value in the clear, and never to a private key.",
Input: schema(map[string]string{"name": "the secret's name",
"public_key": "the asker's X25519 public key as SPKI PEM — never a private key"},
[]string{"name", "public_key"})},
}},
// Named for its scope since 2026-09-30 (novox/hq ADR 0156); `the-artifact-store` resolves to it as
// an alias on a mesh that predates the rename. It serves artifacts of every kind a build makes —
// images and archives, by digest — which is why the provision is the artifact store and not an
@@ -0,0 +1,194 @@
package catalogue
import (
"bytes"
"encoding/json"
"os"
"strings"
"testing"
)
// influxdb and grafana as the catalogue declares them, cut to what composes the data source on a
// machine like ace (novox/hq issue 696): what influxdb serves for influxdb-api and its mergeable
// runtime configuration, and grafana's binding, data source file and credential.
const grafanaLikeTheCatalogue = `{
"module": "grafana", "version": "1",
"requires": [{"name": "influxdb-api", "optional": true}],
"contributes": {"influxdb-api": {"access": "read"}},
"binds": {"influxdb-api": "${dir:state}/influxdb.json"},
"secrets": {"influxdb-api": "${dir:mesh-state}/influxdb-api"},
"resources": [
{"id": "mesh-state", "type": "directory", "mode": "0700", "place": "mesh"},
{"id": "state", "type": "directory", "mode": "0700", "place": "."},
{"id": "influxdb-datasources", "type": "directory", "mode": "0755", "owner": "472:0"},
{"id": "influxdb-datasource", "type": "file", "path": "${dir:influxdb-datasources}/mesh-influxdb.yaml", "mode": "0644",
"content": "apiVersion: 1\n# Written by the mesh from grafana's influxdb-api binding, only while it is bound (novox/hq ADR 0330):\n# without influxdb, grafana runs with no mesh data source. grafana reads it at start. Its own name and\n# uid, so a data source somebody made in the UI is never overwritten, and read-only in the UI because\n# the mesh resets it. The password is read from the file the mesh delivers, never written here.\ndatasources:\n - name: InfluxDB (mesh)\n uid: mesh-influxdb-api\n type: influxdb\n access: proxy\n url: ${bound:influxdb-api:scheme}://${bound:influxdb-api:at}:${bound:influxdb-api:port}\n user: ${bound:influxdb-api:as}\n isDefault: false\n editable: false\n jsonData:\n dbName: ${bound:influxdb-api:bucket}\n httpMode: POST\n secureJsonData:\n password: $__file{/run/secrets/influxdb/influxdb-api}\n",
"only-when-bound": "influxdb-api"}
]
}`
// influxdbServing is influxdb as the catalogue declared it before issue 696 (literal served org and
// bucket) or after (served from its settings, today's literals as their defaults).
func influxdbServing(after bool) string {
served := `"org": "mesh", "bucket": "default"`
settings, takes := ``, ``
if after {
served = `"org": "${setting:org}", "bucket": "${setting:bucket}"`
settings = `"settings": {
"org": {"kind": "preference", "default": "mesh", "why": "the org influxdb makes at its first start"},
"bucket": {"kind": "preference", "default": "default", "why": "the bucket influxdb makes at its first start"}},`
takes = `, "takes-settings": ["bucket", "org", "url"], "trusted": true`
}
return `{
"module": "influxdb", "version": "1",
"provides": [{"name": "influxdb-api", "scope": "mesh", "identity": {"in": "an InfluxDB v1 authorization"}}],
"listens": [{"name": "api", "port": 8086, "protocol": "tcp", "from": "mesh", "why": "queries and writes"}],
` + settings + `
"serves": {"influxdb-api": {"scheme": "http", "port": 8086, ` + served + `}},
"resources": [
{"id": "mesh-state", "type": "directory", "mode": "0700", "place": "mesh"},
{"id": "runtime-config", "type": "file", "path": "${dir:mesh-state}/config.json", "mode": "0600",
"content": "{}\n", "merge": "json"` + takes + `}
]
}`
}
// composedLikeAce is every resource of grafana and influxdb as one machine composes them, by id, with
// influxdb's settings layer on it.
func composedLikeAce(t *testing.T, influxdb string, layers []Layer) map[string]string {
t.Helper()
var ms []Manifest
for _, raw := range []string{influxdb, grafanaLikeTheCatalogue} {
m, err := ParseManifest([]byte(raw))
if err != nil {
t.Fatal(err)
}
ms = append(ms, m)
}
r, err := Resolve(shelf(ms...), []string{"influxdb", "grafana"}, reachable(), World{})
if err != nil {
t.Fatal(err)
}
out, err := r.Declaration(Rendering{Settings: SettingsBy{"influxdb": layers},
Needed: map[string]map[string]string{}})
if err != nil {
t.Fatal(err)
}
got := map[string]string{}
for _, res := range out {
id, _ := res["id"].(string)
if !strings.HasPrefix(id, "grafana.") && !strings.HasPrefix(id, "influxdb.") {
continue
}
b, err := json.Marshal(res)
if err != nil {
t.Fatal(err)
}
got[id] = string(b)
}
return got
}
// aceLikeLayer is a layer like influxdb's on ace: its own org and bucket (made up here), and the
// endpoint block the mesh reads itself.
func aceLikeLayer() []Layer {
return []Layer{{From: "ace", Values: map[string]any{
"org": "made-up-org", "bucket": "made-up-bucket",
"endpoints": map[string]any{"api": map[string]any{"label": "influx", "port": float64(8086), "reach": "internal"}},
}}}
}
// grafana composes byte for byte as it did before issue 696, on a machine like ace and on one
// without a layer, now that influxdb serves its org and bucket from its settings and no layer
// replaces a served key. The golden files are grafana's resources as the integration branch's
// controller composed them at 71362c1 (Settle still overriding), with influxdb serving the literals.
func TestGrafanaComposesTheSameOnceInfluxdbServesItsSettings(t *testing.T) {
for name, layers := range map[string][]Layer{"ace": aceLikeLayer(), "unlayered": nil} {
raw, err := os.ReadFile("testdata/influxdb-served-by-setting/grafana-" + name + ".json")
if err != nil {
t.Fatal(err)
}
var want map[string]json.RawMessage
if err := json.Unmarshal(raw, &want); err != nil {
t.Fatal(err)
}
got := composedLikeAce(t, influxdbServing(true), layers)
seen := 0
for id, v := range got {
if !strings.HasPrefix(id, "grafana.") {
continue
}
seen++
w, ok := want[id]
if !ok {
t.Errorf("%s: %s composes and did not before", name, id)
continue
}
var compact bytes.Buffer
if err := json.Compact(&compact, w); err != nil {
t.Fatal(err)
}
if compact.String() != v {
t.Errorf("%s: %s changed:\n got %s\nwant %s", name, id, v, compact.String())
}
}
if seen != len(want) {
t.Errorf("%s: %d of grafana's %d resources compose", name, seen, len(want))
}
}
}
// What a provider serves is its definition's, filled from settings only where it asks with
// ${setting:<key>} (novox/hq issue 696, ADR 0348 clause 11): a layer that sets a key the provider
// serves as a literal replaces nothing, and a key it does not serve is never added.
func TestALayerNoLongerReplacesAServedKey(t *testing.T) {
served, err := Settle(map[string]any{"host": "mail", "port": 25},
[]Layer{{From: "the mesh", Values: map[string]any{"host": "post", "sitename": "Mail"}}})
if err != nil {
t.Fatal(err)
}
if served["host"] != "mail" {
t.Errorf("a layer replaced the served host: %v", served)
}
if _, leaked := served["sitename"]; leaked {
t.Errorf("a setting for a file reached the consumers: %v", served)
}
// And a layer setting it is said to reach nothing, rather than taken for a change.
m := Manifest{Module: "mail", Serves: map[string]map[string]any{"smtp": {"host": "mail"}}}
if unused := UnusedSettings(m, []Layer{{From: "the mesh", Values: map[string]any{"host": "post"}}}); len(unused) != 1 {
t.Errorf("a layer setting a literal served key was not said to reach nothing: %v", unused)
}
// influxdb as it was before issue 696, serving literals, under a layer like ace's: the layer no
// longer reaches grafana — which is why the catalogue moves influxdb first.
got := composedLikeAce(t, influxdbServing(false), aceLikeLayer())
for _, id := range []string{"grafana.influxdb-datasource", "grafana.bound-influxdb-api"} {
if strings.Contains(got[id], "made-up") {
t.Errorf("%s still carries the layer's value over a literal served key: %s", id, got[id])
}
}
if !strings.Contains(got["grafana.influxdb-datasource"], `dbName: default\n`) {
t.Errorf("the data source does not name the served literal: %s", got["grafana.influxdb-datasource"])
}
}
// influxdb's own runtime configuration takes the keys it names: org and bucket from a layer like
// ace's, and not the endpoint block the mesh reads itself, which today's form wrote there too.
func TestInfluxdbsConfigTakesItsOrgAndBucket(t *testing.T) {
got := composedLikeAce(t, influxdbServing(true), aceLikeLayer())["influxdb.runtime-config"]
var r map[string]any
if err := json.Unmarshal([]byte(got), &r); err != nil {
t.Fatal(err)
}
if want := "{\n \"bucket\": \"made-up-bucket\",\n \"org\": \"made-up-org\"\n}\n"; r["content"] != want {
t.Errorf("config.json: got %q, want %q", r["content"], want)
}
got = composedLikeAce(t, influxdbServing(true), nil)["influxdb.runtime-config"]
if err := json.Unmarshal([]byte(got), &r); err != nil {
t.Fatal(err)
}
if r["content"] != "{}\n" {
t.Errorf("a default reached the mergeable file: %q", r["content"])
}
}
+28 -1
View File
@@ -31,6 +31,28 @@ type SettingDeclaration struct {
Default any `json:"default"`
// Why this default: one sentence for whoever wonders whether to change it.
Why string `json:"why"`
// The config form's declaration (novox/hq ADR 0348 §4, config.go): none of these is said under
// `settings`, and neither Kind nor Why under `config`.
// Type is number, string or boolean; object or list only when Trusted.
Type string `json:"type,omitempty"`
// Description says what the value is, including its unit.
Description string `json:"description,omitempty"`
// Enum, Minimum, Maximum and Pattern are its validation: the values it may take, the bounds of a
// number, and the whole of a string matching the pattern.
Enum []any `json:"enum,omitempty"`
Minimum *float64 `json:"minimum,omitempty"`
Maximum *float64 `json:"maximum,omitempty"`
Pattern string `json:"pattern,omitempty"`
// Trusted is a value that decides what runs or who can reach what (ADR 0348 §8).
Trusted bool `json:"trusted,omitempty"`
// Predefined is the predefined key the value falls back to, of the same type (ADR 0348 §2, §7).
Predefined string `json:"predefined,omitempty"`
// Required is the operator's own value: it has no default (ADR 0348 §4).
Required bool `json:"required,omitempty"`
// Runtime is a value the module's code reads at runtime rather than one placed in a file (ADR 0348 §5).
Runtime bool `json:"runtime,omitempty"`
}
// KindPreference is the kind of a setting that may have a default.
@@ -203,11 +225,16 @@ type SettingSource struct {
func Effective(m Manifest, layers []Layer) []SettingSource {
byKey := map[string]*SettingSource{}
for key, d := range m.Settings {
if d.Default == nil {
// A required config value has no default (ADR 0348 §4); a settings declaration always has one.
continue
}
byKey[key] = &SettingSource{Key: key, Value: d.Default, From: DefaultLayer, FromDefault: true,
Default: d.Default, HasDefault: true}
}
for _, layer := range layers {
for key, v := range layer.Values {
// As the layer is said, placement too: `settings` answers it as a person set it.
for key, v := range layer.Whole() {
s, ok := byKey[key]
if !ok {
s = &SettingSource{Key: key}
+1 -1
View File
@@ -427,7 +427,7 @@ func TestALeftOutModulesUnplaceableBackupLineCostsOnlyThatLine(t *testing.T) {
r := anAdoptedAnchor()
r.Modules = append(r.Modules, holder, media, placedBadly)
with := anchorRendering(false)
with.Settings["notes"] = []Layer{{From: "anchor", Values: map[string]any{PlacesSetting: "not a map"}}}
with.Settings["notes"] = []Layer{LayerOf("anchor", map[string]any{PlacesSetting: "not a map"})}
composed, err := r.Compose(with)
if err != nil {
t.Fatalf("an unplaceable line of a left-out module failed the machine: %v", err)
+41 -16
View File
@@ -1,6 +1,7 @@
package catalogue
import (
"errors"
"fmt"
"regexp"
"sort"
@@ -47,10 +48,30 @@ func settingsUsed(content string) []string {
type UnsetSettingError struct {
Module, Setting string
said string
// where says what asks for it — "a file", "a service whose unit", "a directory <id> whose owner" — and
// setToday which of the module's keys a layer sets, for saying it again in the config form's words.
where, setToday string
}
func (e *UnsetSettingError) Error() string { return e.said }
// unsetInConfigWords is the refusal of a required config value nothing sets, in the words of a module written in the
// config form (novox/hq ADR 0348 §1, §4): `${config:<key>}` and the config verb, never `settings set`. Any other
// error, and any error of a module in the settings form, is returned as it was.
func unsetInConfigWords(err error, m Manifest) error {
var unset *UnsetSettingError
if err == nil || !m.DeclaresConfig() || !errors.As(err, &unset) {
return err
}
return &UnsetSettingError{Module: unset.Module, Setting: unset.Setting, where: unset.where,
setToday: unset.setToday, said: fmt.Sprintf(
"%s has %s that says ${config:%s}, and nothing sets the required config value %q for it — a "+
"required value is the operator's own and has no default (novox/hq ADR 0348 §4): set it with "+
"the config verb, for %s on this node, key %q%s",
unset.Module, unset.where, unset.Setting, unset.Setting, unset.Module, unset.Setting,
strings.ReplaceAll(unset.setToday, "no setting is set", "no config value is set"))}
}
// settingInto fills a file's ${setting:…} placeholders from the layers over a module.
//
// The last layer setting a key wins, which is the node's over the mesh's over the module's own
@@ -74,12 +95,13 @@ func settingInto(resource map[string]any, layers []Layer, module string) error {
for _, key := range settingsUsed(content) {
value, set := settingValue(layers, key)
if !set {
return &UnsetSettingError{Module: module, Setting: key, said: fmt.Sprintf(
"%s has a file that says ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112), and only a "+
"preference has a default in the definition (ADR 0262): "+
"`settings set %s <file>` with {%q: …}%s",
module, key, key, module, key, orNoSettings(layers))}
return &UnsetSettingError{Module: module, Setting: key, where: "a file",
setToday: orNoSettings(layers), said: fmt.Sprintf(
"%s has a file that says ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112), and only a "+
"preference has a default in the definition (ADR 0262): "+
"`settings set %s <file>` with {%q: …}%s",
module, key, key, module, key, orNoSettings(layers))}
}
content = strings.ReplaceAll(content, "${setting:"+key+"}", plainly(value))
}
@@ -130,11 +152,12 @@ func settingIntoUnit(resource map[string]any, layers []Layer, module string) err
for _, key := range settingsUsed(unit) {
value, set := settingValue(layers, key)
if !set {
return &UnsetSettingError{Module: module, Setting: key, said: fmt.Sprintf(
"%s has a service whose unit says ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112): "+
"`settings set %s <file>` with {%q: …}%s",
module, key, key, module, key, orNoSettings(layers))}
return &UnsetSettingError{Module: module, Setting: key, where: "a service whose unit",
setToday: orNoSettings(layers), said: fmt.Sprintf(
"%s has a service whose unit says ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112): "+
"`settings set %s <file>` with {%q: …}%s",
module, key, key, module, key, orNoSettings(layers))}
}
v := plainly(value)
if !unitPart.MatchString(v) {
@@ -229,11 +252,13 @@ func settingIntoOwner(resource map[string]any, layers []Layer, module string) er
for _, key := range settingsUsed(owner) {
value, set := settingValue(layers, key)
if !set {
return &UnsetSettingError{Module: module, Setting: key, said: fmt.Sprintf(
"%s has a directory %v whose owner says ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112): "+
"`settings set %s <file>` with {%q: …}%s",
module, resource["id"], key, key, module, key, orNoSettings(layers))}
return &UnsetSettingError{Module: module, Setting: key,
where: fmt.Sprintf("a directory %v whose owner", resource["id"]), setToday: orNoSettings(layers),
said: fmt.Sprintf(
"%s has a directory %v whose owner says ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112): "+
"`settings set %s <file>` with {%q: …}%s",
module, resource["id"], key, key, module, key, orNoSettings(layers))}
}
v := plainly(value)
id, whole := wholeID(v)
+2 -1
View File
@@ -83,7 +83,8 @@ func TakenSettingsProblems(m Manifest) []string {
// Said only when none of the module's mergeable files names its keys: when one does, §2 already
// names every file that does not, and a file is not reported twice.
func WrittenSettingsProblems(m Manifest) []string {
if len(m.Settings) == 0 {
if len(m.Settings) == 0 || m.DeclaresConfig() {
// A module in the config form is held to this where it is read, in its own words (ConfigProblems).
return nil
}
var silent []string
+44 -27
View File
@@ -37,6 +37,11 @@ type Layer struct {
// Default marks the layer a module's own defaults make (novox/hq ADR 0262). Marked rather than
// recognised by From, which is a node's name for a node's layer, and a node may be called anything.
Default bool
// Placement is the assignment's placement at this layer's scope — `places`, `accesses`, `networks`
// — which the controller holds apart from settings (novox/hq ADR 0348 clause 11, issue 651): read
// by Places, AccessPlaces and KeptNetworks, and never from Values. LayerOf splits a layer as a person
// says it into the two.
Placement map[string]any
}
// ApplySettings produces a resource's final content from the module's own and the layers over it.
@@ -45,6 +50,10 @@ type Layer struct {
// module first, then this node's. A node that differs from the rest is expressed by differing,
// rather than by repeating everything the rest already say.
func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, error) {
// Today's form reaches a mergeable file with every key of the layer as it was said, placement too
// (novox/hq ADR 0326 §3, ADR 0348 clause 11: a module not yet moved keeps it), so a file composes
// byte for byte as it did while placement was kept in the layer.
layers = asSaid(layers)
how, _ := resource["merge"].(string)
taken, names := takenKeys(resource)
if how == "" {
@@ -108,25 +117,27 @@ func ApplySettings(resource map[string]any, layers []Layer) (map[string]any, err
return out, nil
}
// Settle lays settings over what a provider serves. Exported because a served fact is settled where
// the mesh is walked rather than where a node is declared.
// Settle fills what a provider serves from its settings. Exported because a served fact is settled
// where the mesh is walked rather than where a node is declared.
//
// **A setting overrides a served key; it never adds one** (novox/hq 04-ISSUES/173). What a consumer
// is told is the provider's contract, and a setting made for one of the provider's files — a site
// name, a public address — is not part of it. Before this, every setting of a module reached every
// consumer of every provision it served.
// **What a provider serves is its definition's; a layer replaces none of it** (novox/hq ADR 0348
// clause 11, issue 696). A served value is filled from the layers only where the definition asks,
// with `${setting:<key>}` (ADR 0155) — a mail provider's domain, an identity provider's issuer, a
// time-series store's org and bucket — and a layer that sets a key the provider serves as a literal
// changes nothing a consumer is told. Nor does a setting add a key (novox/hq 04-ISSUES/173): a
// setting made for one of the provider's files — a site name, a public address — is not part of the
// contract. Until issue 696, a layer replaced any served key it named; influxdb on ace was the one
// layer relying on it, and serves `${setting:org}` and `${setting:bucket}` since.
func Settle(base map[string]any, layers []Layer) (map[string]any, error) {
return overridden(base, layers, "what is served")
return filledFromSettings(deepCopy(base), layers, "what is served")
}
// overridden lays settings over a map whose keys are its contract: a contribution, a served fact.
// Only the keys the map already declares are touched; the rest of a layer is somebody else's
// business (a file's, another destination's) and is left to reach it there.
// overridden lays settings over a contribution: a map whose keys are its contract. Only the keys the
// map already declares are touched; the rest of a layer is somebody else's business (a file's,
// another destination's) and is left to reach it there.
//
// A declared value may itself be the operator's, `${setting:<key>}` (ADR 0155): a mail provider
// serves its domain, an identity provider its issuer, and neither is the definition's to state.
// Filled from the layers after the overrides, and refused by name when nothing sets it — a literal
// placeholder handed to a consumer is a service configured against a string nobody meant.
// A declared value may itself be the operator's, `${setting:<key>}` (ADR 0155), filled after the
// overrides (filledFromSettings).
func overridden(base map[string]any, layers []Layer, what string) (map[string]any, error) {
kept := make([]Layer, 0, len(layers))
for _, layer := range layers {
@@ -142,6 +153,14 @@ func overridden(base map[string]any, layers []Layer, what string) (map[string]an
if err != nil {
return nil, err
}
return filledFromSettings(merged, layers, what)
}
// filledFromSettings fills each `${setting:<key>}` in a map's top-level strings from the layers,
// later beating earlier and a module's defaults lowest, and refuses one nothing sets by name — a
// literal placeholder handed to a consumer is a service configured against a string nobody meant.
// It fills merged in place and returns it.
func filledFromSettings(merged map[string]any, layers []Layer, what string) (map[string]any, error) {
for key, value := range merged {
s, ok := value.(string)
if !ok {
@@ -168,7 +187,8 @@ func overridden(base map[string]any, layers []Layer, what string) (map[string]an
// module says what it means by default, and somebody says what it means here. A contribution that
// could not be settled would have to be edited to be reused anywhere else. The two differ in one
// respect, and the caller decides it: a file takes keys it did not declare (a setting may add to a
// configuration), a contribution or served fact does not (overridden).
// configuration), a contribution does not (overridden). A served fact takes no layer at all; only
// its ${setting:<key>} is filled (Settle).
func settle(base map[string]any, layers []Layer, protected map[string]bool, what string) (
map[string]any, error) {
merged := deepCopy(base)
@@ -344,8 +364,9 @@ func sortedKeysAny(m map[string]any) []string {
// way there is. This is what makes that visible at the moment they set it.
//
// Where a key can land: any mergeable file takes any key; a file asking for `${setting:<key>}`
// takes that key (ADR 0155); a contribution or a served fact takes a key it declares, and no other
// (novox/hq 04-ISSUES/173); and the mesh's own words — `expose`, `ports`, `reach`, `endpoints` —
// takes that key (ADR 0155); a contribution takes a key it declares, and no other (novox/hq
// 04-ISSUES/173); a served fact takes only the keys it asks for as `${setting:<key>}`, since a layer
// replaces no served key (novox/hq issue 696); and the mesh's own words — `expose`, `ports`, `reach`, `endpoints` —
// are read by the mesh. A key none of those takes is stray, and is said so rather than dropped.
func UnusedSettings(m Manifest, layers []Layer) []string {
named := namesItsKeys(m)
@@ -388,14 +409,9 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
}
}
}
for _, served := range m.Serves {
for key := range served {
lands[key] = true
}
}
var unused []string
for _, layer := range layers {
for key := range layer.Values {
for key := range layer.Whole() {
if lands[key] {
continue
}
@@ -435,7 +451,7 @@ func UnusedSettings(m Manifest, layers []Layer) []string {
}
unused = append(unused, fmt.Sprintf(
"%s sets %q, and %s has no file that merges it, asks for no ${setting:%s}, and "+
"declares no %q in what it contributes or serves%s",
"declares no %q in what it contributes%s",
layer.From, key, m.Module, key, key, takenSaid(m)))
}
}
@@ -483,7 +499,7 @@ func KeptNetworks(m Manifest, layers []Layer, adopted bool) (map[string][]string
}
out := map[string][]string{}
for _, layer := range layers {
raw, ok := layer.Values[NetworksSetting]
raw, ok := layer.Placement[NetworksSetting]
if !ok {
continue
}
@@ -543,7 +559,8 @@ var networkName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
// refused where it is stored (SetSettings, with UnusedSettings) and said where a plan is read,
// and never costs a module its place.
func JudgeSettings(m Manifest, layers []Layer, adopted bool) error {
if err := settingsHoldOneLine(m.Module, layers); err != nil {
// Placement is judged for line ends as it always was: a path is written into files as well.
if err := settingsHoldOneLine(m.Module, asSaid(layers)); err != nil {
return err
}
// Which keys each mergeable file takes is the definition's to say right (novox/hq ADR 0326);
@@ -580,7 +597,7 @@ func JudgeSettings(m Manifest, layers []Layer, adopted bool) error {
for k, v := range settled {
copied[k] = v
}
if err := settingInto(copied, WithDefaults(m, layers), m.Module); err != nil {
if err := unsetInConfigWords(settingInto(copied, WithDefaults(m, layers), m.Module), m); err != nil {
return err
}
}
+7 -7
View File
@@ -27,7 +27,7 @@ func TestASettingThatCannotComposeIsRefusedByNameAndLeavesOnlyItsModuleOut(t *te
if c.name == "a mesh-wide port" {
from = MeshWideLayer
}
err := JudgeSettings(web, []Layer{{From: from, Values: c.layer}}, true)
err := JudgeSettings(web, []Layer{LayerOf(from, c.layer)}, true)
if err == nil || !strings.Contains(err.Error(), c.refuse) {
t.Errorf("%s: judged %v, want %q", c.name, err, c.refuse)
}
@@ -69,8 +69,8 @@ func TestASettingThatCannotComposeIsRefusedByNameAndLeavesOnlyItsModuleOut(t *te
// declaration as the networks it also joins.
func TestAKeptNetworkReachesTheContainerOnAnAdoptedMachineOnly(t *testing.T) {
r := anAdoptedAnchor()
keep := SettingsBy{"hello-web": {{From: "anchor",
Values: map[string]any{NetworksSetting: map[string]any{"server": []any{"predecessor_default"}}}}}}
keep := SettingsBy{"hello-web": {LayerOf("anchor",
map[string]any{NetworksSetting: map[string]any{"server": []any{"predecessor_default"}}})}}
with := anchorRendering(true)
with.Settings = keep
@@ -107,13 +107,13 @@ func TestAKeptNetworkReachesTheContainerOnAnAdoptedMachineOnly(t *testing.T) {
layer Layer
want string
}{
{"mesh-wide", Layer{From: MeshWideLayer, Values: map[string]any{NetworksSetting: map[string]any{"server": []any{"x"}}}},
{"mesh-wide", LayerOf(MeshWideLayer, map[string]any{NetworksSetting: map[string]any{"server": []any{"x"}}}),
"a found network is a fact about one machine"},
{"an unknown container", Layer{From: "anchor", Values: map[string]any{NetworksSetting: map[string]any{"db": []any{"x"}}}},
{"an unknown container", LayerOf("anchor", map[string]any{NetworksSetting: map[string]any{"db": []any{"x"}}}),
`names the container "db", which it does not declare`},
{"not a list", Layer{From: "anchor", Values: map[string]any{NetworksSetting: map[string]any{"server": "x"}}},
{"not a list", LayerOf("anchor", map[string]any{NetworksSetting: map[string]any{"server": "x"}}),
"is a list of network names"},
{"not a network name", Layer{From: "anchor", Values: map[string]any{NetworksSetting: map[string]any{"server": []any{"a/b"}}}},
{"not a network name", LayerOf("anchor", map[string]any{NetworksSetting: map[string]any{"server": []any{"a/b"}}}),
"which is not a network name"},
} {
_, err := KeptNetworks(web, []Layer{c.layer}, true)
-16
View File
@@ -190,22 +190,6 @@ func TestASettingLandsOnlyWhereSomethingDeclaresIt(t *testing.T) {
}
}
func TestASettingOverridesAServedKeyAndAddsNone(t *testing.T) {
// What a consumer is told is the provider's contract. A setting made for one of the
// provider's files — its site name, its public address — is not part of it.
served, err := Settle(map[string]any{"host": "mail", "port": 25},
[]Layer{{From: "the mesh", Values: map[string]any{"host": "post", "sitename": "Mail"}}})
if err != nil {
t.Fatal(err)
}
if served["host"] != "post" {
t.Errorf("the setting did not override the served host: %v", served)
}
if _, leaked := served["sitename"]; leaked {
t.Errorf("a setting for a file reached the consumers: %v", served)
}
}
func TestContentThatIsNotJSONIsRefusedWhereSomebodyIsLooking(t *testing.T) {
// Rather than on the machine, at apply time, as a file the program cannot read.
_, err := ApplySettings(file(`this is not json`), nil)
+334
View File
@@ -0,0 +1,334 @@
package catalogue
import (
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path"
"path/filepath"
"sort"
"strings"
"unicode/utf8"
)
// A file's text kept as a template file in its module's repository (novox/hq ADR 0349 §2, design 57 package T1,
// issue 654).
//
// **Two forms, one composition.** A file resource says its text inline (`content`) or names a template file
// beside its manifest (`"template": "files/<name>"`), never both. The module's build packs `files/` into an
// archive artifact of its own — an `archive` artifact built `from: "files"` — and resolving the build reads each
// template's text back from that packed archive into `content` and drops `template`. So the manifest the mesh
// holds, composition and the node-engine are exactly as they were: a module with its text in `files/` composes
// byte-identically to the same module with it inline. Where there is no build — `module check`, the merge gate,
// `module add` — the text is read from the repository tree beside the manifest (WithTemplatesFrom).
//
// **Read back by the builder, carried in the resolved manifest.** The builder reads each template from the very
// bytes it hashed into the archive's digest, so the text the mesh holds is the archive's. The controller
// registers the resolved manifest as the build result carries it and fetches nothing from the artifact store; a
// manifest that still names a template when it would be held — built by a builder that predates templates — is
// refused at registration (HeldTemplateProblem).
// TemplateField is the key a file resource names its template file by.
const TemplateField = "template"
// TemplatesDir is the module's directory its template files are kept in, and the directory its templates archive
// is built from.
const TemplatesDir = "files"
// Templated is whether any of the manifest's resources names a template.
func (m Manifest) Templated() bool {
for _, r := range m.Resources {
if _, has := r[TemplateField]; has {
return true
}
}
return false
}
// TemplatesArtifact is the name of the archive artifact the module packs its templates into — the one built from
// files/ — or "" when it declares none, or more than one (both refused by templateProblems for a module naming a
// template).
func TemplatesArtifact(m Manifest) string {
found := templatesArtifacts(m)
if len(found) != 1 {
return ""
}
return found[0]
}
func templatesArtifacts(m Manifest) []string {
if m.Build == nil {
return nil
}
var found []string
for _, a := range m.Build.Artifacts {
if a.Kind == ArtifactArchive && a.URL == "" && cleanFrom(a.From) == TemplatesDir {
found = append(found, a.Name)
}
}
return found
}
// cleanFrom is an artifact's directory as one relative path: "./files/" is "files".
func cleanFrom(from string) string {
c := path.Clean(strings.TrimSpace(from))
return strings.TrimPrefix(c, "./")
}
// templateOf is the template a resource names, whether it names one, and what is wrong with it ("" when nothing).
func templateOf(module string, r map[string]any) (string, bool, string) {
raw, has := r[TemplateField]
if !has {
return "", false, ""
}
id := fmt.Sprint(r["id"])
if t := fmt.Sprint(r["type"]); t != "file" {
return "", true, fmt.Sprintf("%s: %s is a %s and names a template; only a file takes its text from one "+
"(novox/hq ADR 0349 §2)", module, id, t)
}
name, ok := raw.(string)
if !ok || strings.TrimSpace(name) == "" {
return "", true, fmt.Sprintf("%s: %s names a template that is not a path: %v; a template is %q",
module, id, raw, TemplatesDir+"/<name>")
}
if _, both := r["content"]; both {
return name, true, fmt.Sprintf("%s: %s names the template %q and says its content too; a file's text is "+
"in one place, the template or the content, never both (novox/hq ADR 0349 §2)", module, id, name)
}
if strings.HasPrefix(name, "/") || path.Clean(name) != name || !strings.HasPrefix(name, TemplatesDir+"/") ||
strings.Contains(name, "\\") {
return name, true, fmt.Sprintf("%s: %s names the template %q, which is not a file under %s/ of its own "+
"repository: a template is %q, a plain relative path, and the build packs only %s/",
module, id, name, TemplatesDir, TemplatesDir+"/<name>", TemplatesDir)
}
return name, true, ""
}
// templateProblems is what is wrong with the manifest's templates, as written: a template on something not a file,
// beside content, outside files/, or a module naming templates whose build packs no archive from files/. Whether
// each template exists is judged where a tree is (ReadTemplates) or the archive's text is (Resolve).
func templateProblems(m Manifest) []string {
var problems []string
templated := false
for _, r := range m.Resources {
_, has, wrong := templateOf(m.Module, r)
templated = templated || has
if wrong != "" {
problems = append(problems, wrong)
}
}
if !templated {
return problems
}
switch found := templatesArtifacts(m); len(found) {
case 0:
problems = append(problems, fmt.Sprintf("%s names templates and its build packs no archive from %s/: "+
"declare an artifact {\"name\": \"files\", \"kind\": \"archive\", \"from\": %q} (novox/hq ADR 0349 §2)",
m.Module, TemplatesDir, TemplatesDir))
case 1:
default:
sort.Strings(found)
problems = append(problems, fmt.Sprintf("%s packs %s/ into %s; its templates are read from one archive",
m.Module, TemplatesDir, strings.Join(found, " and ")))
}
return problems
}
// fillTemplates is resources with each template's text as its content and the template key dropped: text says the
// text of a template by its path ("files/<name>"), and whether it has one. A template it has no text for, or text
// that is not UTF-8, is refused by name. Every resource is copied; the manifest's own are left as they were.
func fillTemplates(module string, resources []map[string]any, text func(string) (string, bool),
where string) ([]map[string]any, error) {
out := make([]map[string]any, 0, len(resources))
var problems []string
for _, r := range resources {
name, has, wrong := templateOf(module, r)
if !has {
out = append(out, r)
continue
}
if wrong != "" {
problems = append(problems, wrong)
continue
}
body, ok := text(name)
if !ok {
problems = append(problems, fmt.Sprintf("%s: %v names the template %q, and %s holds no such file",
module, r["id"], name, where))
continue
}
if !utf8.ValidString(body) {
// A manifest is JSON: text that is not UTF-8 would be changed on the way, silently.
problems = append(problems, fmt.Sprintf("%s: %v names the template %q, which is not UTF-8 text; a "+
"file's text is carried as JSON and would not reach the machine as written", module, r["id"], name))
continue
}
filled := make(map[string]any, len(r))
for k, v := range r {
filled[k] = v
}
delete(filled, TemplateField)
filled["content"] = body
out = append(out, filled)
}
if len(problems) > 0 {
return nil, errors.New(strings.Join(problems, "; "))
}
return out, nil
}
// ReadTemplates is the text of each template the manifest names, read from the module's directory in a repository
// tree, by template path. Refused by name: a template the directory lacks, and anything under files/ the build's
// packing refuses — files/ itself, or any directory or file under it, that is a symlink or not a plain file — so
// nothing judged here can be read from outside the tree, and no tree passes here that the build would refuse. Each
// template is read through an os.Root of the module's directory besides, which no path element can leave.
func ReadTemplates(m Manifest, dir string) (map[string]string, error) {
problems := templatesTreeProblems(m.Module, dir)
root, err := os.OpenRoot(dir)
if err != nil {
return nil, fmt.Errorf("%s: the module's directory cannot be opened to read its templates: %w", m.Module, err)
}
defer root.Close()
texts := map[string]string{}
for _, r := range m.Resources {
name, has, wrong := templateOf(m.Module, r)
if !has {
continue
}
if wrong != "" {
problems = append(problems, wrong)
continue
}
if _, read := texts[name]; read {
continue
}
info, err := root.Lstat(name)
switch {
case errors.Is(err, fs.ErrNotExist):
problems = append(problems, fmt.Sprintf("%s: %v names the template %q, and the module's directory "+
"has no such file", m.Module, r["id"], name))
continue
case err != nil:
problems = append(problems, fmt.Sprintf("%s: %v names the template %q, which cannot be read: %v",
m.Module, r["id"], name, err))
continue
case !info.Mode().IsRegular():
problems = append(problems, fmt.Sprintf("%s: %v names the template %q, which is not a plain file; "+
"the build packs only files", m.Module, r["id"], name))
continue
}
body, err := root.ReadFile(name)
if err != nil {
problems = append(problems, fmt.Sprintf("%s: %v names the template %q, which cannot be read: %v",
m.Module, r["id"], name, err))
continue
}
texts[name] = string(body)
}
if len(problems) > 0 {
return nil, errors.New(strings.Join(problems, "; "))
}
return texts, nil
}
// templatesTreeProblems is everything under the module's files/ that the build's packing refuses: files/ itself not
// a plain directory, and any entry under it that is neither a directory nor a plain file — a symlink, to a file or a
// directory, a device, a pipe. Each said by its path. A files/ that does not exist is no problem here: each template
// it would have held is refused as missing.
func templatesTreeProblems(module, dir string) []string {
top := filepath.Join(dir, TemplatesDir)
info, err := os.Lstat(top)
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
return []string{fmt.Sprintf("%s: %s/ cannot be read: %v", module, TemplatesDir, err)}
}
if !info.IsDir() {
return []string{fmt.Sprintf("%s: %s/ is %s, not a directory of the module's own; its templates are read "+
"only from inside the repository, and the build packs only directories and files", module, TemplatesDir,
kindOf(info.Mode()))}
}
var problems []string
err = filepath.WalkDir(top, func(at string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || d.Type().IsRegular() {
return nil
}
rel, _ := filepath.Rel(dir, at)
problems = append(problems, fmt.Sprintf("%s: %s is %s; the build packs only directories and files, so "+
"%s/ holds nothing else", module, filepath.ToSlash(rel), kindOf(d.Type()), TemplatesDir))
return nil
})
if err != nil {
problems = append(problems, fmt.Sprintf("%s: %s/ cannot be read: %v", module, TemplatesDir, err))
}
return problems
}
// kindOf says what a file that is not a plain file is.
func kindOf(mode fs.FileMode) string {
switch {
case mode&fs.ModeSymlink != 0:
return "a symlink"
case mode&fs.ModeDevice != 0:
return "a device"
case mode&fs.ModeNamedPipe != 0:
return "a named pipe"
case mode&fs.ModeSocket != 0:
return "a socket"
case mode.IsDir():
return "a directory"
}
return "not a plain file"
}
// WithTemplatesFrom is the manifest with each template's text read from the module's directory in a repository
// tree as its content: what `module check`, the merge gate and `module add` judge, where nothing was built. Read
// again strictly afterwards, so every rule about a file's text judges the template's text exactly as it judges
// inline content. A manifest naming no template is returned as it is.
func WithTemplatesFrom(m Manifest, dir string) (Manifest, error) {
if !m.Templated() {
return m, nil
}
texts, err := ReadTemplates(m, dir)
if err != nil {
return Manifest{}, err
}
filled, err := fillTemplates(m.Module, m.Resources, func(name string) (string, bool) {
t, ok := texts[name]
return t, ok
}, "the module's directory")
if err != nil {
return Manifest{}, err
}
out := m
out.Resources = filled
raw, err := json.Marshal(out)
if err != nil {
return Manifest{}, err
}
return ParseManifest(raw)
}
// HeldTemplateProblem is why a manifest cannot be held by the mesh because it still names a template, or "": the
// manifest the mesh holds carries every file's text, which a build reads back from its templates archive. One
// naming a template was resolved by a builder that does not read templates, and would compose files with no text.
func HeldTemplateProblem(m Manifest) string {
var named []string
for _, r := range m.Resources {
if t, has := r[TemplateField]; has {
named = append(named, fmt.Sprintf("%v (%v)", r["id"], t))
}
}
if len(named) == 0 {
return ""
}
return fmt.Sprintf("%s still names the template(s) of %s: the manifest the mesh holds carries each file's "+
"text, read back from the build's templates archive — this one was resolved by a builder that does not "+
"read templates, or not built at all (novox/hq ADR 0349 §2)", m.Module, strings.Join(named, ", "))
}
+323
View File
@@ -0,0 +1,323 @@
package catalogue
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// A file resource may name a template file in its module's repository (novox/hq ADR 0349 §2, design 57 package
// T1, issue 654): the build packs files/ into an archive, and resolving the build reads each template's text back
// into content, so the mesh holds, composes and sends exactly what it would for the same text inline.
// The two forms of one module: its text inline, and its text in files/.
const (
inlineTwin = `{"module":"tidy","version":"1",
"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]},
"resources":[
{"id":"conf","type":"directory","path":"${dir:conf}"},
{"id":"main","type":"file","path":"${dir:conf}/tidy.conf","mode":"0644","content":"` + mainText + `"},
{"id":"extra","type":"file","path":"${dir:conf}/extra.conf","content":"` + extraText + `"}]}`
templatedTwin = `{"module":"tidy","version":"1",
"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]},
"resources":[
{"id":"conf","type":"directory","path":"${dir:conf}"},
{"id":"main","type":"file","path":"${dir:conf}/tidy.conf","mode":"0644","template":"files/tidy.conf"},
{"id":"extra","type":"file","path":"${dir:conf}/extra.conf","template":"files/deeper/extra.conf"}]}`
mainText = `# tidy\nroot = ${dir:conf}\nquoted = \"yes\" é\n`
extraText = `one\ntwo\n`
)
// templateTexts is what a build reads back from the packed files/ archive: by path inside it.
func templateTexts(t *testing.T) map[string]string {
t.Helper()
var m struct{ Main, Extra string }
if err := json.Unmarshal([]byte(`{"Main":"`+mainText+`","Extra":"`+extraText+`"}`), &m); err != nil {
t.Fatal(err)
}
return map[string]string{"tidy.conf": m.Main, "deeper/extra.conf": m.Extra, "unused.conf": "not named"}
}
func builtFiles(texts map[string]string) []Built {
return []Built{{Name: "files", Kind: ArtifactArchive,
Reference: ArtifactStoreScheme + "tidy/files/blobs/sha256:" + strings.Repeat("c", 64),
Digest: "sha256:" + strings.Repeat("c", 64), Texts: texts}}
}
// held is a manifest as registration holds it: resolved with what was built, then read again strictly.
func held(t *testing.T, raw string, built []Built) Manifest {
t.Helper()
m, err := ParseManifest([]byte(raw))
if err != nil {
t.Fatal(err)
}
resolved, err := m.Resolve(built)
if err != nil {
t.Fatal(err)
}
again, err := json.Marshal(resolved)
if err != nil {
t.Fatal(err)
}
out, err := ParseManifest(again)
if err != nil {
t.Fatal(err)
}
if problem := HeldTemplateProblem(out); problem != "" {
t.Fatal(problem)
}
return out
}
func composed(t *testing.T, m Manifest) string {
t.Helper()
got, err := Resolve(shelf(m), []string{m.Module}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := json.Marshal(mustDeclare(t, got))
if err != nil {
t.Fatal(err)
}
return string(out)
}
func TestATemplatedModuleComposesByteIdenticallyToItsInlineTwin(t *testing.T) {
inline := composed(t, held(t, inlineTwin, builtFiles(nil)))
templated := composed(t, held(t, templatedTwin, builtFiles(templateTexts(t))))
if inline != templated {
t.Fatalf("the two forms compose differently:\ninline: %s\ntemplated: %s", inline, templated)
}
if !strings.Contains(templated, "quoted = \\\"yes\\\"") || strings.Contains(templated, `"template"`) {
t.Fatalf("the composition does not carry the template's text as content: %s", templated)
}
}
// What `module check`, the merge gate and `module add` judge: the text read from the tree beside the manifest.
func TestATemplatedModuleReadFromItsTreeComposesByteIdenticallyToItsInlineTwin(t *testing.T) {
dir := t.TempDir()
for name, text := range templateTexts(t) {
full := filepath.Join(dir, "files", filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(text), 0o644); err != nil {
t.Fatal(err)
}
}
m, err := ParseManifest([]byte(templatedTwin))
if err != nil {
t.Fatal(err)
}
filled, err := WithTemplatesFrom(m, dir)
if err != nil {
t.Fatal(err)
}
if filled.Templated() {
t.Fatal("a manifest read from its tree still names a template")
}
resolved, err := filled.Resolve(builtFiles(nil))
if err != nil {
t.Fatal(err)
}
if got, want := composed(t, resolved), composed(t, held(t, inlineTwin, builtFiles(nil))); got != want {
t.Fatalf("read from the tree, the module composes differently:\ninline: %s\ntemplated: %s", want, got)
}
}
func TestAFileNamingBothATemplateAndContentIsRefused(t *testing.T) {
raw := strings.Replace(templatedTwin, `"template":"files/tidy.conf"`,
`"template":"files/tidy.conf","content":"x"`, 1)
_, err := ParseManifest([]byte(raw))
if err == nil || !strings.Contains(err.Error(), "main") || !strings.Contains(err.Error(), "never both") {
t.Fatalf("a file with both a template and content was not refused by name: %v", err)
}
}
func TestATemplateOutsideFilesIsRefusedByName(t *testing.T) {
for _, bad := range []string{"../tidy.conf", "/etc/tidy.conf", "files", "files/", "files/../tidy.conf",
"files/./tidy.conf", "other/tidy.conf", "tidy.conf", "files//tidy.conf", `files\\tidy.conf`} {
raw := strings.Replace(templatedTwin, `"files/tidy.conf"`, `"`+bad+`"`, 1)
_, err := ParseManifest([]byte(raw))
if err == nil || !strings.Contains(err.Error(), "main") || !strings.Contains(err.Error(), "not a file under files/") {
t.Errorf("the template %q was not refused by name: %v", bad, err)
}
}
}
func TestATemplateOnAResourceThatIsNotAFileIsRefused(t *testing.T) {
raw := strings.Replace(templatedTwin, `{"id":"conf","type":"directory","path":"${dir:conf}"}`,
`{"id":"conf","type":"directory","path":"${dir:conf}","template":"files/tidy.conf"}`, 1)
_, err := ParseManifest([]byte(raw))
if err == nil || !strings.Contains(err.Error(), "conf is a directory and names a template") {
t.Fatalf("a directory naming a template was not refused: %v", err)
}
}
func TestATemplatedModuleWhoseBuildPacksNoFilesIsRefused(t *testing.T) {
for _, build := range []string{
``,
`"build":{"artifacts":[{"name":"look","kind":"archive","from":"theme"}]},`,
`"build":{"artifacts":[{"name":"a","kind":"archive","from":"files"},{"name":"b","kind":"archive","from":"./files/"}]},`,
} {
raw := strings.Replace(templatedTwin,
`"build":{"artifacts":[{"name":"files","kind":"archive","from":"files"}]},`, build, 1)
_, err := ParseManifest([]byte(raw))
if err == nil || !strings.Contains(err.Error(), "files/") {
t.Errorf("a templated module with the build %q was not refused: %v", build, err)
}
}
}
func TestAMissingTemplateIsRefusedByName(t *testing.T) {
m, err := ParseManifest([]byte(templatedTwin))
if err != nil {
t.Fatal(err)
}
// In the tree beside the manifest.
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "files"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "files", "tidy.conf"), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
_, err = WithTemplatesFrom(m, dir)
if err == nil || !strings.Contains(err.Error(), "extra") || !strings.Contains(err.Error(), "files/deeper/extra.conf") {
t.Fatalf("a template missing from the tree was not refused by name: %v", err)
}
// And in what the build read back from its archive.
texts := templateTexts(t)
delete(texts, "deeper/extra.conf")
_, err = m.Resolve(builtFiles(texts))
if err == nil || !strings.Contains(err.Error(), "extra") || !strings.Contains(err.Error(), "files/deeper/extra.conf") {
t.Fatalf("a template missing from the archive was not refused by name: %v", err)
}
}
func TestATemplateThatIsASymlinkIsRefused(t *testing.T) {
m, err := ParseManifest([]byte(templatedTwin))
if err != nil {
t.Fatal(err)
}
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "files", "deeper"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "elsewhere"), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "files", "tidy.conf"), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.Symlink(filepath.Join(dir, "elsewhere"), filepath.Join(dir, "files", "deeper", "extra.conf")); err != nil {
t.Fatal(err)
}
if _, err := WithTemplatesFrom(m, dir); err == nil || !strings.Contains(err.Error(), "not a plain file") {
t.Fatalf("a template that is a symlink was not refused: %v", err)
}
}
func TestAResolveWithoutTheArchivesTextIsRefused(t *testing.T) {
m, err := ParseManifest([]byte(templatedTwin))
if err != nil {
t.Fatal(err)
}
if _, err := m.Resolve(builtFiles(nil)); err == nil || !strings.Contains(err.Error(), "files") {
t.Fatalf("a templated module resolved with no text read back from its archive: %v", err)
}
}
func TestATemplateThatIsNotUTF8IsRefused(t *testing.T) {
m, err := ParseManifest([]byte(templatedTwin))
if err != nil {
t.Fatal(err)
}
texts := templateTexts(t)
texts["tidy.conf"] = "\xff\xfe"
if _, err := m.Resolve(builtFiles(texts)); err == nil || !strings.Contains(err.Error(), "UTF-8") {
t.Fatalf("a template that is not UTF-8 was resolved: %v", err)
}
}
// What registration refuses: a manifest to be held that still names a template — one resolved by a builder that
// predates templates would otherwise compose files with no text.
func TestAManifestStillNamingATemplateIsNotHeld(t *testing.T) {
m, err := ParseManifest([]byte(templatedTwin))
if err != nil {
t.Fatal(err)
}
if problem := HeldTemplateProblem(m); !strings.Contains(problem, "main") || !strings.Contains(problem, "files/tidy.conf") {
t.Fatalf("a manifest naming a template would be held: %q", problem)
}
if problem := HeldTemplateProblem(held(t, inlineTwin, builtFiles(nil))); problem != "" {
t.Fatalf("an inline manifest is refused: %s", problem)
}
}
// A symlinked files/, or a symlinked directory under it, would read a file outside the tree: refused by name,
// as is any symlink or special file under files/ that no template names, which the build's packing refuses.
func TestASymlinkAnywhereUnderFilesIsRefusedByName(t *testing.T) {
m, err := ParseManifest([]byte(templatedTwin))
if err != nil {
t.Fatal(err)
}
outside := t.TempDir()
if err := os.MkdirAll(filepath.Join(outside, "deeper"), 0o755); err != nil {
t.Fatal(err)
}
for _, name := range []string{"tidy.conf", "deeper/extra.conf", "secret"} {
if err := os.WriteFile(filepath.Join(outside, filepath.FromSlash(name)), []byte("outside"), 0o644); err != nil {
t.Fatal(err)
}
}
write := func(t *testing.T, dir, name string) {
t.Helper()
full := filepath.Join(dir, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte("inside"), 0o644); err != nil {
t.Fatal(err)
}
}
link := func(t *testing.T, target, at string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(at), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink(target, at); err != nil {
t.Fatal(err)
}
}
for _, c := range []struct {
name string
make func(t *testing.T, dir string)
names string
}{
{"files/ itself", func(t *testing.T, dir string) {
link(t, outside, filepath.Join(dir, "files"))
}, "files/ is a symlink"},
{"a directory under files/", func(t *testing.T, dir string) {
write(t, dir, "files/tidy.conf")
link(t, filepath.Join(outside, "deeper"), filepath.Join(dir, "files", "deeper"))
}, "files/deeper is a symlink"},
{"a file no template names", func(t *testing.T, dir string) {
write(t, dir, "files/tidy.conf")
write(t, dir, "files/deeper/extra.conf")
link(t, filepath.Join(outside, "secret"), filepath.Join(dir, "files", "unnamed"))
}, "files/unnamed is a symlink"},
} {
dir := t.TempDir()
c.make(t, dir)
_, err := WithTemplatesFrom(m, dir)
if err == nil || !strings.Contains(err.Error(), c.names) {
t.Errorf("a symlinked %s was not refused by name: %v", c.name, err)
}
if err != nil && strings.Contains(err.Error(), "outside") {
t.Errorf("a symlinked %s was read: %v", c.name, err)
}
}
}
+7 -7
View File
@@ -13,22 +13,22 @@ func TestAPlacementOrAnAccessAtTheMachinesOwnIsRefused(t *testing.T) {
for _, path := range []string{"/", "/etc", "/etc/", "/etc/sudoers.d", "/usr/bin", "/root", "/var", "/var/lib",
"/home", "/home/", "/run", "/run/user/1000", "/var/lib/mesh/x", "/var/lib/mesh-host", "/var/lib/mesh-host/identity",
"/srv/../etc", "//etc", "/proc/1", "/sys", "/dev", "/boot/efi", "/bin", "/sbin", "/lib", "/lib64"} {
layers := []Layer{{From: "laptop", Values: map[string]any{PlacesSetting: map[string]any{"data": path}}}}
layers := []Layer{LayerOf("laptop", map[string]any{PlacesSetting: map[string]any{"data": path}})}
if _, err := Places(m, layers); err == nil || !strings.Contains(err.Error(), "issue 339") {
t.Errorf("a place at %s: %v", path, err)
}
layers = []Layer{{From: "laptop", Values: map[string]any{AccessesSetting: map[string]any{"media": path}}}}
layers = []Layer{LayerOf("laptop", map[string]any{AccessesSetting: map[string]any{"media": path}})}
if _, err := AccessPlaces(m, layers); err == nil || !strings.Contains(err.Error(), "issue 339") {
t.Errorf("an access at %s: %v", path, err)
}
}
for _, path := range []string{"/srv/notes", "/mnt/plex/data", "/storage/media", "/services/media/movies",
"/var/lib/notes/data", "/home/restic/repo", "/var/lib/mesh-store"} {
layers := []Layer{{From: "laptop", Values: map[string]any{PlacesSetting: map[string]any{"data": path}}}}
layers := []Layer{LayerOf("laptop", map[string]any{PlacesSetting: map[string]any{"data": path}})}
if got, err := Places(m, layers); err != nil || got["data"].Path != path {
t.Errorf("a place at %s: %v %v", path, got, err)
}
layers = []Layer{{From: "laptop", Values: map[string]any{AccessesSetting: map[string]any{"media": path}}}}
layers = []Layer{LayerOf("laptop", map[string]any{AccessesSetting: map[string]any{"media": path}})}
if got, err := AccessPlaces(m, layers); err != nil || got["media"] != path {
t.Errorf("an access at %s: %v %v", path, got, err)
}
@@ -111,17 +111,17 @@ func TestTheRuntimesDataAndAnyHomesSSHAreTheMachinesOwn(t *testing.T) {
for _, path := range []string{"/var/spool", "/var/spool/cron", "/var/lib/docker", "/var/lib/docker/volumes",
"/var/lib/containers/storage", "/var/lib/containerd", "/var/lib/containerd/io.containerd.snapshotter.v1", "/opt", "/opt/app", "/home/alice/.ssh", "/home/alice/.ssh/keys",
"/srv/backup/.ssh", "/root/.ssh"} {
layers := []Layer{{From: "laptop", Values: map[string]any{PlacesSetting: map[string]any{"data": path}}}}
layers := []Layer{LayerOf("laptop", map[string]any{PlacesSetting: map[string]any{"data": path}})}
if _, err := Places(m, layers); err == nil || !strings.Contains(err.Error(), "issue 339") {
t.Errorf("a place at %s: %v", path, err)
}
layers = []Layer{{From: "laptop", Values: map[string]any{AccessesSetting: map[string]any{"media": path}}}}
layers = []Layer{LayerOf("laptop", map[string]any{AccessesSetting: map[string]any{"media": path}})}
if _, err := AccessPlaces(m, layers); err == nil || !strings.Contains(err.Error(), "issue 339") {
t.Errorf("an access at %s: %v", path, err)
}
}
for _, path := range []string{"/var/lib/dockerish", "/home/alice/ssh", "/home/alice/.sshd-notes", "/storage/media"} {
layers := []Layer{{From: "laptop", Values: map[string]any{PlacesSetting: map[string]any{"data": path}}}}
layers := []Layer{LayerOf("laptop", map[string]any{PlacesSetting: map[string]any{"data": path}})}
if _, err := Places(m, layers); err != nil {
t.Errorf("a place at %s: %v", path, err)
}
@@ -0,0 +1,35 @@
{
"grafana.bound-influxdb-api": {
"content": "{\n \"as\": \"mesh_workstation_grafana\",\n \"at\": \"workstation.internal\",\n \"binding\": 1,\n \"from\": \"workstation\",\n \"generated\": \"by the mesh — do not edit; replaced whenever this changes. The credential is not here: it is sealed, in the file this module's manifest names under `secrets`\",\n \"provision\": \"influxdb-api\",\n \"serves\": {\n \"bucket\": \"made-up-bucket\",\n \"org\": \"made-up-org\",\n \"port\": 8086,\n \"scheme\": \"http\"\n }\n}\n",
"id": "grafana.bound-influxdb-api",
"mode": "0644",
"path": "/var/lib/grafana/influxdb.json",
"type": "file"
},
"grafana.influxdb-datasource": {
"content": "apiVersion: 1\n# Written by the mesh from grafana's influxdb-api binding, only while it is bound (novox/hq ADR 0330):\n# without influxdb, grafana runs with no mesh data source. grafana reads it at start. Its own name and\n# uid, so a data source somebody made in the UI is never overwritten, and read-only in the UI because\n# the mesh resets it. The password is read from the file the mesh delivers, never written here.\ndatasources:\n - name: InfluxDB (mesh)\n uid: mesh-influxdb-api\n type: influxdb\n access: proxy\n url: http://workstation.internal:8086\n user: mesh_workstation_grafana\n isDefault: false\n editable: false\n jsonData:\n dbName: made-up-bucket\n httpMode: POST\n secureJsonData:\n password: $__file{/run/secrets/influxdb/influxdb-api}\n",
"id": "grafana.influxdb-datasource",
"mode": "0644",
"path": "/var/lib/grafana/influxdb-datasources/mesh-influxdb.yaml",
"type": "file"
},
"grafana.influxdb-datasources": {
"id": "grafana.influxdb-datasources",
"mode": "0755",
"owner": "472:0",
"path": "/var/lib/grafana/influxdb-datasources",
"type": "directory"
},
"grafana.mesh-state": {
"id": "grafana.mesh-state",
"mode": "0700",
"path": "/var/lib/mesh/grafana",
"type": "directory"
},
"grafana.state": {
"id": "grafana.state",
"mode": "0700",
"path": "/var/lib/grafana",
"type": "directory"
}
}
@@ -0,0 +1,35 @@
{
"grafana.bound-influxdb-api": {
"content": "{\n \"as\": \"mesh_workstation_grafana\",\n \"at\": \"workstation.internal\",\n \"binding\": 1,\n \"from\": \"workstation\",\n \"generated\": \"by the mesh — do not edit; replaced whenever this changes. The credential is not here: it is sealed, in the file this module's manifest names under `secrets`\",\n \"provision\": \"influxdb-api\",\n \"serves\": {\n \"bucket\": \"default\",\n \"org\": \"mesh\",\n \"port\": 8086,\n \"scheme\": \"http\"\n }\n}\n",
"id": "grafana.bound-influxdb-api",
"mode": "0644",
"path": "/var/lib/grafana/influxdb.json",
"type": "file"
},
"grafana.influxdb-datasource": {
"content": "apiVersion: 1\n# Written by the mesh from grafana's influxdb-api binding, only while it is bound (novox/hq ADR 0330):\n# without influxdb, grafana runs with no mesh data source. grafana reads it at start. Its own name and\n# uid, so a data source somebody made in the UI is never overwritten, and read-only in the UI because\n# the mesh resets it. The password is read from the file the mesh delivers, never written here.\ndatasources:\n - name: InfluxDB (mesh)\n uid: mesh-influxdb-api\n type: influxdb\n access: proxy\n url: http://workstation.internal:8086\n user: mesh_workstation_grafana\n isDefault: false\n editable: false\n jsonData:\n dbName: default\n httpMode: POST\n secureJsonData:\n password: $__file{/run/secrets/influxdb/influxdb-api}\n",
"id": "grafana.influxdb-datasource",
"mode": "0644",
"path": "/var/lib/grafana/influxdb-datasources/mesh-influxdb.yaml",
"type": "file"
},
"grafana.influxdb-datasources": {
"id": "grafana.influxdb-datasources",
"mode": "0755",
"owner": "472:0",
"path": "/var/lib/grafana/influxdb-datasources",
"type": "directory"
},
"grafana.mesh-state": {
"id": "grafana.mesh-state",
"mode": "0700",
"path": "/var/lib/mesh/grafana",
"type": "directory"
},
"grafana.state": {
"id": "grafana.state",
"mode": "0700",
"path": "/var/lib/grafana",
"type": "directory"
}
}
+31
View File
@@ -23,3 +23,34 @@ func TestTheVaultsSeatDeliversSecret(t *testing.T) {
t.Fatal("a provider of secret that does not claim the seat was allowed to hold it")
}
}
// The vault answers `get` on request and reply (novox/hq ADR 0349 §11, issue 674): the seat promises the
// verb, so a holder serving it is accepted — and, optional while the vault that runs is the TypeScript one,
// so is a holder that does not yet.
func TestTheVaultsSeatPromisesGet(t *testing.T) {
seat, _ := SeatNamed("mesh-vault")
var get *Verb
for i := range seat.Serves {
if seat.Serves[i].Name == "get" {
get = &seat.Serves[i]
}
}
if get == nil {
t.Fatal("the vault's seat does not promise get")
}
for _, arg := range []string{"name", "public_key"} {
if props, _ := get.Input["properties"].(map[string]any); props[arg] == nil {
t.Errorf("get takes no %s", arg)
}
}
serving := Manifest{Module: "mesh-vault", Provides: []Offer{{Name: "secret", Scope: ScopeMesh}},
Claims: []Claim{{Name: "mesh-vault", Scope: ScopeMesh, Serves: []string{"get"}}}}
if err := CanHold(serving, seat); err != nil {
t.Fatalf("the vault serving get was refused: %v", err)
}
notYet := Manifest{Module: "mesh-vault", Provides: []Offer{{Name: "secret", Scope: ScopeMesh}},
Claims: []Claim{{Name: "mesh-vault", Scope: ScopeMesh}}}
if err := CanHold(notYet, seat); err != nil {
t.Fatalf("the vault that does not serve get yet was refused: %v", err)
}
}
+83 -19
View File
@@ -117,6 +117,11 @@ func (i *Inventory) RegisterModule(ctx context.Context, m catalogue.Manifest, fr
}
}
}
// **The manifest the mesh holds carries each file's text** (novox/hq ADR 0349 §2): one still naming a template
// would compose a file with no text, so it is refused by name — a build by a builder that predates templates.
if why := catalogue.HeldTemplateProblem(m); why != "" {
return fmt.Errorf("%s is not registered: %s", m.Module, why)
}
raw, err := json.Marshal(m)
if err != nil {
return err
@@ -839,7 +844,18 @@ func (i *Inventory) KeepSettings(ctx context.Context, nodeName, module string, v
}
func (i *Inventory) setSettings(ctx context.Context, nodeName, module string, values map[string]any, judge bool, setBy string) error {
raw, err := json.Marshal(values)
// **Placement is not config** (novox/hq ADR 0348 clause 11, issue 651): the layer is said whole, as
// it always was, and its placement keys are kept beside the assignment, never in the settings row.
// Both are written in one transaction, and the layer a set replaces is the whole of both.
config, placement := catalogue.SplitPlacement(values)
if config == nil {
config = map[string]any{}
}
raw, err := json.Marshal(config)
if err != nil {
return err
}
placed, err := placementJSON(placement)
if err != nil {
return err
}
@@ -881,6 +897,9 @@ func (i *Inventory) setSettings(ctx context.Context, nodeName, module string, va
do update set values = excluded.values, set_at = now(), set_by = excluded.set_by`, module, raw, nullable(setBy)); err != nil {
return wrapModule(err, module)
}
if err := keepPlacement(ctx, tx, nil, module, placed, setBy); err != nil {
return err
}
return tx.Commit(ctx)
}
node, err := i.NodeByName(ctx, nodeName)
@@ -912,6 +931,9 @@ func (i *Inventory) setSettings(ctx context.Context, nodeName, module string, va
if err != nil {
return wrapModule(err, module)
}
if err := keepPlacement(ctx, tx, &node.ID, module, placed, setBy); err != nil {
return err
}
// A given port replaces what the mesh assigned for that port: the assignment is given back,
// so the number is free for the next module rather than held for ever in the name of a port
// that now lives elsewhere.
@@ -942,21 +964,15 @@ func (i *Inventory) judgeSettings(ctx context.Context, nodeName, module string,
return err
}
where, from, adopted = nodeName, nodeName, node.Adopted
var meshWide []byte
err = i.store.Pool().QueryRow(ctx,
`select values from settings where module = $1 and node is null`, module).Scan(&meshWide)
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
under, has, err := i.Layer(ctx, "", module)
if err != nil {
return err
}
if len(meshWide) > 0 {
var under map[string]any
if err := json.Unmarshal(meshWide, &under); err != nil {
return err
}
layers = append(layers, catalogue.Layer{From: catalogue.MeshWideLayer, Values: under})
if has {
layers = append(layers, catalogue.LayerOf(catalogue.MeshWideLayer, under))
}
}
layers = append(layers, catalogue.Layer{From: from, Values: values})
layers = append(layers, catalogue.LayerOf(from, values))
if err := catalogue.JudgeSettings(m, layers, adopted); err != nil {
return fmt.Errorf("refused: %s on %s cannot compose with the layer %q — %w", module, where, from, err)
}
@@ -1089,7 +1105,8 @@ func refuseGivenCollisions(ctx context.Context, tx pgx.Tx, nodeID any, node, mod
}
func wrapModule(err error, module string) error {
if err != nil && strings.Contains(err.Error(), "settings_module_fkey") {
if err != nil && (strings.Contains(err.Error(), "settings_module_fkey") ||
strings.Contains(err.Error(), "placement_module_fkey")) {
return fmt.Errorf("%w: %s", ErrNoSuchModule, module)
}
return err
@@ -1130,6 +1147,11 @@ func (i *Inventory) ClearSettingsBy(ctx context.Context, nodeName, module, clear
`delete from settings where module = $1 and node is not distinct from $2::uuid`, module, nodeID); err != nil {
return err
}
// And its placement, which is the same layer as a person says it (novox/hq ADR 0348 clause 11).
if _, err := tx.Exec(ctx,
`delete from placement where module = $1 and node is not distinct from $2::uuid`, module, nodeID); err != nil {
return err
}
return tx.Commit(ctx)
}
@@ -1142,10 +1164,13 @@ func (i *Inventory) SettingsFor(ctx context.Context, nodeName, module string) ([
if err != nil {
return nil, err
}
// Each layer's placement beside it, from the placement the controller holds (novox/hq ADR 0348
// clause 11): a layer's Values hold its config values only, its Placement where things are.
rows, err := i.store.Pool().Query(ctx,
`select node is null, values from settings
where module = $1 and (node is null or node = $2)
order by node is null desc`, module, node.ID)
`select s.node is null, s.values, p.values from settings s
left join placement p on p.module = s.module and p.node is not distinct from s.node
where s.module = $1 and (s.node is null or s.node = $2)
order by s.node is null desc`, module, node.ID)
if err != nil {
return nil, err
}
@@ -1154,23 +1179,62 @@ func (i *Inventory) SettingsFor(ctx context.Context, nodeName, module string) ([
var layers []catalogue.Layer
for rows.Next() {
var meshWide bool
var raw []byte
if err := rows.Scan(&meshWide, &raw); err != nil {
var raw, placed []byte
if err := rows.Scan(&meshWide, &raw, &placed); err != nil {
return nil, err
}
values := map[string]any{}
if err := json.Unmarshal(raw, &values); err != nil {
return nil, err
}
var placement map[string]any
if len(placed) > 0 {
if err := json.Unmarshal(placed, &placement); err != nil {
return nil, err
}
}
from := nodeName
if meshWide {
from = catalogue.MeshWideLayer
}
layers = append(layers, catalogue.Layer{From: from, Values: values})
layers = append(layers, catalogue.Layer{From: from, Values: values, Placement: placement})
}
return layers, rows.Err()
}
// placementJSON is a layer's placement as the placement table keeps it, or nil when it has none.
func placementJSON(placement map[string]any) ([]byte, error) {
if len(placement) == 0 {
return nil, nil
}
return json.Marshal(placement)
}
// keepPlacement writes one layer's placement beside its settings row, in the write's transaction: the
// placement the layer says, or none when it says none — a set replaces the layer whole, placement too.
func keepPlacement(ctx context.Context, tx pgx.Tx, nodeID *string, module string, placed []byte, setBy string) error {
if placed == nil {
_, err := tx.Exec(ctx,
`delete from placement where module = $1 and node is not distinct from $2::uuid`, module, nodeID)
return err
}
var err error
if nodeID == nil {
_, err = tx.Exec(ctx,
`insert into placement (node, module, values, set_by) values (null, $1, $2, $3)
on conflict (module) where node is null
do update set values = excluded.values, set_at = now(), set_by = excluded.set_by`,
module, placed, nullable(setBy))
} else {
_, err = tx.Exec(ctx,
`insert into placement (node, module, values, set_by) values ($1, $2, $3, $4)
on conflict (node, module) where node is not null
do update set values = excluded.values, set_at = now(), set_by = excluded.set_by`,
*nodeID, module, placed, nullable(setBy))
}
return wrapModule(err, module)
}
// PinProvision records which provider a machine gets a provision from: a module, and the node it
// runs on — both, always (novox/hq #258). A provision comes from a module, and the same module on
// two machines is two answers, so neither half alone says which.
+92
View File
@@ -2,6 +2,7 @@ package inventory
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
@@ -75,3 +76,94 @@ func (i *Inventory) RemoveNodeForTest(ctx context.Context, name string) (bool, e
}
return tag.RowsAffected() == 1, nil
}
// ForTestBefore is ForTest with the store migrated only below migration `number`, and a function that runs the
// rest: for a test of a migration that moves data, which seeds the store as the mesh holds it before the
// migration, migrates, and reads what the migration made of it (novox/hq issue 651).
func ForTestBefore(t *testing.T, number int) (*Inventory, func()) {
t.Helper()
admin := os.Getenv("MESH_TEST_POSTGRES")
if admin == "" {
t.Skip("no MESH_TEST_POSTGRES; run `make check` to raise one")
}
name := fmt.Sprintf("invb_%d_%s", time.Now().UnixNano()%1_000_000,
strings.ToLower(strings.NewReplacer("/", "", "-", "").Replace(t.Name())))
if len(name) > 60 {
name = name[:60]
}
all, err := Migrations()
if err != nil {
t.Fatal(err)
}
var before []store.Migration
for _, m := range all {
if m.Number < number {
before = append(before, m)
}
}
store.CreateMigratedForTest(t, admin, name, before)
busPasswordCostForTest.Store(int32(bcrypt.MinCost))
cut := strings.LastIndex(admin, "/")
t.Setenv(store.Variable(Name), admin[:cut]+"/"+name+"?sslmode=disable")
inv, err := Open(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
inv.Close()
c, err := pgx.Connect(context.Background(), admin)
if err != nil {
return
}
defer c.Close(context.Background())
_, _ = c.Exec(context.Background(), "drop database if exists "+name+" with (force)")
})
if err := inv.Ready(t.Context(), 20*time.Second); err != nil {
t.Fatal(err)
}
return inv, func() {
t.Helper()
if _, err := inv.store.Migrate(t.Context(), all); err != nil {
t.Fatal(err)
}
}
}
// ExecForTest runs one statement against a test's store, for a fixture no verb of the mesh writes: a row as an
// older migration left it.
func (i *Inventory) ExecForTest(t *testing.T, sql string, args ...any) {
t.Helper()
if _, err := i.store.Pool().Exec(t.Context(), sql, args...); err != nil {
t.Fatalf("%s: %v", sql, err)
}
}
// QueryForTest reads rows of a test's store, each row's columns as JSON text, for a test that checks what a
// migration left.
func (i *Inventory) QueryForTest(t *testing.T, sql string, args ...any) [][]string {
t.Helper()
rows, err := i.store.Pool().Query(t.Context(), sql, args...)
if err != nil {
t.Fatalf("%s: %v", sql, err)
}
defer rows.Close()
var out [][]string
for rows.Next() {
values, err := rows.Values()
if err != nil {
t.Fatal(err)
}
row := make([]string, len(values))
for k, v := range values {
row[k] = fmt.Sprint(v)
if b, err := json.Marshal(v); err == nil {
row[k] = string(b)
}
}
out = append(out, row)
}
if err := rows.Err(); err != nil {
t.Fatal(err)
}
return out
}
@@ -0,0 +1,46 @@
-- Placement is not config (novox/hq ADR 0348 clause 11, issue 651).
--
-- Where a module's directories are on a machine (`places`), where the operator's data it reads is
-- (`accesses`) and which found networks a taken container keeps (`networks`) were kept as keys of a
-- settings layer. They are facts of an assignment that the controller decides, never config values, and
-- the config values are leaving for the mesh-config module (ADR 0348 clause 9) while placement stays with
-- the controller. So they get a table of their own beside the assignment, and every settings layer's
-- placement keys move into it, unchanged.
--
-- Scoped exactly as the layer they came from: one machine's, or the whole mesh's (node null) for a
-- module's every assignment, which is how a mesh-wide `places` or `accesses` layer composes today. The
-- composer reads placement from here only; the `settings` verb still takes the three keys and routes
-- them here, so nothing a person types changes.
create table placement (
node uuid references node(id) on delete cascade,
module text not null references module(name) on delete cascade,
-- Only `places`, `accesses` and `networks`, as the settings layer held them.
values jsonb not null,
set_at timestamptz not null default now(),
set_by text
);
create unique index placement_for_the_mesh on placement (module) where node is null;
create unique index placement_for_a_node on placement (node, module) where node is not null;
-- Why a layer went into the history, beside how (`replaced_by`): said for a layer this migration moved.
alter table settings_history add column why text;
-- Each layer that held a placement key is kept whole in the history first, as a `settings history` reads it.
insert into settings_history (node, module, values, set_at, replaced_by, set_by, why)
select node, module, values, set_at, 'moved', set_by, 'moved to placement, ADR 0348 clause 11'
from settings
where values ?| array['places', 'accesses', 'networks'];
insert into placement (node, module, values, set_at, set_by)
select node, module,
(select jsonb_object_agg(e.key, e.value) from jsonb_each(values) as e
where e.key in ('places', 'accesses', 'networks')),
set_at, set_by
from settings
where values ?| array['places', 'accesses', 'networks'];
-- The layer keeps its other keys, when it was set and who set it; a layer that held only placement stays,
-- empty, so a layer that existed still exists.
update settings
set values = values - 'places' - 'accesses' - 'networks'
where values ?| array['places', 'accesses', 'networks'];
+222
View File
@@ -0,0 +1,222 @@
package inventory
import (
"encoding/json"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// placedLibrary is a module with each kind of placement: a directory to place, two accesses, a container that
// may keep a found network.
func placedLibrary() catalogue.Manifest {
return catalogue.Manifest{Module: "library", Version: "1",
Accesses: []catalogue.Access{{ID: "series"}, {ID: "downloads"}},
Resources: []map[string]any{
{"id": "config", "type": "directory", "mode": "0750"},
{"id": "server", "type": "container",
"image": "registry.invalid:5000/library@sha256:" + strings.Repeat("a", 64),
"volumes": []any{"${dir:config}:/config", "${access:series}:/series", "${access:downloads}:/downloads"}},
{"id": "runtime-config", "type": "file", "path": "${dir:config}/runtime.json", "mode": "0640",
"content": `{"log":"info"}`, "merge": "json"},
}}
}
// The layers of the issue (651): a mesh-wide layer with an access and a config value, a node's layer with a
// place, an access and a kept network beside a config value, a node's layer of placement only, and one with
// none.
var placementLayers = []struct {
node string
values string
}{
{"", `{"accesses": {"downloads": "/storage/downloads"}, "log": "warn"}`},
{"anchor", `{"places": {"config": {"path": "/var/lib/library/config", "owner": "1001:2000"}},
"accesses": {"series": "/storage/media/series"}, "networks": {"server": ["predecessor_default"]}, "workers": 4}`},
{"laptop", `{"accesses": {"series": "/home/media/series"}}`},
{"spare", `{"log": "debug"}`},
}
// **Placement moves out of the settings layers, and every layer reads as it did** (novox/hq ADR 0348 clause
// 11, issue 651). Migration 0097 moves each layer's places, accesses and kept networks beside it: no settings
// row holds a placement key after, every layer the settings verb and composition read is what it was, each
// moved layer is kept whole in the history with why, and a layer without placement is not touched.
func TestPlacementMovesOutOfTheSettingsLayers(t *testing.T) {
inv, migrate := ForTestBefore(t, 97)
ctx := t.Context()
if err := inv.RegisterModule(ctx, placedLibrary(), Source{}); err != nil {
t.Fatal(err)
}
for _, n := range []string{"anchor", "laptop", "spare"} {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAdopted(ctx, "anchor", true); err != nil {
t.Fatal(err)
}
// As the mesh holds them before the migration: placement among the settings.
for _, l := range placementLayers {
if l.node == "" {
inv.ExecForTest(t, `insert into settings (node, module, values, set_by) values (null, 'library', $1, 'the operator')`, l.values)
continue
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, 'library', $2 from node where name = $1`,
l.node, l.values)
}
setAt := inv.QueryForTest(t, `select coalesce(n.name, ''), s.set_at from settings s left join node n on n.id = s.node order by 1`)
migrate()
if held := inv.QueryForTest(t, `select module from settings where values ?| array['places', 'accesses', 'networks']`); len(held) != 0 {
t.Fatalf("a settings layer still holds a placement key: %v", held)
}
if after := inv.QueryForTest(t, `select coalesce(n.name, ''), s.set_at from settings s left join node n on n.id = s.node order by 1`); !reflect.DeepEqual(after, setAt) {
t.Fatalf("the layers or when they were set changed: %v, was %v", after, setAt)
}
for _, l := range placementLayers {
var want map[string]any
if err := json.Unmarshal([]byte(l.values), &want); err != nil {
t.Fatal(err)
}
got, has, err := inv.Layer(ctx, l.node, "library")
if err != nil || !has || !reflect.DeepEqual(roundTrip(t, got), want) {
t.Errorf("the layer on %q reads %v (%v, %v), was %v", l.node, got, has, err, want)
}
}
// Composition's view: the config values in Values, the placement in Placement, together the layer.
layers, err := inv.SettingsFor(ctx, "anchor", "library")
if err != nil || len(layers) != 2 {
t.Fatalf("anchor's layers: %v, %v", layers, err)
}
if layers[1].Values["workers"] == nil || layers[1].Values["places"] != nil || layers[1].Placement["places"] == nil ||
layers[1].Placement["networks"] == nil || layers[0].Placement["accesses"] == nil || layers[0].Values["log"] != "warn" {
t.Fatalf("anchor's layers split wrong: %+v", layers)
}
for i, l := range layers {
var want map[string]any
_ = json.Unmarshal([]byte(placementLayers[i].values), &want)
if !reflect.DeepEqual(roundTrip(t, l.Whole()), want) {
t.Errorf("anchor's layer %s is %v whole, was %v", l.From, l.Whole(), want)
}
}
// Judged as composition judges them, as before: anchor is adopted, so its kept network composes.
if err := catalogue.JudgeSettings(placedLibrary(), layers, true); err != nil {
t.Fatalf("the moved layers no longer compose: %v", err)
}
moved := inv.QueryForTest(t, `select coalesce(n.name, ''), h.replaced_by, h.why, h.set_by from settings_history h
left join node n on n.id = h.node order by 1`)
want := [][]string{
{`""`, `"moved"`, `"moved to placement, ADR 0348 clause 11"`, `"the operator"`},
{`"anchor"`, `"moved"`, `"moved to placement, ADR 0348 clause 11"`, `null`},
{`"laptop"`, `"moved"`, `"moved to placement, ADR 0348 clause 11"`, `null`},
}
if !reflect.DeepEqual(moved, want) {
t.Fatalf("the history of what moved: %v, want %v", moved, want)
}
past, err := inv.SettingsHistory(ctx, "anchor", "library")
if err != nil || len(past) != 1 || past[0].Why != "moved to placement, ADR 0348 clause 11" || past[0].Values["places"] == nil {
t.Fatalf("settings history on anchor: %+v, %v", past, err)
}
}
// **The settings verb's path still takes a placement key, and routes it** (issue 651): a set keeps the
// placement beside the layer and the rest in it, reads back whole, is refused as before when the placement
// cannot compose (nothing written), and a set or a clear replaces the placement with the layer.
func TestASettingRoutesPlacementBesideTheLayer(t *testing.T) {
inv := ForTest(t)
ctx := t.Context()
if err := inv.RegisterModule(ctx, placedLibrary(), Source{}); err != nil {
t.Fatal(err)
}
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
// Placement alone is a layer as well: the whole mesh's says where both accesses are.
meshWide := map[string]any{"accesses": map[string]any{
"downloads": "/storage/downloads", "series": "/storage/series"}}
if err := inv.SetSettings(ctx, "", "library", meshWide); err != nil {
t.Fatal(err)
}
if got, has, _ := inv.Layer(ctx, "", "library"); !has || !reflect.DeepEqual(roundTrip(t, got), roundTrip(t, meshWide)) {
t.Fatalf("the mesh-wide layer of placement only reads %v (%v)", got, has)
}
set := map[string]any{"log": "warn",
"places": map[string]any{"config": map[string]any{"path": "/var/lib/library/config"}},
"accesses": map[string]any{"series": "/storage/media/series"}}
if err := inv.SetSettings(ctx, "anchor", "library", set); err != nil {
t.Fatal(err)
}
stored := inv.QueryForTest(t, `select s.values, p.values from settings s join placement p using (module, node)
where s.node is not null`)
if len(stored) != 1 || stored[0][0] != `{"log":"warn"}` ||
!strings.Contains(stored[0][1], `"places"`) || !strings.Contains(stored[0][1], `"accesses"`) {
t.Fatalf("the layer and its placement are kept as %v", stored)
}
if got, _, _ := inv.Layer(ctx, "anchor", "library"); !reflect.DeepEqual(roundTrip(t, got), roundTrip(t, set)) {
t.Fatalf("the layer reads %v, was set %v", got, set)
}
// Refused as before, by the same judgement, and nothing changes.
for name, bad := range map[string]map[string]any{
"an undeclared directory": {"places": map[string]any{"cache": "/var/lib/library/cache"}},
"the machine's own": {"accesses": map[string]any{"series": "/etc"}},
"a kept network on a converged machine": {
"networks": map[string]any{"server": []any{"predecessor_default"}}},
} {
if err := inv.SetSettings(ctx, "anchor", "library", bad); err == nil {
t.Errorf("%s was kept", name)
}
}
if err := inv.SetSettings(ctx, "", "library", map[string]any{"accesses": meshWide["accesses"],
"networks": map[string]any{"server": []any{"x"}}}); err == nil || !strings.Contains(err.Error(), "per node") {
t.Errorf("a mesh-wide kept network: %v", err)
}
if got, _, _ := inv.Layer(ctx, "anchor", "library"); !reflect.DeepEqual(roundTrip(t, got), roundTrip(t, set)) {
t.Fatalf("a refused set changed the layer to %v", got)
}
// A set without placement replaces it too, and the history keeps the layer whole.
if err := inv.SetSettings(ctx, "anchor", "library", map[string]any{"log": "debug"}); err != nil {
t.Fatal(err)
}
if rows := inv.QueryForTest(t, `select module from placement where node is not null`); len(rows) != 0 {
t.Fatalf("a set without placement left it: %v", rows)
}
past, err := inv.SettingsHistory(ctx, "anchor", "library")
if err != nil || len(past) != 1 || past[0].Values["places"] == nil || past[0].Values["log"] != "warn" {
t.Fatalf("the replaced layer is not kept whole: %+v, %v", past, err)
}
// The merge gate's way in keeps a layer as the facts carry it, whole: split the same way.
if err := inv.KeepSettings(ctx, "anchor", "library", map[string]any{"log": "info",
"places": map[string]any{"config": "/var/lib/library/config"}}); err != nil {
t.Fatal(err)
}
if rows := inv.QueryForTest(t, `select values from settings where values ? 'places'`); len(rows) != 0 {
t.Fatalf("a kept layer put placement among the settings: %v", rows)
}
for _, node := range []string{"", "anchor"} {
if err := inv.ClearSettings(ctx, node, "library"); err != nil {
t.Fatal(err)
}
}
if rows := inv.QueryForTest(t, `select module from placement`); len(rows) != 0 {
t.Fatalf("a clear left the placement: %v", rows)
}
}
func roundTrip(t *testing.T, v map[string]any) map[string]any {
t.Helper()
b, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
out := map[string]any{}
if err := json.Unmarshal(b, &out); err != nil {
t.Fatal(err)
}
return out
}
+25
View File
@@ -0,0 +1,25 @@
package inventory
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The manifest the mesh holds carries each file's text (novox/hq ADR 0349 §2): one still naming a template — a
// build by a builder that predates templates, or a manifest handed over unbuilt — would compose a file with no
// text, and is refused by name.
func TestAManifestStillNamingATemplateIsNotRegistered(t *testing.T) {
inv := ForTest(t)
m := catalogue.Manifest{Module: "notifier", Version: "1", Resources: []map[string]any{
{"id": "conf", "type": "file", "path": "/etc/notifier.conf", "template": "files/notifier.conf"}}}
err := inv.RegisterModule(t.Context(), m, Source{})
if err == nil || !strings.Contains(err.Error(), "files/notifier.conf") {
t.Fatalf("a manifest naming a template was registered: %v", err)
}
m.Resources[0] = map[string]any{"id": "conf", "type": "file", "path": "/etc/notifier.conf", "content": "x"}
if err := inv.RegisterModule(t.Context(), m, Source{}); err != nil {
t.Fatalf("the same manifest with its text was refused: %v", err)
}
}
+21 -7
View File
@@ -20,9 +20,11 @@ func (i *Inventory) Layer(ctx context.Context, nodeName, module string) (map[str
if err != nil {
return nil, false, err
}
// Said whole, as it was set: its config values and the placement kept beside them (novox/hq ADR 0348
// clause 11), so a person reading it before setting it sees every key they may keep or remove.
var raw []byte
err = i.store.Pool().QueryRow(ctx,
`select values from settings where module = $1 and node is not distinct from $2::uuid`,
err = i.store.Pool().QueryRow(ctx, `select `+layerWhole+` from settings s
where s.module = $1 and s.node is not distinct from $2::uuid`,
module, nodeID).Scan(&raw)
if errors.Is(err, pgx.ErrNoRows) {
return nil, false, nil
@@ -46,6 +48,9 @@ type PastLayer struct {
ReplacedBy string
// SetBy is who had set the layer, when it was kept (novox/hq ADR 0277); empty for one set before that was kept.
SetBy string
// Why is why it went, where that was said: "moved to placement, ADR 0348 clause 11" for a layer whose
// placement keys migration 0097 moved out of it (ReplacedBy "moved").
Why string
}
// LayerOrigin is who set a layer and when (novox/hq ADR 0277): empty words for a layer set before who set it was
@@ -79,7 +84,7 @@ func (i *Inventory) SettingsHistory(ctx context.Context, nodeName, module string
return nil, err
}
rows, err := i.store.Pool().Query(ctx,
`select values, set_at, replaced_at, replaced_by, set_by from settings_history
`select values, set_at, replaced_at, replaced_by, set_by, why from settings_history
where module = $1 and node is not distinct from $2::uuid order by replaced_at desc`,
module, nodeID)
if err != nil {
@@ -90,13 +95,16 @@ func (i *Inventory) SettingsHistory(ctx context.Context, nodeName, module string
for rows.Next() {
var raw []byte
var p PastLayer
var by *string
if err := rows.Scan(&raw, &p.SetAt, &p.ReplacedAt, &p.ReplacedBy, &by); err != nil {
var by, why *string
if err := rows.Scan(&raw, &p.SetAt, &p.ReplacedAt, &p.ReplacedBy, &by, &why); err != nil {
return nil, err
}
if by != nil {
p.SetBy = *by
}
if why != nil {
p.Why = *why
}
if err := json.Unmarshal(raw, &p.Values); err != nil {
return nil, err
}
@@ -107,9 +115,15 @@ func (i *Inventory) SettingsHistory(ctx context.Context, nodeName, module string
// keepReplacedSQL copies a layer into the history before it is replaced or cleared; run in the same
// transaction as the write where there is one, so a write that fails leaves no history of it.
//
// The layer is kept whole, as it was set: its config values and its placement (novox/hq ADR 0348 clause 11).
const keepReplacedSQL = `insert into settings_history (node, module, values, set_at, replaced_by, set_by)
select node, module, values, set_at, $3, set_by from settings
where module = $1 and node is not distinct from $2::uuid`
select s.node, s.module, ` + layerWhole + `, s.set_at, $3, s.set_by from settings s
where s.module = $1 and s.node is not distinct from $2::uuid`
// layerWhole is a settings row `s` with the placement kept beside it folded back in: the layer as it was set.
const layerWhole = `s.values || coalesce((select p.values from placement p
where p.module = s.module and p.node is not distinct from s.node), '{}'::jsonb)`
// layerNode is the node id of a layer, nil for the whole mesh's.
func (i *Inventory) layerNode(ctx context.Context, nodeName string) (*string, error) {
+3
View File
@@ -61,3 +61,6 @@
./cmd/mesh-controller TestAProviderElsewhereServesItsOwnComposedName
./cmd/mesh-controller TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne
./internal/inventory TestAReportOfABusyContainerIsKeptAndReadBack
./internal/inventory TestPlacementMovesOutOfTheSettingsLayers
./internal/inventory TestASettingRoutesPlacementBesideTheLayer
./cmd/mesh-controller TestADeclarationIsTheSameAfterPlacementMoves