Merge remote-tracking branch 'origin/main' into fix/delivery-records-true

This commit is contained in:
2026-10-26 22:21:12 +01:00
69 changed files with 1366 additions and 100 deletions
+2
View File
@@ -35,6 +35,8 @@ modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image
/checks/paths/paths
/modules/mesh-build-go/mesh-build-go
/modules/mesh-delivery-gate/mesh-delivery-gate
modules/mesh-config/cmd/mesh-config/mesh-config
checks/placement/placement
# Built by their modules' build step (issue 1577), never committed.
modules/forticlient/cmd/forticlient-tools/forticlient-tools
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and the files it names.
modules
+10 -1
View File
@@ -1 +1,10 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every manifest through mesh-sdk's ReadTreeManifest: the root's predefined keys and each templated module's
# templates (files/ by default, or the directory its manifest names); and the root's kept count of resources that do not say how they are ready.
modules/*/module.json
predefined.json
modules/*/files
modules/*/templates
health-undeclared
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest, and whether the path each image artifact builds from is there.
modules
+7 -1
View File
@@ -1 +1,7 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest, a TypeScript module's package.json and whether its lock is there.
modules/*/module.json
modules/*/package.json
modules/*/package-lock.json
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every file under modules/ (git ls-files and a walk).
modules
+7 -1
View File
@@ -1 +1,7 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# that none of the old check scripts stands at the root.
?merge-check.sh
?merge-check-typescript.sh
?check-nested-modules.sh
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest; its lists are this package's own files.
modules/*/module.json
+6 -1
View File
@@ -1 +1,6 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# check("../.."): the root's list and every file of the modules it names.
particulars
modules
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and every file of a module.
modules
+5
View File
@@ -0,0 +1,5 @@
module checks/placement
go 1.26.0
require git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117
+14
View File
@@ -0,0 +1,14 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117 h1:88CNZOoQgu7YPYKW16qkWT8Gw9Ucg07aTrIImA2nYUk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+80
View File
@@ -0,0 +1,80 @@
// Command placement is the catalogue's check of its placement files (novox/hq ADR 0549 §1, ADR 0555; issue 1126): every
// `placement/<node>.json` reads as a placement of the node its name says, and the files together are refused, every
// finding named at once, for two modules of one node on one host port, a pin, binding or holding naming a provider not
// placed where it says, a seat held once for the mesh held twice, two nodes with one address on the private network, or
// more than one hub (mesh-sdk delivery.CheckPlacements).
//
// A module the catalogue does not hold and a node not enrolled are the composition check's to refuse: placement names
// modules of every repository the indexer reads and the nodes the mesh enrolled, neither of which this tree holds.
//
// go run . <catalogue root>
//
// Exits 1 on a finding.
package main
import (
"fmt"
"os"
"path/filepath"
"sort"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Check reads every placement file under root and answers each finding; none is a catalogue whose placement holds. A
// catalogue with no placement directory has none to check.
func Check(root string) ([]string, error) {
dir := filepath.Join(root, delivery.PlacementDir)
entries, err := os.ReadDir(dir)
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, err
}
var found []string
placements := map[string]delivery.Placement{}
for _, e := range entries {
rel := delivery.PlacementDir + "/" + e.Name()
node, ok := delivery.PlacementFileNode(rel)
if e.IsDir() || !ok {
found = append(found, fmt.Sprintf("%s is not a placement file: the directory holds only <node>.json, one per node", rel))
continue
}
raw, err := os.ReadFile(filepath.Join(dir, e.Name()))
if err != nil {
return nil, err
}
p, err := delivery.DecodePlacement(raw)
if err != nil {
found = append(found, fmt.Sprintf("%s does not read as a placement: %v", rel, err))
continue
}
if p.Revision != "" {
found = append(found, fmt.Sprintf("%s names a revision, which only the indexer writes into the key", rel))
}
placements[node] = p
}
all := func(string) bool { return true }
found = append(found, delivery.CheckPlacements(placements, all, all)...)
sort.Strings(found)
return found, nil
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: placement <catalogue root>")
os.Exit(2)
}
found, err := Check(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
for _, f := range found {
fmt.Println(f)
}
if len(found) > 0 {
os.Exit(1)
}
}
+55
View File
@@ -0,0 +1,55 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// The catalogue's placement files as the merge check reads them (ADR 0549 §1, ADR 0555): the real tree holds every
// rule, and a tree that breaks them is refused with every finding named at once.
func TestPlacementFiles(t *testing.T) {
found, err := Check("../..")
if err != nil {
t.Fatal(err)
}
for _, f := range found {
t.Errorf("%s (novox/hq ADR 0549 §1)", f)
}
root := t.TempDir()
write := func(name, body string) {
t.Helper()
if err := os.MkdirAll(filepath.Join(root, "placement"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, "placement", name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
if found, err := Check(root); err != nil || len(found) != 0 {
t.Fatalf("a catalogue with no placement directory: %v, %v", found, err)
}
write("ace.json", `{"v":1,"node":"ace","decided":{"hub":true,"address":"10.77.0.1"},"modules":[{"module":"a","ports":{"80":8080}},{"module":"b","ports":{"81":8080}}]}`)
write("g14.json", `{"v":1,"node":"g14","decided":{"hub":true,"address":"10.77.0.1"},"modules":[{"module":"c","bindings":{"pg":{"node":"ace","module":"pg"}}}]}`)
write("novox.json", `{"v":1,"node":"shanks","modules":[]}`)
write("bad.json", `{"v":1,"node":"bad","modules":{}}`)
write("dated.json", `{"v":1,"node":"dated","revision":"abc","modules":[]}`)
write("notes.md", "x")
found, err = Check(root)
if err != nil {
t.Fatal(err)
}
for _, want := range []string{"a and b are both published on host port 8080", "bound to ace/pg", "both have the address 10.77.0.1",
"ace, g14 are each placed as it", "placement file of novox names the node", "bad.json does not read", "dated.json names a revision",
"notes.md is not a placement file"} {
hit := false
for _, f := range found {
hit = hit || strings.Contains(f, want)
}
if !hit {
t.Errorf("no finding says %q; found %v", want, found)
}
}
}
+6 -1
View File
@@ -1 +1,6 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the root's list of predefined keys and every manifest that names one.
predefined.json
modules/*/module.json
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and the files it names.
modules
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest and its templates.
modules
+5 -1
View File
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# Check("../.."): every manifest.
modules/*/module.json
-1
View File
@@ -1 +0,0 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+7 -1
View File
@@ -1 +1,7 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# run("../.."): the root's lists of retired words and lines, and every module's texts.
retired-words
retired-lines
modules
@@ -1 +1,9 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of this package in every module that carries one.
modules/xorg/internal/desktop/*.go
modules/lemurs/internal/desktop/*.go
modules/i3/internal/desktop/*.go
modules/xterm/internal/desktop/*.go
modules/adwaita/internal/desktop/*.go
@@ -1 +1,25 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of desktop.go and its tests in every module that carries one.
modules/blueman/cmd/blueman-tools/desktop.go
modules/blueman/cmd/blueman-tools/desktop_test.go
modules/blueman/cmd/blueman-tools/copies_test.go
modules/forticlient/cmd/forticlient-tools/desktop.go
modules/forticlient/cmd/forticlient-tools/desktop_test.go
modules/forticlient/cmd/forticlient-tools/copies_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/copies_test.go
modules/nm-applet/cmd/nm-applet-tools/desktop.go
modules/nm-applet/cmd/nm-applet-tools/desktop_test.go
modules/nm-applet/cmd/nm-applet-tools/copies_test.go
modules/openrazer/cmd/openrazer-tools/desktop.go
modules/openrazer/cmd/openrazer-tools/desktop_test.go
modules/openrazer/cmd/openrazer-tools/copies_test.go
modules/polychromatic/cmd/polychromatic-tools/desktop.go
modules/polychromatic/cmd/polychromatic-tools/desktop_test.go
modules/polychromatic/cmd/polychromatic-tools/copies_test.go
modules/slack/cmd/slack-tools/desktop.go
modules/slack/cmd/slack-tools/desktop_test.go
modules/slack/cmd/slack-tools/copies_test.go
+6 -1
View File
@@ -1 +1,6 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# mesh-mail's store.json is the same as brother's; no module scan is left.
modules/mesh-mail/files/store.json
?modules/scan
@@ -1 +1,25 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of desktop.go and its tests in every module that carries one.
modules/blueman/cmd/blueman-tools/desktop.go
modules/blueman/cmd/blueman-tools/desktop_test.go
modules/blueman/cmd/blueman-tools/copies_test.go
modules/forticlient/cmd/forticlient-tools/desktop.go
modules/forticlient/cmd/forticlient-tools/desktop_test.go
modules/forticlient/cmd/forticlient-tools/copies_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/copies_test.go
modules/nm-applet/cmd/nm-applet-tools/desktop.go
modules/nm-applet/cmd/nm-applet-tools/desktop_test.go
modules/nm-applet/cmd/nm-applet-tools/copies_test.go
modules/openrazer/cmd/openrazer-tools/desktop.go
modules/openrazer/cmd/openrazer-tools/desktop_test.go
modules/openrazer/cmd/openrazer-tools/copies_test.go
modules/polychromatic/cmd/polychromatic-tools/desktop.go
modules/polychromatic/cmd/polychromatic-tools/desktop_test.go
modules/polychromatic/cmd/polychromatic-tools/copies_test.go
modules/slack/cmd/slack-tools/desktop.go
modules/slack/cmd/slack-tools/desktop_test.go
modules/slack/cmd/slack-tools/copies_test.go
+19 -3
View File
@@ -559,7 +559,7 @@ export class GiteaClient {
* **A listing the forge cut is no listing** (it says `truncated` on a page with nothing more to read, or the pages
* run past a bound): the call fails naming the repository and the commit, never answering the directories it did
* see, since a directory left out would be taken for one without a module. Reads only. */
async commitTrees(owner: string, repo: string, sha: string, paths?: Record<string, string[]>): Promise<{ root: string; trees: Record<string, string>; paths?: Record<string, Record<string, string>> }> {
async commitTrees(owner: string, repo: string, sha: string, paths?: Record<string, string[]>, listed?: string[]): Promise<{ root: string; trees: Record<string, string>; paths?: Record<string, Record<string, string>>; listed?: Record<string, Record<string, string>> }> {
const base = `${repoPath(owner, repo)}/git/trees/${commitSegment(sha)}?recursive=true`;
const cut = (why: string) => new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} was cut (${why}); no trees are answered from a part`);
const entries: any[] = [];
@@ -586,7 +586,23 @@ export class GiteaClient {
if (!tree) throw new Error(`the forge's tree listing of ${owner}/${repo} at ${sha} holds ${e.path} but not the tree of ${dir}`);
trees[dir] = tree;
}
if (paths === undefined) return { root, trees };
// **Listed directories** (novox/hq ADR 0549 §2, issue 1126): each asked directory's files directly under it, by name
// to blob id, from the same listing: how the forge bridge finds the catalogue's placement files, one per node, at a
// merge. A directory the commit lacks lists nothing (the catalogue before its first placement file).
let files: Record<string, Record<string, string>> | undefined;
if (listed !== undefined) {
files = {};
for (const dir of listed) {
files[dir] = {};
const prefix = dir === "" ? "" : `${dir}/`;
for (const e of entries) {
if (e?.type !== "blob" || typeof e.path !== "string" || !e.path.startsWith(prefix)) continue;
const name = e.path.slice(prefix.length);
if (name !== "" && !name.includes("/")) files[dir][name] = String(e.sha);
}
}
}
if (paths === undefined) return files === undefined ? { root, trees } : { root, trees, listed: files };
// **Named paths** (novox/hq ADR 0499 §3, issue 1758): each asked path's git object id at the commit, a file's or a
// directory's, from the same listing, by directory and by path relative to it. A path the listing lacks fails
// naming both, and nothing is answered in part.
@@ -601,7 +617,7 @@ export class GiteaClient {
named[dir][p] = id;
}
}
return { root, trees, paths: named };
return files === undefined ? { root, trees, paths: named } : { root, trees, paths: named, listed: files };
}
/** Whether a repository holds a file at a commit: true, false for a 404, thrown when the forge cannot say. */
+15 -1
View File
@@ -183,6 +183,15 @@ test("named paths: each asked path's git object id at the commit, a file's or a
});
assert.deepEqual(got.trees, { "": ROOT, "modules/a": h("4"), "modules/b/c": h("8") });
assert.equal((await client.commitTrees("novox", "hq", COMMIT)).paths, undefined, "no paths asked, none answered");
// Listed directories (ADR 0549 §2, issue 1126): the files directly under each, not deeper; a missing one lists none.
const listed = await client.commitTrees("novox", "hq", COMMIT, undefined, ["modules/a", "modules/b", "", "placement"]);
assert.deepEqual(listed.listed, {
"modules/a": { "main.go": h("6"), "module.json": h("5") },
"modules/b": {},
"": { "README.md": h("1"), "module.json": h("2") },
placement: {},
});
assert.equal((await client.commitTrees("novox", "hq", COMMIT)).listed, undefined, "no directory asked, none listed");
} finally {
server.close();
}
@@ -209,7 +218,12 @@ test("gitea_commit_trees takes paths as {directory: [paths]} and refuses another
const { client, server, asked } = await serve([{ sha: COMMIT, url: "u", tree: LISTING, truncated: false, page: 1, total_count: LISTING.length }]);
try {
const tool = getGiteaTools(client).find((t) => t.name === "gitea_commit_trees")!;
assert.ok("paths" in tool.input, "the tool names the argument");
assert.ok("paths" in tool.input && "listed" in tool.input, "the tool names the arguments");
const listedAnswer: any = await tool.run({ owner: "novox", repo: "hq", sha: COMMIT, listed: ["modules/d"] } as any);
assert.deepEqual(listedAnswer.listed, { "modules/d": { "notes.md": h("b") } });
for (const bad of ["modules/d", ["../x"], ["/abs"], [1]]) {
await assert.rejects(tool.run({ owner: "novox", repo: "hq", sha: COMMIT, listed: bad } as any), /listed/, JSON.stringify(bad));
}
const got: any = await tool.run({ owner: "novox", repo: "hq", sha: COMMIT, paths: { "modules/a": ["main.go"] } } as any);
assert.deepEqual(got.paths, { "modules/a": { "main.go": h("6") } });
const before = asked.length;
+14 -2
View File
@@ -76,6 +76,17 @@ function namedPaths(raw: unknown): Record<string, string[]> | undefined {
return out;
}
/** The `listed` argument of gitea_commit_trees: [directory], each plain (no `.` or `..` segment, not absolute, the root
* the empty name); anything else is refused before the forge is asked. Absent is absent. */
function listedDirs(raw: unknown): string[] | undefined {
if (raw === undefined || raw === null) return undefined;
if (!Array.isArray(raw) || raw.some((d) => typeof d !== "string")) throw new Error("listed is [directory, ...]");
for (const d of raw as string[]) {
if (d !== "" && d.split("/").some((seg) => seg === "" || seg === "." || seg === "..")) throw new Error(`listed: ${JSON.stringify(d)} is not a plain relative directory`);
}
return raw as string[];
}
export function getGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefinition[] {
return namesChecked(allGiteaTools(gitea, admin));
}
@@ -550,19 +561,20 @@ function allGiteaTools(gitea: GiteaClient, admin?: () => GiteaAdmin): ToolDefini
},
{
name: "gitea_commit_trees",
description: "A commit's trees from the forge's one recursive listing: the root tree, and the tree of every directory that holds a module.json (the root as an empty name), each by its git hash. What the forge bridge puts in a merge's message (novox/hq ADR 0437 §5). With paths, {directory: [path relative to it]}, the answer also carries paths, {directory: {path: git object id}} of each named file or directory at the commit (novox/hq ADR 0499 §3); a path the commit lacks fails naming the directory and the path. The commit is a full 40-hex sha. A listing the forge cut is refused, never answered in part. Reads only.",
description: "A commit's trees from the forge's one recursive listing: the root tree, and the tree of every directory that holds a module.json (the root as an empty name), each by its git hash. What the forge bridge puts in a merge's message (novox/hq ADR 0437 §5). With paths, {directory: [path relative to it]}, the answer also carries paths, {directory: {path: git object id}} of each named file or directory at the commit (novox/hq ADR 0499 §3); a path the commit lacks fails naming the directory and the path. The commit is a full 40-hex sha. With listed, [directory], the answer also carries listed, {directory: {name: git blob id}} of the files directly under each (the catalogue's placement files, novox/hq ADR 0549 §2). A listing the forge cut is refused, never answered in part. Reads only.",
input: {
owner: { type: "string", description: "the repository owner" },
repo: { type: "string", description: "the repository name" },
sha: { type: "string", description: "the commit, its full 40-hex sha" },
paths: { type: "object", description: "optional, {directory: [path relative to it]}: the git object id of each named file or directory is answered under paths; the root directory is the empty name" },
listed: { type: "array", items: { type: "string" }, description: "optional, [directory]: the files directly under each, {directory: {name: git blob id}}, are answered under listed; a directory the commit lacks lists none" },
},
run: async (args) => {
const owner = String(args.owner ?? "").trim(), repo = String(args.repo ?? "").trim();
const sha = String(args.sha ?? "").trim();
if (!owner || !repo) throw new Error("name the repository: owner and repo");
if (!/^[0-9a-f]{40}$/i.test(sha)) throw new Error("sha is a commit's full 40-hex sha");
return gitea.commitTrees(owner, repo, sha, namedPaths(args.paths));
return gitea.commitTrees(owner, repo, sha, namedPaths(args.paths), listedDirs(args.listed));
},
},
{
+9 -1
View File
@@ -1 +1,9 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of this package in every module that carries one.
modules/xorg/internal/desktop/*.go
modules/lemurs/internal/desktop/*.go
modules/i3/internal/desktop/*.go
modules/xterm/internal/desktop/*.go
modules/adwaita/internal/desktop/*.go
@@ -1 +1,9 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of this package in every module that carries one.
modules/xorg/internal/desktop/*.go
modules/lemurs/internal/desktop/*.go
modules/i3/internal/desktop/*.go
modules/xterm/internal/desktop/*.go
modules/adwaita/internal/desktop/*.go
@@ -116,7 +116,7 @@ func TestTheIntegrationLevelDigestsUnderBothTags(t *testing.T) {
t.Errorf("the integration digest listed %q", s.command)
}
}
if DigestVersion != "5" {
if DigestVersion != "6" {
t.Errorf("the digest's form changed in kind and is still version %s", DigestVersion)
}
}
+138 -14
View File
@@ -16,6 +16,7 @@ import (
"io"
"io/fs"
"os"
"path"
"path/filepath"
"sort"
"strconv"
@@ -29,17 +30,25 @@ import (
// commits (the lead's decision L5) and the integration level's tags integration,isolation (the checker's agreement).
// 4: the siblings' commits enter only the digest of a package whose tests reach a file naming MESH_CHECK_BESIDE (ADR 0536 §1).
// 5: a catalogue module's non-Go files enter the digest of its packages (issue 1935).
const DigestVersion = "5"
// 6: a reads-outside marker that lists what its tests read enters the digest with those files' contents (issue 1936).
const DigestVersion = "6"
// besideVariable is the name a file bears when it reads where the siblings lie; a package whose tests reach such a file
// depends on the siblings' commits.
const besideVariable = "MESH_CHECK_BESIDE"
// ReadsOutsideMarker is the name of a file in a package's directory that says its tests read files outside it (a
// repository's documents, a sibling directory): such a package is never kept (issue 1935). The digest hashes only what
// the Go toolchain lists, so it cannot name what a test opens elsewhere; a marker file, beside the Go files where a
// reader of the package sees it, is the plain way to say so until the checker keeps the files a test opened (the Go
// test log) with each verdict.
// repository's documents, a sibling directory), which the Go toolchain does not list (issue 1935). It lists, one per
// line, what those tests read, as paths from the repository's root; a path may be a glob (filepath.Match), and a
// directory it names is read whole (issue 1936). Blank lines and lines starting with # are not read. A line starting
// with ? names a path whose absence is what the tests read (a file that must not be there): it may match nothing.
//
// The digest then hashes, besides what the toolchain lists, every line of the marker, every file it names (under its
// path in the tree, by content) and every file of the package's own directory, Go files included (a file its build tags
// exclude is one go list does not hash). A marker that lists nothing, a line that is not a clean relative path, or a
// line without ? that matches nothing makes the package one that is never kept (its digest a fresh random hex): a list
// nobody can trust keeps nothing. The exact fix, the files a test opened kept
// from the Go test log with each verdict, replaces the lists (issue 1936).
const ReadsOutsideMarker = "reads-outside"
// NeedsMarker is the name of a file in a package's directory that names, one per line, what its integration tests need
@@ -128,9 +137,13 @@ type DigestResult struct {
// only when they are the siblings it gave the fetch.
Beside []string `json:"beside,omitempty"`
WithoutTests []string `json:"without_tests,omitempty"`
// ReadsOutside names the packages that carry the marker: each has a digest that no kept verdict can have (a fresh
// random one), so the check runs it every time.
// ReadsOutside names the packages whose marker lists nothing it can trust (ReadsOutsideMarker): each has a digest
// that no kept verdict can have (a fresh random one), so the check runs it every time.
ReadsOutside []string `json:"reads_outside,omitempty"`
// neverKept is, per package of ReadsOutside, why its marker cannot be trusted; hashedFiles caches the content lines
// of the files markers name, which several packages share (the catalogue's manifests).
neverKept map[string]string
hashedFiles map[string]string
// Needs are the packages whose tests at this level need something of the node, as their NeedsMarker names it.
Needs []delivery.PackageNeeds `json:"needs,omitempty"`
// JudgedBy is the program that ran the digest (ADR 0535 §5): "installed", or "candidate <merge commit>" under the
@@ -247,7 +260,7 @@ func (m *Module) digestHere(ctx context.Context, workspace, level string) (Diges
sort.Slice(res.Needs, func(i, j int) bool { return res.Needs[i].Package < res.Needs[j].Package })
for _, p := range res.ReadsOutside {
if m.Say != nil {
m.Say("the %s level of %s always runs: its directory carries %s, so its tests read files outside it that no digest names", level, p, ReadsOutsideMarker)
m.Say("the %s level of %s always runs: its %s %s, so its tests read files outside it that no digest names", level, p, ReadsOutsideMarker, res.neverKept[p])
}
}
return res, nil
@@ -316,18 +329,27 @@ func digestModule(res *DigestResult, pkgs map[string]*goPackage, tree string, he
res.Needs = append(res.Needs, delivery.PackageNeeds{Package: path, Needs: n})
}
}
if _, err := os.Stat(filepath.Join(root.Dir, ReadsOutsideMarker)); err == nil {
// The digest of a package that reads outside itself is one nothing can have kept: the key is a fresh random
// hex, so the ask finds no kept verdict and runs it, and what the checker writes under it is never read again.
declared, why, err := declaredReads(tree, root, res.hashed())
if err != nil {
return fmt.Errorf("%s: its %s: %w", path, ReadsOutsideMarker, err)
}
if why != "" {
// The digest of a package whose reads outside itself are not listed is one nothing can have kept: the key is a
// fresh random hex, so the ask finds no kept verdict and runs it, and what the checker writes under it is never
// read again.
var nonce [32]byte
if _, err := rand.Read(nonce[:]); err != nil {
return err
}
res.Packages[path] = hex.EncodeToString(nonce[:])
res.ReadsOutside = append(res.ReadsOutside, path)
if res.neverKept == nil {
res.neverKept = map[string]string{}
}
res.neverKept[path] = why
continue
}
d, err := packageDigest(root, testMain, pkgs, tree, header, beside)
d, err := packageDigest(root, testMain, pkgs, tree, header, beside, declared)
if err != nil {
return fmt.Errorf("%s: %w", path, err)
}
@@ -340,14 +362,15 @@ func digestModule(res *DigestResult, pkgs map[string]*goPackage, tree string, he
// tests reach in the module's own tree (by their content, under their path in the tree), the version and checksum of
// every module outside it, and the package's own test data. The beside lines (the siblings' commits) join the header
// only when a file the tests reach names MESH_CHECK_BESIDE. A package of the standard library is the Go version's.
func packageDigest(root, testMain *goPackage, pkgs map[string]*goPackage, tree string, header, beside []string) (string, error) {
// The lines a package's ReadsOutsideMarker declares join them (declaredReads).
func packageDigest(root, testMain *goPackage, pkgs map[string]*goPackage, tree string, header, beside, declared []string) (string, error) {
reached := map[string]*goPackage{root.ImportPath: root}
for _, dep := range testMain.Deps {
if p := pkgs[dep]; p != nil {
reached[dep] = p
}
}
lines := []string{}
lines := append([]string{}, declared...)
readsBeside := false
for _, p := range reached {
if p.Standard || p.Module == nil {
@@ -469,3 +492,104 @@ func contentLine(tree string, p *goPackage, path string) (string, error) {
sum := sha256.Sum256(raw)
return "file " + label + " " + hex.EncodeToString(sum[:]), nil
}
// hashed is the cache of content lines a digest step shares across its packages.
func (r *DigestResult) hashed() map[string]string {
if r.hashedFiles == nil {
r.hashedFiles = map[string]string{}
}
return r.hashedFiles
}
// declaredReads are the digest lines a package's ReadsOutsideMarker adds: each of its lines, the content line of every
// file it names, and those of every file of the package's own directory. Without a marker it answers nothing. why is
// set, and the lines nil, when the marker cannot be trusted: it lists nothing, a line is not a clean relative path, or
// a line without ? matches nothing.
func declaredReads(tree string, root *goPackage, cache map[string]string) (lines []string, why string, err error) {
raw, err := os.ReadFile(filepath.Join(root.Dir, ReadsOutsideMarker))
if errors.Is(err, fs.ErrNotExist) {
return nil, "", nil
}
if err != nil {
return nil, "", err
}
var patterns []string
for _, l := range strings.Split(string(raw), "\n") {
if l = strings.TrimSpace(l); l != "" && !strings.HasPrefix(l, "#") {
patterns = append(patterns, l)
}
}
if len(patterns) == 0 {
return nil, "lists nothing", nil
}
seen := map[string]bool{}
add := func(path string) error {
if seen[path] {
return nil
}
seen[path] = true
l, ok := cache[path]
if !ok {
var err error
if l, err = contentLine(tree, root, path); err != nil {
return err
}
cache[path] = l
}
lines = append(lines, "read "+l)
return nil
}
for _, l := range patterns {
pattern, absent := strings.CutPrefix(l, "?")
if !cleanTreePattern(pattern) {
return nil, fmt.Sprintf("names %q, which is not a clean path from the repository's root", l), nil
}
matches, err := filepath.Glob(filepath.Join(tree, filepath.FromSlash(pattern)))
if err != nil {
return nil, fmt.Sprintf("names %q, which is not a glob", l), nil
}
if len(matches) == 0 && !absent {
return nil, fmt.Sprintf("names %q, which matches nothing (a path whose absence is read starts with ?)", l), nil
}
lines = append(lines, "reads "+l)
for _, match := range matches {
if err := walkFiles(match, add); err != nil {
return nil, "", err
}
}
}
// Every file of the package's own directory: its lists, its marker, and its Go files too, since go list leaves a
// file its build tags exclude (an integration test a unit test globs) out of what the digest hashes. A file the
// toolchain lists is hashed twice, which changes nothing.
err = walkFiles(root.Dir, add)
return lines, "", err
}
// cleanTreePattern says a marker's line is a path from the repository's root that stays inside it.
func cleanTreePattern(p string) bool {
if p == "" || strings.ContainsAny(p, " \t\\") || strings.HasPrefix(p, "/") || path.Clean(p) != p {
return false
}
for _, e := range strings.Split(p, "/") {
if e == ".." || e == "." {
return false
}
}
return true
}
// walkFiles calls each with every file at or under path, skipping .git and node_modules, in lexical order.
func walkFiles(path string, each func(string) error) error {
return filepath.WalkDir(path, func(p string, e fs.DirEntry, err error) error {
if err != nil {
return err
}
if e.IsDir() {
if n := e.Name(); p != path && (n == ".git" || n == "node_modules") {
return filepath.SkipDir
}
return nil
}
return each(p)
})
}
@@ -173,7 +173,7 @@ func TestAModuleOutsideTheTreeCountsByVersionAndChecksum(t *testing.T) {
testMain := &goPackage{ImportPath: "x/r.test", Deps: []string{ext.ImportPath}}
pkgs[root.ImportPath] = root
digest := func() string {
d, err := packageDigest(root, testMain, pkgs, dir, []string{"digest 1"}, nil)
d, err := packageDigest(root, testMain, pkgs, dir, []string{"digest 1"}, nil, nil)
if err != nil {
t.Fatal(err)
}
@@ -190,28 +190,74 @@ func TestAModuleOutsideTheTreeCountsByVersionAndChecksum(t *testing.T) {
}
}
// Issue 1935: a package whose directory carries the reads-outside marker is never kept: its digest is one no kept verdict
// can have (it differs on every call, though nothing changed), and the answer names it. A package without the marker
// keeps its digest.
func TestAPackageWithTheReadsOutsideMarkerIsNeverKept(t *testing.T) {
// Issue 1935, 1936: a package whose reads-outside marker lists nothing it can trust is never kept: its digest is one no
// kept verdict can have (it differs on every call, though nothing changed), and the answer names it. A marker that lists
// what the tests read is kept: a change to a file it names changes the package's digest, a change elsewhere does not.
func TestAPackageWithTheReadsOutsideMarkerIsKeptOnlyByWhatItLists(t *testing.T) {
t.Parallel()
m, tree := digestFixture(t)
plain := digests(t, m, "unit")
if len(plain.ReadsOutside) != 0 {
t.Fatalf("a package carries the marker in the fixture: %v", plain.ReadsOutside)
}
edit(t, tree, "c/"+ReadsOutsideMarker, "the tests read ../docs\n")
first, second := digests(t, m, "unit"), digests(t, m, "unit")
c := "fixture/digest/c"
if len(first.Packages[c]) != 64 || first.Packages[c] == second.Packages[c] || first.Packages[c] == plain.Packages[c] {
t.Errorf("c carries the marker and its digest is stable: %s, %s, before %s", first.Packages[c], second.Packages[c], plain.Packages[c])
c, a := "fixture/digest/c", "fixture/digest/a"
if err := os.MkdirAll(filepath.Join(tree, "docs"), 0o755); err != nil {
t.Fatal(err)
}
if strings.Join(first.ReadsOutside, ",") != c {
t.Errorf("the answer does not name the package that reads outside: %v", first.ReadsOutside)
edit(t, tree, "docs/one.md", "one\n")
edit(t, tree, "other.md", "other\n")
for name, marker := range map[string]string{
"an empty marker": "# nothing listed\n\n",
"a sentence": "the tests read ../docs\n",
"a path climbing out": "../docs\n",
"a path matching nothing": "docs/none.md\n",
"an absolute path": "/etc/passwd\n",
"a path that is not clean": "docs/./one.md\n",
"a good line and a bad line": "docs\nnope/*.md\n",
} {
edit(t, tree, "c/"+ReadsOutsideMarker, marker)
first, second := digests(t, m, "unit"), digests(t, m, "unit")
if len(first.Packages[c]) != 64 || first.Packages[c] == second.Packages[c] || first.Packages[c] == plain.Packages[c] {
t.Errorf("%s: c's digest is stable: %s, %s, before %s", name, first.Packages[c], second.Packages[c], plain.Packages[c])
}
if strings.Join(first.ReadsOutside, ",") != c {
t.Errorf("%s: the answer does not name the package that reads outside: %v", name, first.ReadsOutside)
}
if first.Packages[a] != plain.Packages[a] || first.Packages[a] != second.Packages[a] {
t.Errorf("%s: a has no marker and its digest moved: %s, %s, before %s", name, first.Packages[a], second.Packages[a], plain.Packages[a])
}
}
a := "fixture/digest/a"
if first.Packages[a] != plain.Packages[a] || first.Packages[a] != second.Packages[a] {
t.Errorf("a has no marker and its digest moved: %s, %s, before %s", first.Packages[a], second.Packages[a], plain.Packages[a])
// A list: a glob, a directory read whole, and a path whose absence is read.
edit(t, tree, "c/"+ReadsOutsideMarker, "# what c's tests read\ndocs/*.md\ndocs\n?gone\n")
before := digests(t, m, "unit")
if len(before.ReadsOutside) != 0 || len(before.Packages[c]) != 64 || before.Packages[c] == plain.Packages[c] {
t.Fatalf("a listed marker is not kept: %v, %s (no marker %s)", before.ReadsOutside, before.Packages[c], plain.Packages[c])
}
if again := digests(t, m, "unit"); again.Packages[c] != before.Packages[c] {
t.Errorf("nothing changed and c's digest moved")
}
for _, step := range []struct {
file, text string
moves bool
}{
{"other.md", "changed\n", false}, // outside the list
{"d/d.go", "package d\n\n// D.\n", false}, // a package c's tests do not reach
{"docs/one.md", "changed\n", true}, // a listed file
{"docs/two.md", "new\n", true}, // a new file the glob matches
{"gone", "here\n", true}, // a path whose absence was read
{"c/note.txt", "changed\n", true}, // the package's own file that is not Go
{"c/x_integration_test.go", "//go:build integration\n\npackage c\n", true}, // a Go file its build tags leave out (go list's IgnoredGoFiles)
} {
edit(t, tree, step.file, step.text)
after := digests(t, m, "unit")
if moved := after.Packages[c] != before.Packages[c]; moved != step.moves {
t.Errorf("a change to %s: c's digest moved %v, want %v", step.file, moved, step.moves)
}
if after.Packages[a] != plain.Packages[a] {
t.Errorf("a change to %s moved a's digest", step.file)
}
before = after
}
// Issue 890/AC3: a package whose directory carries the needs marker is named with its needs at the integration level,
@@ -1 +1,7 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the manifests through mesh-sdk's validator (the predefined keys), gitea's and mesh-build-node's.
predefined.json
modules/gitea
modules/mesh-build-node
@@ -1 +1,10 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the manifests through mesh-sdk's validator (the predefined keys), with the modules they compose with.
predefined.json
modules/gitea
modules/mesh-build-go
modules/mesh-build-node
modules/buildah
modules/docker
@@ -1 +1,5 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# its manifest through mesh-sdk's validator, which reads the root's predefined keys.
predefined.json
+1 -1
View File
@@ -115,7 +115,7 @@ func resolving(c *Config) {
}
return nil
}
for _, read := range []string{AssignmentsRead, DeclaredConfigRead, PredefinedRead} {
for _, read := range []string{AssignmentsRead, PlacementRead, DeclaredConfigRead, PredefinedRead} {
go func(read string) {
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
time.Sleep(wait)
@@ -21,4 +21,16 @@ func TestAModulePulledInHasItsValuesResolvedOnItsNode(t *testing.T) {
if r, _ := c.Reconcile(); len(r.Moved) != 4 || !r.Moved[0].Gone {
t.Errorf("a module no longer pulled in moved %+v", r.Moved)
}
// 1126 (ADR 0549 §3, §5): a module the catalogue's placement names on a node resolves there as an assigned one does,
// while the two sources stand; it goes when neither names it.
m.mirror(t, PlacementRead, "placement.ace", `{"v":1,"node":"ace","modules":[{"module":"dunst"}]}`)
if r, err := c.Reconcile(); err != nil || len(r.Moved) != 4 {
t.Fatalf("a placed module's values moved %+v (%v)", r.Moved, err)
}
expect(t, m, "ace", "10", SourceDefault)
_ = m.Delete(PlacementRead, "placement.ace")
if r, _ := c.Reconcile(); len(r.Moved) != 4 || !r.Moved[0].Gone {
t.Errorf("a module no longer placed moved %+v", r.Moved)
}
}
@@ -20,7 +20,10 @@ const (
PredefinedState = "predefined"
ProposalsState = "proposals"
AssignmentsRead = "mesh-controller.assignments"
AssignmentsRead = "mesh-controller.assignments"
// PlacementRead is the indexer's placement, read from the catalogue at each merge (novox/hq ADR 0549 §2, §3): each
// node's key `placement.<node>` with the modules placed there.
PlacementRead = "mesh-delivery-indexer.placement"
DeclaredConfigRead = "mesh-controller.declared-config"
PredefinedRead = "mesh-controller.predefined"
@@ -27,6 +27,38 @@ func (c *Config) sources() (Sources, error) {
}); err != nil {
return s, fmt.Errorf("the assignments cannot be read: %w", err)
}
// And every module the catalogue's placement names on a node (ADR 0549 §3): until ADR 0547's third milestone both
// sources stand (ADR 0549 §5), so a pair either names is resolved, and none loses its values while the two differ.
// Placement that cannot be read yet (the indexer's state not on the bus) leaves the assignments alone, said.
standing := map[string]bool{}
for _, a := range s.Assignments {
standing[a.Node+"."+a.Module] = true
}
if err := c.each(PlacementRead, func(key string, raw []byte) error {
node, ok := strings.CutPrefix(key, "placement.")
if !ok {
return nil
}
var p struct {
Node string `json:"node"`
Modules []struct {
Module string `json:"module"`
} `json:"modules"`
}
if err := json.Unmarshal(raw, &p); err != nil || p.Node != node {
problems = append(problems, fmt.Sprintf("the placement %s is not one", key))
return nil
}
for _, m := range p.Modules {
if m.Module != "" && !standing[node+"."+m.Module] {
standing[node+"."+m.Module] = true
s.Assignments = append(s.Assignments, Assignment{Node: node, Module: m.Module})
}
}
return nil
}); err != nil {
problems = append(problems, fmt.Sprintf("the catalogue's placement cannot be read, so the assignments alone are resolved: %v", err))
}
if err := c.each(DeclaredConfigRead, func(key string, raw []byte) error {
m, err := ParseModuleDeclarations(raw)
if err != nil {
+1
View File
@@ -29,6 +29,7 @@
],
"reads": [
"mesh-controller.assignments",
"mesh-delivery-indexer.placement",
"mesh-controller.declared-config",
"mesh-controller.predefined"
],
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every manifest: the gate is the seat's only claimant.
modules/*/module.json
@@ -1 +1,7 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every module folder: its manifest, and whether it holds code (go.mod, package.json).
modules/*/module.json
modules/*/go.mod
modules/*/package.json
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the root's retired lines.
retired-lines
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# mesh-mail's mailevent.go, of which this package holds a copy.
modules/mesh-mail/cmd/mesh-mail/mailevent.go
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# mesh-mail's mailevent.go, of which this package holds a copy.
modules/mesh-mail/cmd/mesh-mail/mailevent.go
@@ -1 +1,5 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# messenger's content rule, of which content_rule.go is a copy.
modules/messenger/cmd/messenger/content.go
@@ -1 +1,7 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every manifest and the templates the moved modules name (in files/, or templates/).
modules/*/module.json
modules/*/files
modules/*/templates
@@ -5,6 +5,8 @@ package main
// key (or a first one) raises `secret-changed` for the machine's core secrets, the same key says nothing.
import (
"git.novox.be/novox/mesh-sdk/go/seal"
"encoding/json"
"os"
"path/filepath"
@@ -100,14 +102,17 @@ authorization {
users = [
{ user: "novox.mesh-vault", password: "pw", permissions: {
publish: { deny: [">"] }
subscribe: { allow: ["mesh.seat.mesh-vault.call.node-key.*"] }
subscribe: { allow: ["mesh.seat.mesh-vault.call.node-key.*", "mesh.seat.mesh-vault.call.own.*"] }
allow_responses: true } }
{ user: "node.ace", password: "pw", permissions: {
publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_ace"] }
publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_ace", "mesh.seat.mesh-vault.call.own.node_ace"] }
subscribe: { allow: ["_INBOX.node.ace.>"] } } }
{ user: "node.g14", password: "pw", permissions: {
publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_g14"] }
publish: { allow: ["mesh.seat.mesh-vault.call.node-key.node_g14", "mesh.seat.mesh-vault.call.own.node_g14"] }
subscribe: { allow: ["_INBOX.node.g14.>"] } } }
{ user: "controller", password: "pw", permissions: {
publish: { allow: ["mesh.seat.mesh-vault.call.own.controller"] }
subscribe: { allow: ["_INBOX.controller.>"] } } }
]
}
`
@@ -135,39 +140,48 @@ func TestOnARealBusANodeEngineStatesOnlyItsOwnKey(t *testing.T) {
t.Cleanup(s.Shutdown)
v := newVault(t, nil)
tool := toolNamed(t, definitions(&Env{Vault: v.Vault}), Seat+".node-key")
v.Askers = Declared{Read: assignments{"ace.web": `{"node":"ace","module":"web"}`}.read, Store: v.Store}
tools := map[string]toolDef{"node-key": toolNamed(t, definitions(&Env{Vault: v.Vault}), Seat+".node-key"),
"own": toolNamed(t, definitions(&Env{Vault: v.Vault}), Seat+".own")}
vaultConn := connectAs(t, s.ClientURL(), "novox.mesh-vault")
if _, err := vaultConn.Subscribe("mesh.seat.mesh-vault.call.node-key.*", func(m *nats.Msg) {
serve := func(m *nats.Msg) {
parts := strings.Split(m.Subject, ".")
var args map[string]any
_ = json.Unmarshal(m.Data, &args)
answer := busAnswer{}
if got, err := tool.runAs(untoken(parts[len(parts)-1]), args); err != nil {
if got, err := tools[parts[len(parts)-2]].runAs(untoken(parts[len(parts)-1]), args); err != nil {
answer.Error = err.Error()
} else {
answer.Result, _ = json.Marshal(got)
}
raw, _ := json.Marshal(answer)
_ = m.Respond(raw)
}); err != nil {
t.Fatal(err)
}
for _, verb := range []string{"node-key", "own"} {
if _, err := vaultConn.Subscribe("mesh.seat.mesh-vault.call."+verb+".*", serve); err != nil {
t.Fatal(err)
}
}
_ = vaultConn.Flush()
say := func(user, subject, key string) (string, error) {
raw, _ := json.Marshal(map[string]string{"key": key})
call := func(user, subject string, args any) (json.RawMessage, error) {
raw, _ := json.Marshal(args)
msg, err := connectAs(t, s.ClientURL(), user).Request(subject, raw, 3*time.Second)
if err != nil {
return "", err
return nil, err
}
var a busAnswer
_ = json.Unmarshal(msg.Data, &a)
if a.Error != "" {
return a.Error, errString(a.Error)
return nil, errString(a.Error)
}
return "", nil
return a.Result, nil
}
aceKey, _ := aSealingKey(t)
say := func(user, subject, key string) (string, error) {
_, err := call(user, subject, map[string]string{"key": key})
return "", err
}
aceKey, acePriv := aSealingKey(t)
g14Key, _ := aSealingKey(t)
if _, err := say("node.ace", "mesh.seat.mesh-vault.call.node-key.node_ace", aceKey); err != nil {
t.Fatal(err)
@@ -188,4 +202,32 @@ func TestOnARealBusANodeEngineStatesOnlyItsOwnKey(t *testing.T) {
if got, held, _ := v.Store.NodeKey("g14"); !held || got != g14Key {
t.Errorf("g14's key is %q, %v", got, held)
}
// Issue 1126 (ADR 0547 §2, ADR 0550 §5): `own` on the same bus. node.ace asks its own and gets its module's secret
// sealed to the key it stated, opening with its key alone; g14 cannot ask in ace's name, and asking in its own gets
// none of ace's; the controller is refused, as no node-engine.
if _, err := moveIn(t, v, "node.ace", MoveOf{Node: "ace", Module: "web", Name: "password", Origin: "made", Generation: "g-1",
SealingKey: aceKey}, "ace-web-pw"); err != nil {
t.Fatal(err)
}
raw, err := call("node.ace", "mesh.seat.mesh-vault.call.own.node_ace", map[string]any{})
var own OwnAnswer
if err != nil || json.Unmarshal(raw, &own) != nil || own.Node != "ace" || len(own.Own) != 1 {
t.Fatalf("ace's own over the bus: %s, %v", raw, err)
}
if value, err := seal.Open(own.Own[0].Sealed, acePriv); err != nil || string(value) != "ace-web-pw" {
t.Errorf("ace's secret does not open with ace's key: %q, %v", value, err)
}
if strings.Contains(string(raw), "ace-web-pw") {
t.Error("ace's secret crossed the bus in the clear")
}
if _, err := call("node.g14", "mesh.seat.mesh-vault.call.own.node_ace", map[string]any{}); err == nil {
t.Error("node.g14 asked ace's own in ace's name")
}
if raw, err := call("node.g14", "mesh.seat.mesh-vault.call.own.node_g14", map[string]any{}); err != nil || strings.Contains(string(raw), "web") {
t.Errorf("g14 asking its own was answered %s, %v", raw, err)
}
if _, err := call("controller", "mesh.seat.mesh-vault.call.own.controller", map[string]any{}); err == nil || !strings.Contains(err.Error(), "not a node-engine") {
t.Errorf("the controller asking own: %v", err)
}
}
@@ -0,0 +1,288 @@
package main
// A machine's node-engine composes its own modules (novox/hq ADR 0547 §2, issue 1126), and asks the vault for their own
// secrets sealed to its own key (ADR 0349 §11, ADR 0550 §5): the verb `own`, answered to the node-engine of a machine
// alone, about that machine alone (the machine is the caller's, as the bus proved it, never an argument). Each value is
// sealed to the sealing key the machine's engine stated (`node-key`), so what crosses the bus opens on that machine only,
// and the composition carries it sealed (no composition carries a secret in the clear). A module handed to the
// provisioner is left out: its client fills its secrets from `get` (ADR 0349 §17).
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// NodeOwnSecret is one own secret of a module on the asking machine, sealed to the machine's key.
type NodeOwnSecret struct {
Module string `json:"module"`
Name string `json:"name"`
Origin string `json:"origin"`
Generation int `json:"generation"`
Sealed seal.Box `json:"sealed"`
}
// OwnAnswer is what `own` answers: the machine, the sealing key every value is sealed to (raw X25519, base64, as its
// engine stated it), each own secret of its modules, and each pair credential it asked for that the vault holds.
type OwnAnswer struct {
Node string `json:"node"`
Key string `json:"key"`
Own []NodeOwnSecret `json:"own"`
Credentials []NodeCredential `json:"credentials,omitempty"`
Missing []CredentialAsked `json:"missing,omitempty"`
}
// CredentialAsked names one pair credential by its ends, as a node composing its own asks for it (mesh-sdk
// compose.CredentialKey): Role is `consumer` when the asking machine consumes it, `provider` when it grants it.
type CredentialAsked struct {
Role string `json:"role"`
Provision string `json:"provision"`
Consumer string `json:"consumer"`
ConsumerModule string `json:"consumer_module"`
Provider string `json:"provider"`
Local string `json:"local,omitempty"`
}
// NodeCredential is one pair credential answered, sealed to the asking machine's key, with the name the vault keeps it
// under (`pair/...` per pair, `provision/<provider>/<provision>/<node>/<module>` issued for one holder, or the provision's
// one shared value `provision/<provider>/<provision>`).
type NodeCredential struct {
CredentialAsked
Name string `json:"name"`
Generation int `json:"generation"`
Sealed seal.Box `json:"sealed"`
}
// credentialNames are the names a pair credential may be kept under, the most particular first.
func credentialNames(c CredentialAsked) []string {
pair := "pair/" + c.Provider + "/" + c.Provision + "/" + c.Consumer + "/" + c.ConsumerModule
if c.Local != "" {
pair += "/" + c.Local
}
return []string{pair, "provision/" + c.Provider + "/" + c.Provision + "/" + c.Consumer + "/" + c.ConsumerModule,
"provision/" + c.Provider + "/" + c.Provision}
}
// credentialRefusal says why node may not have c, or "": a consumer's credential to the machine its consuming module
// stands on, a provider's to the providing machine, and nothing to a machine that is neither.
func (v *Vault) credentialRefusal(node string, c CredentialAsked) string {
for _, part := range []string{c.Provision, c.Consumer, c.ConsumerModule, c.Provider} {
if !namePart.MatchString(part) {
return fmt.Sprintf("%q is not a name the mesh writes", part)
}
}
if c.Local != "" && !namePart.MatchString(c.Local) {
return fmt.Sprintf("%q is not a name the mesh writes", c.Local)
}
switch c.Role {
case "consumer":
if c.Consumer != node {
return fmt.Sprintf("%s asks as a consumer on %s, and is not that machine", node, c.Consumer)
}
// The consuming module stands on the machine: read, never assumed (an assignment that cannot be read refuses).
read, why := v.assignmentReader(node)
if why != "" {
return why
}
raw, err := read(node + "." + c.ConsumerModule)
if err != nil {
return fmt.Sprintf(notYetPrefix+"the assignment %s.%s could not be read: %v", node, c.ConsumerModule, err)
}
if raw == nil {
return fmt.Sprintf("%s is not on %s: neither assigned there nor required by a module there", c.ConsumerModule, node)
}
return ""
case "provider":
if c.Provider != node {
return fmt.Sprintf("%s asks as the provider %s, and is not that machine", node, c.Provider)
}
return ""
}
return fmt.Sprintf("the role %q is not consumer or provider", c.Role)
}
// assignmentReader is the vault's reader of the assignments, or why it has none: the rule `own` keeps reads them,
// and without them it refuses rather than hands (a rule that cannot be read is no rule).
func (v *Vault) assignmentReader(node string) (AssignmentReader, string) {
d, ok := v.Askers.(Declared)
if !ok || d.Read == nil {
return nil, "the vault cannot read the assignments, so it cannot tell what " + node + "'s modules are; nothing is handed"
}
return standing(d.Read), ""
}
// sharedRefusal says why node may not have the provision's one shared value `provision/<provider>/<provision>` for c,
// or "": the vault's record of who holds it names the consumer's module among its consumers, or the machine as its
// provider. Without that record nothing is handed: the shared value is not one any machine may have by naming a
// provider (ADR 0550 §5).
func (v *Vault) sharedRefusal(node string, c CredentialAsked) (string, error) {
name := "provision/" + c.Provider + "/" + c.Provision
sh, ok, err := v.Store.ReadShared(name)
if err != nil {
return "", err
}
if !ok {
return fmt.Sprintf("the vault keeps no record of who holds %s", name), nil
}
switch c.Role {
case "consumer":
want := consumerOf(c.Consumer, c.ConsumerModule, c.Local)
for _, held := range sh.Consumers {
if held == want {
return "", nil
}
}
return fmt.Sprintf("%s is held by %s, not by %s", name, strings.Join(sh.Consumers, ", "), want), nil
case "provider":
if sh.Provider == node {
return "", nil
}
return fmt.Sprintf("%s is provided from %s, not %s", name, sh.Provider, node), nil
}
return fmt.Sprintf("the role %q is not consumer or provider", c.Role), nil
}
// composeOrigin is the vault's origin of a value as the composition reads it: made by the mesh, or given (accepted at a
// terminal, typed at a desk, issued by a provider's software).
func composeOrigin(origin string) string {
if origin == "" || origin == OriginMade {
return OriginMade
}
return OriginAccepted
}
// OwnOfNode answers the node-engine of a machine the own secrets of its modules, each sealed to the machine's stated key.
func (v *Vault) OwnOfNode(caller string, asked ...CredentialAsked) (OwnAnswer, error) {
node, ok := nodeEngineOf(caller)
if !ok {
if caller == "" || caller == stdio.Unproven {
caller = "an unproven caller"
}
return OwnAnswer{}, fmt.Errorf("own answers a machine's node-engine alone, about its own machine; %s is not a node-engine", caller)
}
key, had, err := v.Store.NodeKey(node)
if err != nil {
return OwnAnswer{}, err
}
if !had {
return OwnAnswer{}, fmt.Errorf("the vault holds no secret named own/%s yet: %s's node-engine has stated no sealing key "+
"(node-key), so there is no key of its own to seal to", node, node)
}
to, err := publicPEM(key)
if err != nil {
return OwnAnswer{}, fmt.Errorf("%s's sealing key as recorded is not usable: %w", node, err)
}
all, err := v.Store.List()
if err != nil {
return OwnAnswer{}, err
}
out := OwnAnswer{Node: node, Key: key}
for _, r := range all {
parts := strings.Split(r.Name, "/")
if len(parts) != 4 || parts[0] != "own" || parts[1] != node {
continue
}
handed, err := v.handed(node, parts[2])
if err != nil {
return OwnAnswer{}, err
}
if handed {
continue
}
value, err := seal.Open(r.AtRest, v.Own.PrivateKey)
if err != nil {
return OwnAnswer{}, fmt.Errorf("%s does not open with the vault's key: %w", r.Name, err)
}
sealed, err := seal.Seal(value, to)
if err != nil {
return OwnAnswer{}, err
}
if err := v.recordHanding(Handing{To: caller, Name: r.Name, Generation: r.Generation, Outcome: "handed"}); err != nil {
return OwnAnswer{}, fmt.Errorf("%s is not handed out: who was handed it could not be recorded: %w", r.Name, err)
}
out.Own = append(out.Own, NodeOwnSecret{Module: parts[2], Name: parts[3], Origin: composeOrigin(r.Origin),
Generation: r.Generation, Sealed: sealed})
}
sort.Slice(out.Own, func(i, j int) bool {
if out.Own[i].Module != out.Own[j].Module {
return out.Own[i].Module < out.Own[j].Module
}
return out.Own[i].Name < out.Own[j].Name
})
for _, c := range asked {
if why := v.credentialRefusal(node, c); why != "" {
return OwnAnswer{}, fmt.Errorf("%s may not ask for the credential of %s from %s to %s on %s: %s", caller,
c.Provision, c.Provider, c.ConsumerModule, c.Consumer, why)
}
found := false
names := credentialNames(c)
for i, name := range names {
r, held, err := v.Store.Read(name)
if err != nil {
return OwnAnswer{}, err
}
if !held {
continue
}
if i == len(names)-1 {
// The provision's one shared value: handed only to a machine the record of its holders names.
why, err := v.sharedRefusal(node, c)
if err != nil {
return OwnAnswer{}, err
}
if why != "" {
v.log("%s is not handed %s: %s", node, name, why)
continue
}
}
value, err := seal.Open(r.AtRest, v.Own.PrivateKey)
if err != nil {
return OwnAnswer{}, fmt.Errorf("%s does not open with the vault's key: %w", name, err)
}
sealed, err := seal.Seal(value, to)
if err != nil {
return OwnAnswer{}, err
}
if err := v.recordHanding(Handing{To: caller, Name: name, Generation: r.Generation, Outcome: "handed"}); err != nil {
return OwnAnswer{}, fmt.Errorf("%s is not handed out: who was handed it could not be recorded: %w", name, err)
}
out.Credentials = append(out.Credentials, NodeCredential{CredentialAsked: c, Name: name, Generation: r.Generation, Sealed: sealed})
found = true
break
}
if !found {
out.Missing = append(out.Missing, c)
}
}
v.log("handed %s's node-engine the %d own secrets of its modules and %d credentials (%d not held), sealed to its key",
node, len(out.Own), len(out.Credentials), len(out.Missing))
return out, nil
}
// handed says the plan of module on node hands it to the provisioner, so its secrets are filled from `get`, not composed.
// A plan that cannot be read is an error, never "not handed": `own` refuses rather than hands a secret the provisioner
// holds.
func (v *Vault) handed(node, module string) (bool, error) {
read, why := v.assignmentReader(node)
if why != "" {
return false, errors.New(why)
}
raw, err := read(node + "." + module)
if err != nil {
return false, fmt.Errorf(notYetPrefix+"the assignment %s.%s could not be read, so whether it is handed to the "+
"provisioner is not known: %w", node, module, err)
}
if raw == nil {
return false, nil
}
var a plannedSecrets
if err := json.Unmarshal(raw, &a); err != nil {
return false, fmt.Errorf("the assignment %s.%s is not the controller's: %w", node, module, err)
}
return a.Provisioning != nil, nil
}
@@ -0,0 +1,121 @@
package main
import (
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/seal"
)
// Issue 1126 (ADR 0547 §2, ADR 0550 §5): `own` answers a machine's node-engine the own secrets of its modules, sealed to
// the key that machine's engine stated, about that machine alone; another caller, another machine's secrets and a
// machine with no stated key get nothing.
func TestOwnAnswersAMachineItsModulesSecretsSealedToItsKey(t *testing.T) {
plans := assignments{"anchor.web": `{"node":"anchor","module":"web"}`, "laptop.web": `{"node":"laptop","module":"web"}`}
v := newVault(t, nil)
v.Askers = Declared{Read: plans.read, Store: v.Store}
anchorPub, anchorPriv := aSealingKey(t)
laptopPub, _ := aSealingKey(t)
for _, m := range []struct {
caller string
of MoveOf
value string
}{
{"node.anchor", MoveOf{Node: "anchor", Module: "web", Name: "password", Origin: "made", Generation: "g-1", SealingKey: anchorPub}, "anchor-web-pw"},
{"node.anchor", MoveOf{Node: "anchor", Module: "web", Name: "api-key", Origin: "accepted", Generation: "g-1", SealingKey: anchorPub}, "anchor-web-key"},
{"node.laptop", MoveOf{Node: "laptop", Module: "web", Name: "password", Origin: "made", Generation: "g-1", SealingKey: laptopPub}, "laptop-web-pw"},
} {
if _, err := moveIn(t, v, m.caller, m.of, m.value); err != nil {
t.Fatal(err)
}
}
got, err := v.OwnOfNode("node.anchor")
if err != nil {
t.Fatal(err)
}
if got.Node != "anchor" || got.Key != anchorPub || len(got.Own) != 2 {
t.Fatalf("anchor was answered %+v", got)
}
want := map[string][2]string{"api-key": {"accepted", "anchor-web-key"}, "password": {"made", "anchor-web-pw"}}
for _, s := range got.Own {
w, ok := want[s.Name]
if !ok || s.Module != "web" || s.Origin != w[0] || s.Generation < 1 {
t.Errorf("anchor's %s: %+v", s.Name, s)
continue
}
value, err := seal.Open(s.Sealed, anchorPriv)
if err != nil || string(value) != w[1] {
t.Errorf("anchor's %s does not open to its value with anchor's key: %q, %v", s.Name, value, err)
}
}
for caller, says := range map[string]string{
"controller": "is not a node-engine",
"anchor.web": "is not a node-engine",
"": "an unproven caller",
"node.unknown": "no secret named own/unknown yet",
} {
if _, err := v.OwnOfNode(caller); err == nil || !strings.Contains(err.Error(), says) {
t.Errorf("%q: %v, want a refusal saying %q", caller, err, says)
}
}
if laptop, err := v.OwnOfNode("node.laptop"); err != nil || len(laptop.Own) != 1 || laptop.Own[0].Name != "password" {
t.Errorf("laptop was answered %+v, %v; want its one secret alone", laptop, err)
}
// A pair credential: laptop's web consumes anchor's postgres. Each end is answered the one value sealed to its own key;
// a credential the vault holds none of is named missing; a machine at neither end is refused.
if _, err := moveIn(t, v, "node.laptop", MoveOf{Provider: "anchor", Provision: "postgres", Consumer: "laptop",
ConsumerModule: "web", Origin: "made", Generation: "g-1", SealingKey: laptopPub}, "pg-pair"); err != nil {
t.Fatal(err)
}
pair := CredentialAsked{Provision: "postgres", Consumer: "laptop", ConsumerModule: "web", Provider: "anchor"}
asConsumer, asProvider := pair, pair
asConsumer.Role, asProvider.Role = "consumer", "provider"
none := CredentialAsked{Role: "consumer", Provision: "redis", Consumer: "laptop", ConsumerModule: "web", Provider: "anchor"}
laptop, err := v.OwnOfNode("node.laptop", asConsumer, none)
if err != nil || len(laptop.Credentials) != 1 || len(laptop.Missing) != 1 || laptop.Missing[0] != none {
t.Fatalf("laptop as consumer: %+v, %v", laptop, err)
}
if !strings.HasPrefix(laptop.Credentials[0].Name, "pair/anchor/postgres/laptop/web") {
t.Errorf("laptop's credential is kept as %s", laptop.Credentials[0].Name)
}
anchor, err := v.OwnOfNode("node.anchor", asProvider)
if err != nil || len(anchor.Credentials) != 1 {
t.Fatalf("anchor as provider: %+v, %v", anchor, err)
}
if value, err := seal.Open(anchor.Credentials[0].Sealed, anchorPriv); err != nil || string(value) != "pg-pair" {
t.Errorf("anchor's copy does not open to the value with its key: %q, %v", value, err)
}
for who, c := range map[string]CredentialAsked{"node.anchor": asConsumer, "node.laptop": asProvider} {
if _, err := v.OwnOfNode(who, c); err == nil || !strings.Contains(err.Error(), "is not that machine") {
t.Errorf("%s asking %+v: %v", who, c, err)
}
}
// Review of L2 (C1, C2): `own` fails closed. A provision's one shared value is handed only to a consumer the record
// of its holders names, never to a machine that names its provider; assignments that cannot be read, or a vault
// that reads none, refuse rather than hand.
if _, err := v.keep("provision/anchor/redis", "shared-redis", "test", "", OriginMade); err != nil {
t.Fatal(err)
}
if err := v.Store.WriteShared(Shared{Name: "provision/anchor/redis", Provider: "anchor", Provision: "redis",
Consumers: []string{"anchor/web"}}); err != nil {
t.Fatal(err)
}
if got, err := v.OwnOfNode("node.laptop", none); err != nil || len(got.Credentials) != 0 || len(got.Missing) != 1 {
t.Errorf("laptop, whose web the shared value's record does not name, got %+v, %v", got, err)
}
anchorRedis := CredentialAsked{Role: "consumer", Provision: "redis", Consumer: "anchor", ConsumerModule: "web", Provider: "anchor"}
if got, err := v.OwnOfNode("node.anchor", anchorRedis); err != nil || len(got.Credentials) != 1 {
t.Errorf("anchor's web, a holder the record names, got %+v, %v", got, err)
}
v.Askers = Declared{Read: func(string) ([]byte, error) { return nil, errors.New("the bucket did not answer") }, Store: v.Store}
if _, err := v.OwnOfNode("node.anchor"); err == nil || !strings.Contains(err.Error(), "could not be read") {
t.Errorf("own with the assignments unread: %v", err)
}
v.Askers = nil
if _, err := v.OwnOfNode("node.anchor", anchorRedis); err == nil || !strings.Contains(err.Error(), "cannot read the assignments") {
t.Errorf("own with no reader of the assignments: %v", err)
}
}
@@ -251,6 +251,34 @@ func definitions(env *Env) []toolDef {
// clause 9, adopt.go): a public half, never the private one.
return map[string]any{"recorded": true, "vault_key": v.Own.PublicKey}, nil
}},
{Name: Seat + ".own",
Description: "The own secrets of this machine's modules, each sealed to the machine's sealing key (novox/hq ADR 0547 §2, " +
"ADR 0550 §5, issue 1126): from the node-engine of the machine alone, about its own machine (the bus proves which), " +
"for the compositions it makes itself. Answers the machine, the key each value is sealed to, and per module and " +
"name the origin, the generation and the sealed value; a module handed to the provisioner is left out. With credentials, " +
"also each pair credential the machine consumes or grants, sealed the same way. Never a value in the clear.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"credentials": map[string]any{"type": "array", "description": "the pair credentials the machine's composition " +
"reads, each {role: consumer|provider, provision, consumer, consumer_module, provider, local}: answered " +
"sealed to the machine's key when the machine is that consumer's or that provider, and named missing when " +
"the vault holds none", "items": map[string]any{"type": "object"}}}},
run: func(caller string, a map[string]any) (any, error) {
v, err := env.ready()
if err != nil {
return nil, err
}
var asked []CredentialAsked
if raw, ok := a["credentials"]; ok && raw != nil {
b, err := json.Marshal(raw)
if err == nil {
err = json.Unmarshal(b, &asked)
}
if err != nil {
return nil, fmt.Errorf("credentials is a list of {role, provision, consumer, consumer_module, provider, local}: %w", err)
}
}
return v.OwnOfNode(caller, asked...)
}},
{Name: Seat + ".move-asks",
Description: "Copy rows of the controller's record of asks at a desk into the vault's, as they are and once " +
"(design 57's V3): from the controller alone, at most 500 at a time. No value is in any of them.",
+2 -1
View File
@@ -177,7 +177,8 @@
"rotate",
"move-in",
"node-key",
"move-asks"
"move-asks",
"own"
]
}
]
+5 -1
View File
@@ -1 +1,5 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# brother's store.json, against the object store's.
modules/brother/files/store.json
@@ -1 +1,6 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of uplink-tools in both modules that hold the uplink.
modules/networkmanager/cmd/uplink-tools/*.go
modules/systemd-networkd/cmd/uplink-tools/*.go
@@ -1 +1,25 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of desktop.go and its tests in every module that carries one.
modules/blueman/cmd/blueman-tools/desktop.go
modules/blueman/cmd/blueman-tools/desktop_test.go
modules/blueman/cmd/blueman-tools/copies_test.go
modules/forticlient/cmd/forticlient-tools/desktop.go
modules/forticlient/cmd/forticlient-tools/desktop_test.go
modules/forticlient/cmd/forticlient-tools/copies_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/copies_test.go
modules/nm-applet/cmd/nm-applet-tools/desktop.go
modules/nm-applet/cmd/nm-applet-tools/desktop_test.go
modules/nm-applet/cmd/nm-applet-tools/copies_test.go
modules/openrazer/cmd/openrazer-tools/desktop.go
modules/openrazer/cmd/openrazer-tools/desktop_test.go
modules/openrazer/cmd/openrazer-tools/copies_test.go
modules/polychromatic/cmd/polychromatic-tools/desktop.go
modules/polychromatic/cmd/polychromatic-tools/desktop_test.go
modules/polychromatic/cmd/polychromatic-tools/copies_test.go
modules/slack/cmd/slack-tools/desktop.go
modules/slack/cmd/slack-tools/desktop_test.go
modules/slack/cmd/slack-tools/copies_test.go
@@ -1 +1,25 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of desktop.go and its tests in every module that carries one.
modules/blueman/cmd/blueman-tools/desktop.go
modules/blueman/cmd/blueman-tools/desktop_test.go
modules/blueman/cmd/blueman-tools/copies_test.go
modules/forticlient/cmd/forticlient-tools/desktop.go
modules/forticlient/cmd/forticlient-tools/desktop_test.go
modules/forticlient/cmd/forticlient-tools/copies_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/copies_test.go
modules/nm-applet/cmd/nm-applet-tools/desktop.go
modules/nm-applet/cmd/nm-applet-tools/desktop_test.go
modules/nm-applet/cmd/nm-applet-tools/copies_test.go
modules/openrazer/cmd/openrazer-tools/desktop.go
modules/openrazer/cmd/openrazer-tools/desktop_test.go
modules/openrazer/cmd/openrazer-tools/copies_test.go
modules/polychromatic/cmd/polychromatic-tools/desktop.go
modules/polychromatic/cmd/polychromatic-tools/desktop_test.go
modules/polychromatic/cmd/polychromatic-tools/copies_test.go
modules/slack/cmd/slack-tools/desktop.go
modules/slack/cmd/slack-tools/desktop_test.go
modules/slack/cmd/slack-tools/copies_test.go
@@ -1 +1,25 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of desktop.go and its tests in every module that carries one.
modules/blueman/cmd/blueman-tools/desktop.go
modules/blueman/cmd/blueman-tools/desktop_test.go
modules/blueman/cmd/blueman-tools/copies_test.go
modules/forticlient/cmd/forticlient-tools/desktop.go
modules/forticlient/cmd/forticlient-tools/desktop_test.go
modules/forticlient/cmd/forticlient-tools/copies_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/copies_test.go
modules/nm-applet/cmd/nm-applet-tools/desktop.go
modules/nm-applet/cmd/nm-applet-tools/desktop_test.go
modules/nm-applet/cmd/nm-applet-tools/copies_test.go
modules/openrazer/cmd/openrazer-tools/desktop.go
modules/openrazer/cmd/openrazer-tools/desktop_test.go
modules/openrazer/cmd/openrazer-tools/copies_test.go
modules/polychromatic/cmd/polychromatic-tools/desktop.go
modules/polychromatic/cmd/polychromatic-tools/desktop_test.go
modules/polychromatic/cmd/polychromatic-tools/copies_test.go
modules/slack/cmd/slack-tools/desktop.go
modules/slack/cmd/slack-tools/desktop_test.go
modules/slack/cmd/slack-tools/copies_test.go
@@ -1 +1,5 @@
These tests read the repository's root predefined.json, outside this module (issue 1935): never kept.
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the root's predefined keys.
predefined.json
@@ -1 +1,25 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of desktop.go and its tests in every module that carries one.
modules/blueman/cmd/blueman-tools/desktop.go
modules/blueman/cmd/blueman-tools/desktop_test.go
modules/blueman/cmd/blueman-tools/copies_test.go
modules/forticlient/cmd/forticlient-tools/desktop.go
modules/forticlient/cmd/forticlient-tools/desktop_test.go
modules/forticlient/cmd/forticlient-tools/copies_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/copies_test.go
modules/nm-applet/cmd/nm-applet-tools/desktop.go
modules/nm-applet/cmd/nm-applet-tools/desktop_test.go
modules/nm-applet/cmd/nm-applet-tools/copies_test.go
modules/openrazer/cmd/openrazer-tools/desktop.go
modules/openrazer/cmd/openrazer-tools/desktop_test.go
modules/openrazer/cmd/openrazer-tools/copies_test.go
modules/polychromatic/cmd/polychromatic-tools/desktop.go
modules/polychromatic/cmd/polychromatic-tools/desktop_test.go
modules/polychromatic/cmd/polychromatic-tools/copies_test.go
modules/slack/cmd/slack-tools/desktop.go
modules/slack/cmd/slack-tools/desktop_test.go
modules/slack/cmd/slack-tools/copies_test.go
@@ -1 +1,5 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# every manifest.
modules/*/module.json
+25 -1
View File
@@ -1 +1,25 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of desktop.go and its tests in every module that carries one.
modules/blueman/cmd/blueman-tools/desktop.go
modules/blueman/cmd/blueman-tools/desktop_test.go
modules/blueman/cmd/blueman-tools/copies_test.go
modules/forticlient/cmd/forticlient-tools/desktop.go
modules/forticlient/cmd/forticlient-tools/desktop_test.go
modules/forticlient/cmd/forticlient-tools/copies_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop.go
modules/nextcloud-client/cmd/nextcloud-client-tools/desktop_test.go
modules/nextcloud-client/cmd/nextcloud-client-tools/copies_test.go
modules/nm-applet/cmd/nm-applet-tools/desktop.go
modules/nm-applet/cmd/nm-applet-tools/desktop_test.go
modules/nm-applet/cmd/nm-applet-tools/copies_test.go
modules/openrazer/cmd/openrazer-tools/desktop.go
modules/openrazer/cmd/openrazer-tools/desktop_test.go
modules/openrazer/cmd/openrazer-tools/copies_test.go
modules/polychromatic/cmd/polychromatic-tools/desktop.go
modules/polychromatic/cmd/polychromatic-tools/desktop_test.go
modules/polychromatic/cmd/polychromatic-tools/copies_test.go
modules/slack/cmd/slack-tools/desktop.go
modules/slack/cmd/slack-tools/desktop_test.go
modules/slack/cmd/slack-tools/copies_test.go
@@ -1 +1,6 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of uplink-tools in both modules that hold the uplink.
modules/networkmanager/cmd/uplink-tools/*.go
modules/systemd-networkd/cmd/uplink-tools/*.go
+9 -1
View File
@@ -1 +1,9 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of this package in every module that carries one.
modules/xorg/internal/desktop/*.go
modules/lemurs/internal/desktop/*.go
modules/i3/internal/desktop/*.go
modules/xterm/internal/desktop/*.go
modules/adwaita/internal/desktop/*.go
+9 -1
View File
@@ -1 +1,9 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
# What these tests read outside their package, as paths from the repository's root (a glob, or a directory read whole;
# ? before a path whose absence is read). mesh-build-go hashes them into the package's digest, so its verdict is kept
# until one of them changes (issue 1936). An empty list, or a line matching nothing, keeps nothing.
# the copies of this package in every module that carries one.
modules/xorg/internal/desktop/*.go
modules/lemurs/internal/desktop/*.go
modules/i3/internal/desktop/*.go
modules/xterm/internal/desktop/*.go
modules/adwaita/internal/desktop/*.go