Compare commits

..
Author SHA1 Message Date
jschoubben 78364e318b Put the module trees of a merge into go/buildstep for both fetch steps, and let a preview name the conflicts of a head that does not merge (issues 1675, 1676, 1445)
mesh/repo-check head 78364e318bdc merged into trunk 07a65927c362 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery-group group feat/the-checker rejected: composed together: 1 problem(s) the change brings; the first: mesh-delivery-checker reads "mesh-controller…
mesh/merge-gate head 78364e318bdc merged into trunk 07a65927c362 — fail: group feat/the-checker composed: 1 problem(s) the change brings; the first: mesh-d…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 21:21:48 +02:00
jschoubben afbe371003 Merge main (mesh-sdk #62, issue 1756) into feat/r4c5-sdk 2026-10-21 20:59:47 +02:00
jschoubben 9aaa5c80ac Accept the gate's recheck status mesh/repo-check among the owed contexts, so a pinned recheck is not refused (issue 1673) 2026-10-21 19:35:14 +02:00
mesh-tools 07a65927c3 Merge pull request 'Count an item's text as a placement of a config value (issue 1658)' (#62) from fix/1658-item-placement into main 2026-10-21 17:34:36 +00:00
jschoubben 92fdc8c86a Parse the beside catalogue once per test binary, so mesh-sdk's unit level fits its bound (issue 1756)
mesh/merge-gate head 92fdc8c86a17 merged into trunk 725590e06e75 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 92fdc8c86a17 merged into trunk 725590e06e75 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 19:26:30 +02:00
jschoubben 0730ef6092 Keep the unit level cheap: the merge's real-git tests start processes, so they are integration tests; the places tests wait 10 ms, not 50 (issue 895) 2026-10-21 19:26:02 +02:00
jschoubben 4495a16608 Put the merge of a head onto a trunk commit into go/buildstep, so both build modules' fetch steps call one function and none copies the controller's (issue 1674) 2026-10-21 19:22:33 +02:00
jschoubben b944d07158 Add the checker's owed bucket, the mesh/check status context and the previews layout, and correct the checker header: the checker decides the status, the bridge sets it (issue 1673) 2026-10-21 19:21:22 +02:00
jschoubben 6d0c2076a3 Give the checker's verdict a skipped count, a composition and the over-bound levels, so the line never hides skips or claims a composition not run (issue 1672) 2026-10-21 19:19:14 +02:00
jschoubben 4375ed8b2c Move the builder's shared build-run parts into go/delivery/buildrun, so the checker shares them and no module keeps a copy (issue 1671) 2026-10-21 19:18:11 +02:00
jschoubben c0bbff3b4b Merge remote-tracking branch 'origin/main' into fix/1658-item-placement
mesh/merge-gate head c0bbff3b4bb2 merged into trunk 725590e06e75 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head c0bbff3b4bb2 merged into trunk 725590e06e75 — fail: merge-check.sh (typescript) passed; its go/merge-check.sh failed: make them cheap …
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 19:17:26 +02:00
mesh-tools 725590e06e Merge pull request 'Give each node a declaration-files bucket of its own, so a grant names it and never needs a wildcard (issue 1694, ADR 0488)' (#64) from fix/1694-bucket-per-node into main 2026-10-21 15:39:08 +00:00
jschoubben 5792c3b959 Keep the unit level within its bound: the counter test counts 400 not 2000, the deadline test waits 50 ms not 1 s (issue 895)
mesh/merge-gate head 5792c3b959d5 merged into trunk d4deecdb380b — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 5792c3b959d5 merged into trunk d4deecdb380b — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
Both still prove what they did: the counter test still collides every run and fails when an increment
writes without its revision; the deadline test fails when the read after a failed write takes a later
deadline than the ask's.
2026-10-21 16:57:31 +02:00
mesh-tools d4deecdb38 Merge pull request 'Add upload and download to node-files' verbs, so mesh-files may serve them (issue 1718)' (#63) from feat/files-through-the-store-files-verbs into main 2026-10-21 14:40:48 +00:00
jschoubben 216d2b4411 Check every mesh node's bucket and stream name, not only one (issue 1694, ADR 0488 §4)
mesh/merge-gate head 216d2b441132 merged into trunk d4deecdb380b — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 216d2b441132 merged into trunk d4deecdb380b — fail: merge-check.sh (typescript) passed; its go/merge-check.sh failed: make them cheap …
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 16:03:55 +02:00
jschoubben 86f9a1a5c5 Cite ADR 0488 in the per-node bucket's doc comments (issue 1694) 2026-10-21 15:48:22 +02:00
jschoubben 36d9eba51d Each node has an object bucket of its own for files by digest (issue 1694, ADR 0486) 2026-10-21 15:39:38 +02:00
jschoubben 90b20f5a3b Add upload and download to node-files' verbs, as ADR 0468 §9 decides, so mesh-files may serve them (issue 1718)
mesh/merge-gate head 90b20f5a3bda merged into trunk 85b5a2687e6a — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 90b20f5a3bda merged into trunk 85b5a2687e6a — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 15:21:16 +02:00
mesh-tools 85b5a2687e Merge pull request 'Hold a module whose container names groups off a node that does not report container-groups (issues 1637, 1638, R4d)' (#60) from feat/r4d-groups-capability into main 2026-10-21 13:09:58 +00:00
jschoubben b5d8eb03ce A module whose container names groups needs the container-groups capability, so it is held, naming it, on an engine that does not read them (issue 1638, ADR 0480 clause 7)
mesh/merge-gate head b5d8eb03cea3 merged into trunk 863a981e014e — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head b5d8eb03cea3 merged into trunk 863a981e014e — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-sdk@90b20f5a3bda (merged as d4deecdb into main, walk plan-179259365188251…
2026-10-21 15:03:32 +02:00
mesh-tools 863a981e01 Merge pull request 'Let a declared file travel by its sha256 digest instead of its content, so a node's declaration is no longer bounded by one bus message (issue 1694, ADR 0486 step 1)' (#59) from fix/1694-file-references into main 2026-10-21 12:43:57 +00:00
mesh-tools b68fc75589 Merge pull request 'Let a container name supplementary groups, from a number or a trusted config key, so n8n can join the media group without an image of its own (issue 1636, R4d)' (#58) from feat/r4d-groups into main 2026-10-21 12:30:02 +00:00
jschoubben 6c857e1c02 A bytes file of exactly 64 KiB stays inline: take its exact decoded length (issue 1694)
mesh/merge-gate head 6c857e1c025f merged into trunk b68fc755894c — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 6c857e1c025f merged into trunk b68fc755894c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivering: 0 machine step(s) passed
2026-10-21 14:23:48 +02:00
jschoubben f8a9420952 A declaration's file can be a reference by digest: layout, bucket and capability names, helpers (issues 1694, ADR 0486) 2026-10-21 14:12:19 +02:00
jschoubben f0f66d5b8d A container may add supplementary groups, and a group may be a config value filled at composition (issue 1636, ADR 0480 clause 7)
mesh/merge-gate head f0f66d5b8d42 merged into trunk 66e8fdaa62e2 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head f0f66d5b8d42 merged into trunk 66e8fdaa62e2 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 14:01:19 +02:00
mesh-tools 66e8fdaa62 Merge pull request 'Read an ended or pending assignment as a removed key in the provider helper, on main, so the modules pinning it can pin main (issues 1098, 1663)' (#57) from fix/1663-provider-standing into main 2026-10-21 11:40:14 +00:00
mesh-tools 88375c5786 Merge pull request 'Test that a node without the go or nodejs module refuses the modules needing a toolchain, against the catalogue as it now stands (issue 1573)' (#56) from fix/1573-toolchain-resolve into main 2026-10-21 11:27:27 +00:00
jschoubben 757509366e The provider helper reads an ended or pending assignment as a removed key (issue 1098)
mesh/merge-gate head 757509366e3c merged into trunk 88375c5786d9 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 757509366e3c merged into trunk 88375c5786d9 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
(carried to main for issue 1663)
2026-10-21 13:23:14 +02:00
jschoubben c8e35097e8 Count an item's text as a placement of a config value, so a key only an item says is not refused (issue 1658)
mesh/merge-gate waits for its group's composed check
mesh/repo-check head c8e35097e861 merged into trunk d4deecdb380b beside mesh-controller#420@858b06a9 +1 more — pass: its merge-check.sh passed, each part i…
mesh/delivery-group group fix/1656-xterm-config rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 12:54:08 +02:00
jschoubben ddfa119720 Cite ADR 0478's accepted clause numbers
mesh/merge-gate head ddfa1197209b merged into trunk cb54756c3865 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head ddfa1197209b merged into trunk cb54756c3865 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 12:45:04 +02:00
jschoubben a86d607e48 Refresh the captured catalogue from mesh-catalog main, with the toolchain modules (issue 1573) 2026-10-21 12:45:01 +02:00
jschoubben 86ac7596c9 Test that a node without the go or nodejs module refuses the modules needing a toolchain, and with it composes (issue 1573) 2026-10-21 12:44:34 +02:00
mesh-tools cb54756c38 Merge pull request 'Give delivery.Module the toolchains its fingerprint was computed with, and '.' as a root module's path, so the builder builds a key only with its own toolchain (issue 1615)' (#55) from fix/1615-module-toolchains-root into main 2026-10-21 10:02:09 +00:00
mesh-tools 89dd57bed0 Give delivery.Module the toolchains its fingerprint was computed with, and '.' as a root module's path, so the builder builds a key only with its own toolchain (issue 1615)
mesh/merge-gate head 89dd57bed05d merged into trunk 465beb4ea5e3 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 89dd57bed05d merged into trunk 465beb4ea5e3 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 09:37:14 +00:00
mesh-tools 465beb4ea5 Merge pull request 'Name a bucket's other users in the manifest, and sign and stream store links in one helper (issues 1591, 1592)' (#54) from feat/files-through-the-store into main 2026-10-21 09:36:45 +00:00
mesh-tools c9f978d330 Merge pull request 'Give delivery.Module a path and BuildRecord the artifacts a build made, so the builder can find a module and the view compare digests (issue 1588, ADR 0478)' (#53) from fix/1588-module-path-made into main 2026-10-21 09:27:12 +00:00
jschoubben efd3d224e1 Put takes the body's exact size and always sends it as Content-Length; refuse a link under one second (issue 1592)
mesh/merge-gate head efd3d224e131 merged into trunk c9f978d3308b — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head efd3d224e131 merged into trunk c9f978d3308b — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 11:22:18 +02:00
jschoubben 081dc616ff Add objectlink: one helper signs a short store link and streams a document over it with a size limit and a digest (issue 1592) 2026-10-21 11:18:15 +02:00
jschoubben b7b81e040f Carry shares on a requirement of s3-bucket to the provider's grant, so a bucket's other users are named in the manifest (issue 1591) 2026-10-21 11:15:49 +02:00
mesh-tools 66b052aeee Cite ADR 0478's clause on the module's directory by its number in the record's version 4 (issue 1588)
mesh/merge-gate head 66b052aeee69 merged into trunk 3558903ff9e7 — pass, with a note: builds mesh-sdk (+41 dependent(s)) → ace, g14, novox, shanks; no bus …
mesh/repo-check head 66b052aeee69 merged into trunk 3558903ff9e7 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-21 09:08:15 +00:00
mesh-tools ea1b2ab395 Give delivery.Module a path and BuildRecord the artifacts a build made, so the builder can find a module and compare digests (issue 1588) 2026-10-21 09:04:43 +00:00
111 changed files with 5075 additions and 236 deletions
+112
View File
@@ -0,0 +1,112 @@
package buildstep
import (
"context"
"fmt"
"strings"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// MergeResult is what MergeInto made of a head and a trunk commit: the trunk commit (Base) and the head, whole, and the
// merge commit and the paths the merge changes against the trunk commit; or, when the head does not merge cleanly, the
// files in conflict, with Merge and Changed empty.
type MergeResult struct {
Base, Head, Merge string
Conflicts []string
Changed []string
}
// MergeInto makes, in the workspace dir that FetchCommit left at commit, the merge of commit into the trunk commit
// base, as the forge's default merge would (a merge commit, never a fast-forward), and leaves the workspace at the merge
// (novox/hq ADR 0484 clause 3, ADR 0323). It is the one merge the build modules' fetch steps make, with the identity
// and flags of the controller's check merge (mesh-controller internal/builder/check_merge.go): a name nobody owns, no
// signature, no hook, nothing of the machine's git configuration that changes a merge. Nothing is pushed.
//
// A merge needs both histories, so it makes the fetch whole (the shallow clone FetchCommit made is deepened) and
// fetches base by its sha. A head that does not merge cleanly is an answer, not an error: the files in conflict, the
// merge undone and the workspace left at base. A head already in base answers a Merge equal to Base and no changed
// paths. An error is that the merge could not be tried: a base or commit that is not a full sha, no base, a workspace
// not at commit, or a fetch or git command that failed. remote and the options are those of FetchCommit; the runner's
// bound must allow a whole fetch.
func MergeInto(ctx context.Context, run forgekey.Runner, dir, remote, base, commit string, opts ...Option) (MergeResult, error) {
var o fetchOptions
for _, opt := range opts {
opt(&o)
}
res := MergeResult{Base: base, Head: commit}
if base == "" {
return res, fmt.Errorf("buildstep: no base: a merge needs the trunk commit it is made onto")
}
if err := checkFetch(o, dir, remote, base, ""); err != nil {
return res, err
}
if err := checkFetch(o, dir, remote, commit, ""); err != nil {
return res, err
}
env := []string{"GIT_TERMINAL_PROMPT=0"}
git := func(args ...string) (string, error) {
full := append([]string{"-c", "credential.helper=", "-C", dir}, args...)
r := run(ctx, env, "git", full...)
if r.Status != 0 || r.Err != "" {
return "", fmt.Errorf("git %s: %s", args[0], firstLine(r.Err+" "+r.Stderr))
}
return r.Stdout, nil
}
at, err := git("rev-parse", "HEAD")
if err != nil {
return res, err
}
if head := strings.TrimSpace(at); head != commit {
return res, fmt.Errorf("buildstep: the workspace is at %s, not at the commit %s", head, commit)
}
shallow, err := git("rev-parse", "--is-shallow-repository")
if err != nil {
return res, err
}
fetch := []string{"fetch", "--no-tags", "origin", base}
if strings.TrimSpace(shallow) == "true" {
fetch = []string{"fetch", "--no-tags", "--unshallow", "origin", base}
}
if _, err := git(fetch...); err != nil {
return res, fmt.Errorf("the trunk commit %s: %w", base, err)
}
if _, err := git("checkout", "-q", "--detach", base); err != nil {
return res, fmt.Errorf("the trunk commit %s cannot be checked out: %w", base, err)
}
message := fmt.Sprintf("Check the merge of %s into %s (made for a check, never pushed)", short(commit), short(base))
args := append(append([]string{}, mergeIdentity...), "merge", "--no-ff", "--no-edit", "--no-verify", "--quiet", "-m", message, commit)
if _, err := git(args...); err != nil {
out, _ := git("diff", "--name-only", "--diff-filter=U")
res.Conflicts = strings.Fields(out)
_, _ = git("merge", "--abort")
if len(res.Conflicts) == 0 {
return res, fmt.Errorf("the head %s could not be merged into the trunk commit %s: %w", short(commit), short(base), err)
}
return res, nil
}
merge, err := git("rev-parse", "HEAD")
if err != nil {
return res, fmt.Errorf("the merge cannot be read: %w", err)
}
res.Merge = strings.TrimSpace(merge)
changed, err := git("diff", "--name-only", "--no-renames", base, res.Merge)
if err != nil {
return res, fmt.Errorf("the paths the merge changes cannot be read: %w", err)
}
res.Changed = strings.Fields(changed)
return res, nil
}
// mergeIdentity is who a check's merge is made as: it is never pushed, so it names nobody's account. No hook of the
// clone's or the build machine's git configuration runs, and no setting there changes the merge (the controller's
// check merge, internal/builder/check_merge.go).
var mergeIdentity = []string{"-c", "user.name=mesh merge check", "-c", "user.email=merge-check@mesh.invalid",
"-c", "commit.gpgsign=false", "-c", "core.hooksPath=/dev/null", "-c", "merge.verifySignatures=false"}
func short(commit string) string {
if len(commit) > 12 {
return commit[:12]
}
return commit
}
+172
View File
@@ -0,0 +1,172 @@
//go:build integration
package buildstep
import (
"context"
"os"
"os/exec"
"path/filepath"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// The merge on real git repositories starts git processes, so it is an integration test (the unit level keeps its bound,
// issue 895); the unit tests of merge_test.go prove the commands and the refusals with a fake runner.
func realGit(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", append([]string{"-C", dir, "-c", "user.name=t", "-c", "user.email=t@example.invalid", "-c", "commit.gpgsign=false"}, args...)...)
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return strings.TrimSpace(string(out))
}
func writeFile(t *testing.T, dir, name, content string) {
t.Helper()
p := filepath.Join(dir, name)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// mergeRig is a local bare repository standing in for the forge, a work repository that pushes to it, and an empty
// workspace: the fetch step's shape (FetchCommit of the head into the workspace, then MergeInto).
type mergeRig struct {
t *testing.T
bare, work, ws string
fork, trunk string // the commit both histories share, and the trunk's head
remote string
}
func newMergeRig(t *testing.T) *mergeRig {
root := t.TempDir()
r := &mergeRig{t: t, bare: filepath.Join(root, "r.git"), work: filepath.Join(root, "work"), ws: filepath.Join(root, "ws")}
for _, d := range []string{r.bare, r.work, r.ws} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
r.remote = "file://" + r.bare
realGit(t, r.bare, "init", "-q", "--bare")
realGit(t, r.work, "init", "-q", "-b", "main")
writeFile(t, r.work, "a.txt", "one\ntwo\nthree\n")
writeFile(t, r.work, "modules/m/module.json", "{}\n")
realGit(t, r.work, "add", ".")
realGit(t, r.work, "commit", "-q", "-m", "fork point")
r.fork = realGit(t, r.work, "rev-parse", "HEAD")
return r
}
// head makes a commit on a branch off the fork point, pushes it as the pull request's head ref and answers its sha.
func (r *mergeRig) head(files map[string]string) string {
realGit(r.t, r.work, "checkout", "-q", "-B", "pr", r.fork)
for n, c := range files {
writeFile(r.t, r.work, n, c)
}
realGit(r.t, r.work, "add", ".")
realGit(r.t, r.work, "commit", "-q", "-m", "head")
h := realGit(r.t, r.work, "rev-parse", "HEAD")
realGit(r.t, r.work, "push", "-q", r.bare, "HEAD:refs/pull/1/head")
return h
}
// advance moves the trunk past the fork point with a commit changing files, pushes it and answers its sha.
func (r *mergeRig) advance(files map[string]string) string {
realGit(r.t, r.work, "checkout", "-q", "-B", "main", r.fork)
for n, c := range files {
writeFile(r.t, r.work, n, c)
}
realGit(r.t, r.work, "add", ".")
realGit(r.t, r.work, "commit", "-q", "-m", "trunk moves")
r.trunk = realGit(r.t, r.work, "rev-parse", "HEAD")
realGit(r.t, r.work, "push", "-q", "-f", r.bare, "HEAD:refs/heads/main")
return r.trunk
}
func (r *mergeRig) fetch(head string) {
if err := FetchCommit(context.Background(), forgekey.ExecRunner, r.ws, r.remote, head, "refs/pull/1/head", AllowFileRemote()); err != nil {
r.t.Fatal(err)
}
}
// 1674/AC1: a clean merge answers the merge commit and the changed files, and leaves the workspace at the merge, which
// holds both sides' changes.
func TestMergeIntoACleanMergeAnswersTheMergeCommitAndTheChangedFiles(t *testing.T) {
r := newMergeRig(t)
head := r.head(map[string]string{"b.txt": "head\n"})
base := r.advance(map[string]string{"a.txt": "one\ntwo\nthree\nfour\n"})
r.fetch(head)
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote())
if err != nil {
t.Fatal(err)
}
if res.Base != base || res.Head != head || len(res.Merge) != 40 || res.Merge == base || res.Merge == head || len(res.Conflicts) != 0 {
t.Fatalf("result = %+v", res)
}
if !reflect.DeepEqual(res.Changed, []string{"b.txt"}) {
t.Errorf("changed = %v, want only what the head adds to the trunk: b.txt", res.Changed)
}
if at := realGit(t, r.ws, "rev-parse", "HEAD"); at != res.Merge {
t.Errorf("the workspace is at %s, not at the merge %s", at, res.Merge)
}
for f, want := range map[string]string{"b.txt": "head\n", "a.txt": "one\ntwo\nthree\nfour\n"} {
if got, _ := os.ReadFile(filepath.Join(r.ws, f)); string(got) != want {
t.Errorf("%s = %q, want %q", f, got, want)
}
}
if parents := strings.Fields(realGit(t, r.ws, "rev-list", "--parents", "-n", "1", res.Merge)); len(parents) != 3 || parents[1] != base || parents[2] != head {
t.Errorf("the merge commit's parents = %v, want the trunk commit and the head", parents)
}
if author := realGit(t, r.ws, "log", "-1", "--format=%an <%ae>", res.Merge); author != "mesh merge check <merge-check@mesh.invalid>" {
t.Errorf("the merge is made as %q", author)
}
}
// 1674/AC2: a conflict is an answer: the files in conflict, no merge commit, and the workspace left at the trunk commit.
func TestMergeIntoAConflictAnswersTheFilesAndLeavesTheTreeAtTheTrunk(t *testing.T) {
r := newMergeRig(t)
head := r.head(map[string]string{"a.txt": "one\nHEAD\nthree\n", "c.txt": "c\n"})
base := r.advance(map[string]string{"a.txt": "one\nTRUNK\nthree\n"})
r.fetch(head)
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote())
if err != nil {
t.Fatalf("a conflict is an answer, not an error: %v", err)
}
if !reflect.DeepEqual(res.Conflicts, []string{"a.txt"}) || res.Merge != "" || len(res.Changed) != 0 || res.Base != base || res.Head != head {
t.Fatalf("result = %+v", res)
}
if at := realGit(t, r.ws, "rev-parse", "HEAD"); at != base {
t.Errorf("the workspace is at %s, not at the trunk commit %s", at, base)
}
if got, _ := os.ReadFile(filepath.Join(r.ws, "a.txt")); string(got) != "one\nTRUNK\nthree\n" {
t.Errorf("a.txt = %q, want the trunk's", got)
}
if st := realGit(t, r.ws, "status", "--porcelain"); st != "" {
t.Errorf("the tree is not clean after the aborted merge: %q", st)
}
if _, err := os.Stat(filepath.Join(r.ws, "c.txt")); err == nil {
t.Error("a file of the head is in a tree left at the trunk")
}
}
// A head already in the trunk merges into nothing: the merge is the trunk commit and no path changes.
func TestMergeIntoAHeadAlreadyInTheTrunkChangesNothing(t *testing.T) {
r := newMergeRig(t)
head := r.head(map[string]string{"b.txt": "head\n"})
r.fork = head // the trunk continues from the head
base := r.advance(map[string]string{"a.txt": "one\ntwo\nthree\nfour\n"})
r.fetch(head)
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote())
if err != nil || res.Merge != base || len(res.Changed) != 0 || len(res.Conflicts) != 0 {
t.Errorf("result = %+v, %v", res, err)
}
}
+74
View File
@@ -0,0 +1,74 @@
package buildstep
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// 1674/AC3: no base is refused before anything runs; so is a base or a commit that is not a full sha, a workspace that
// is not at the commit, and a remote that is not ssh://.
func TestMergeIntoRefusesWhatIsNotAMerge(t *testing.T) {
f := &fakeGit{}
for name, c := range map[string]struct{ remote, base, commit string }{
"no base": {"ssh://mesh-forge/o/r.git", "", sha},
"a short base": {"ssh://mesh-forge/o/r.git", "abc123", sha},
"a branch as base": {"ssh://mesh-forge/o/r.git", "main", sha},
"a short commit": {"ssh://mesh-forge/o/r.git", sha, "abc"},
"an https remote": {"https://forge.example/o/r.git", sha, sha},
} {
_, err := MergeInto(context.Background(), f.run, "/w", c.remote, c.base, c.commit)
if err == nil {
t.Errorf("%s accepted", name)
}
}
if len(f.calls) != 0 {
t.Errorf("git ran for a refused merge: %v", f.calls)
}
g := &fakeGit{rule: func(l string) forgekey.Ran {
if strings.Contains(l, "rev-parse HEAD") {
return forgekey.Ran{Stdout: strings.Repeat("f", 40) + "\n"}
}
return forgekey.Ran{}
}}
_, err := MergeInto(context.Background(), g.run, "/w", "ssh://mesh-forge/o/r.git", strings.Repeat("a", 40), sha)
if err == nil || !strings.Contains(err.Error(), "not at the commit") || len(g.calls) != 1 {
t.Errorf("a workspace at another commit: %v after %d commands", err, len(g.calls))
}
}
// 1674/AC4: the merge is made with the check merge's identity and flags, hooks off, and fetches with no credential.
func TestMergeIntoRunsTheCheckMergesCommands(t *testing.T) {
base := strings.Repeat("b", 40)
f := &fakeGit{rule: func(l string) forgekey.Ran {
switch {
case strings.HasSuffix(l, "rev-parse HEAD"):
return forgekey.Ran{Stdout: sha + "\n"}
case strings.Contains(l, "--is-shallow-repository"):
return forgekey.Ran{Stdout: "true\n"}
}
return forgekey.Ran{}
}}
if _, err := MergeInto(context.Background(), f.run, "/w", "ssh://mesh-forge/o/r.git", base, sha); err != nil {
t.Fatal(err)
}
all := strings.Join(f.calls, "\n")
for _, want := range []string{
pre + "fetch --no-tags --unshallow origin " + base,
pre + "checkout -q --detach " + base,
"user.name=mesh merge check", "user.email=merge-check@mesh.invalid", "commit.gpgsign=false", "core.hooksPath=/dev/null",
"merge --no-ff --no-edit --no-verify --quiet -m",
pre + "diff --name-only --no-renames " + base,
} {
if !strings.Contains(all, want) {
t.Errorf("the commands lack %q:\n%s", want, all)
}
}
for _, c := range f.calls {
if strings.Contains(c, "https://") || !strings.Contains(c, "credential.helper= ") {
t.Errorf("command %q carries a credential or https", c)
}
}
}
+67
View File
@@ -0,0 +1,67 @@
package buildstep
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// ModuleTree is what a merge holds of one module directory: its tree id, the sha256 of its module.json and the sha256
// of each lock file it has (go.sum, package-lock.json), from which the checker and the indexer take the module's inputs
// (novox/hq ADR 0484 clause 3).
type ModuleTree struct {
Tree string `json:"tree"`
Manifest string `json:"manifest"`
Locks map[string]string `json:"locks,omitempty"`
}
var lockFiles = []string{"go.sum", "package-lock.json"}
// ModuleTrees are the trees, in commit, of every directory of the workspace dir that holds a module.json, by directory
// (the repository's root is "."). The workspace is at commit, so the manifests and the lock files are read from it. env is
// the environment git runs with (nil for none). Both build modules' fetch steps answer it for a merge.
func ModuleTrees(ctx context.Context, run forgekey.Runner, env []string, dir, commit string) (map[string]ModuleTree, error) {
r := run(ctx, env, "git", "-C", dir, "ls-files", "--", "module.json", "*/module.json")
if r.Status != 0 || r.Err != "" {
return nil, fmt.Errorf("git ls-files: %s", firstLine(r.Err+" "+r.Stderr))
}
trees := map[string]ModuleTree{}
for _, l := range strings.Split(r.Stdout, "\n") {
if l = strings.TrimSpace(l); l == "" {
continue
}
mdir := filepath.ToSlash(filepath.Dir(l))
spec := commit + ":" + mdir
if mdir == "." {
spec = commit + "^{tree}"
}
t := run(ctx, env, "git", "-C", dir, "rev-parse", spec)
if t.Status != 0 || t.Err != "" {
return nil, fmt.Errorf("git rev-parse %s: %s", spec, firstLine(t.Err+" "+t.Stderr))
}
mt := ModuleTree{Tree: strings.TrimSpace(t.Stdout)}
raw, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(l)))
if err != nil {
return nil, err
}
mt.Manifest = sha256Hex(raw)
for _, lock := range lockFiles {
if raw, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(mdir), lock)); err == nil {
if mt.Locks == nil {
mt.Locks = map[string]string{}
}
mt.Locks[lock] = sha256Hex(raw)
}
}
trees[mdir] = mt
}
return trees, nil
}
func sha256Hex(b []byte) string { h := sha256.Sum256(b); return hex.EncodeToString(h[:]) }
+93
View File
@@ -0,0 +1,93 @@
package buildstep
import (
"context"
"crypto/sha256"
"encoding/hex"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
func hexOf(s string) string { h := sha256.Sum256([]byte(s)); return hex.EncodeToString(h[:]) }
func writeTreeFile(t *testing.T, root, name, content string) {
t.Helper()
p := filepath.Join(root, filepath.FromSlash(name))
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
// ModuleTrees answers, for each directory holding a module.json (the root as "."), the tree id git gives, the sha256 of
// module.json and of each lock file present, and reads them from the workspace.
func TestModuleTreesAnswersTreeManifestAndLocksPerModuleDirectory(t *testing.T) {
root := t.TempDir()
writeTreeFile(t, root, "module.json", `{"root":1}`)
writeTreeFile(t, root, "go.sum", "rootsum")
writeTreeFile(t, root, "modules/a/module.json", `{"a":1}`)
writeTreeFile(t, root, "modules/a/go.sum", "asum")
writeTreeFile(t, root, "modules/a/package-lock.json", "alock")
writeTreeFile(t, root, "modules/b/module.json", `{"b":1}`)
merge := strings.Repeat("c", 40)
var calls []string
var envs [][]string
run := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
calls = append(calls, name+" "+strings.Join(args[2:], " "))
envs = append(envs, env)
switch args[2] {
case "ls-files":
return forgekey.Ran{Stdout: "module.json\nmodules/a/module.json\nmodules/b/module.json\n"}
case "rev-parse":
return forgekey.Ran{Stdout: "tree-of-" + args[3] + "\n"}
}
return forgekey.Ran{Status: 1, Stderr: "unexpected"}
}
got, err := ModuleTrees(context.Background(), run, []string{"X=1"}, root, merge)
if err != nil {
t.Fatal(err)
}
want := map[string]ModuleTree{
".": {Tree: "tree-of-" + merge + "^{tree}", Manifest: hexOf(`{"root":1}`), Locks: map[string]string{"go.sum": hexOf("rootsum")}},
"modules/a": {Tree: "tree-of-" + merge + ":modules/a", Manifest: hexOf(`{"a":1}`), Locks: map[string]string{"go.sum": hexOf("asum"), "package-lock.json": hexOf("alock")}},
"modules/b": {Tree: "tree-of-" + merge + ":modules/b", Manifest: hexOf(`{"b":1}`)},
}
if !reflect.DeepEqual(got, want) {
t.Errorf("trees = %+v\nwant %+v", got, want)
}
if got["modules/b"].Locks != nil {
t.Error("a module with no lock file carries a lock map")
}
for _, e := range envs {
if len(e) != 1 || e[0] != "X=1" {
t.Errorf("git ran with env %v, want the one given", e)
}
}
}
func TestModuleTreesFailsNamingTheGitCommandOrTheUnreadableManifest(t *testing.T) {
root := t.TempDir()
fail := func(on string) forgekey.Runner {
return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
if args[2] == on {
return forgekey.Ran{Status: 128, Stderr: "fatal: no\n"}
}
return forgekey.Ran{Stdout: "modules/a/module.json\n"}
}
}
for _, on := range []string{"ls-files", "rev-parse"} {
if _, err := ModuleTrees(context.Background(), fail(on), nil, root, sha); err == nil || !strings.Contains(err.Error(), on) || !strings.Contains(err.Error(), "fatal: no") {
t.Errorf("%s failing: %v", on, err)
}
}
if _, err := ModuleTrees(context.Background(), fail("none"), nil, root, sha); err == nil {
t.Error("a module.json that is not in the workspace was read")
}
}
+124
View File
@@ -0,0 +1,124 @@
package catalogue
import (
"encoding/json"
"fmt"
"regexp"
"sort"
)
// BucketShare is one entry of `shares` on a module's requirement of `s3-bucket` (novox/hq ADR 0468 §8): who else may
// put or get in the consumer's bucket, and with which methods. Not the NFS shares of shares.go (ADR 0263).
//
// "requires": [{"name": "s3-bucket", "shares": [{"to": "seat:node-files", "methods": ["put"]},
// {"to": "print-and-scan", "methods": ["get"]}]}]
//
// `to` is a module name, or `seat:<seat name>`. `methods` is a non-empty subset of get and put, without a repeat.
// Shares are allowed on `s3-bucket` only, and no two entries name the same `to`.
type BucketShare struct {
To string `json:"to"`
Methods []string `json:"methods"`
}
// BucketRequirement is the requirement `shares` may be written on.
const BucketRequirement = "s3-bucket"
// RequireSharesKey is where a stored manifest keeps the shares of its requirements: beside `requires`, which stays a
// plain list of names, for the reason OptionalRequiresKey is beside it. An older controller sets the unknown key
// aside instead of failing the catalogue.
const RequireSharesKey = "requires-shares"
// SharesValue is the key of the value a consumer gives the provider of s3-bucket in its grant: the list of
// {to, methods} sorted by `to`. A consumer's own `contributes` may not name it.
const SharesValue = "shares"
var shareTarget = regexp.MustCompile(`^(seat:)?[a-z0-9][a-z0-9-]*$`)
// checkBucketShares refuses a malformed `shares` of one requirement, naming the entry and ADR 0468 §8.
func checkBucketShares(requirement string, shares []BucketShare) error {
if len(shares) == 0 {
return nil
}
if requirement != BucketRequirement {
return fmt.Errorf("requires %s: `shares` is allowed on %s only (novox/hq ADR 0468 §8)", requirement, BucketRequirement)
}
seen := map[string]bool{}
for i, s := range shares {
at := fmt.Sprintf("requires %s: shares[%d] (to %q)", requirement, i, s.To)
if !shareTarget.MatchString(s.To) {
return fmt.Errorf("%s: `to` is a module name or seat:<seat name> (novox/hq ADR 0468 §8)", at)
}
if seen[s.To] {
return fmt.Errorf("%s: a second entry for the same `to` (novox/hq ADR 0468 §8)", at)
}
seen[s.To] = true
if len(s.Methods) == 0 {
return fmt.Errorf("%s: `methods` is empty; name get, put or both (novox/hq ADR 0468 §8)", at)
}
used := map[string]bool{}
for _, method := range s.Methods {
if method != "get" && method != "put" {
return fmt.Errorf("%s: method %q is not get or put (novox/hq ADR 0468 §8)", at, method)
}
if used[method] {
return fmt.Errorf("%s: method %q twice (novox/hq ADR 0468 §8)", at, method)
}
used[method] = true
}
}
return nil
}
// readRequireShares reads the stored form's side key.
func readRequireShares(raw json.RawMessage) (map[string][]BucketShare, error) {
var out map[string][]BucketShare
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("%s: an object of requirement to a list of {to, methods}: %w", RequireSharesKey, err)
}
for requirement, shares := range out {
if err := checkBucketShares(requirement, shares); err != nil {
return nil, err
}
}
return out, nil
}
// sortedShares is a requirement's shares as they are handed to the provider: sorted by `to` (a file whose lines move
// looks changed), each entry's methods sorted, and a copy so the manifest is not reordered.
func sortedShares(shares []BucketShare) []BucketShare {
out := make([]BucketShare, 0, len(shares))
for _, s := range shares {
methods := append([]string(nil), s.Methods...)
sort.Strings(methods)
out = append(out, BucketShare{To: s.To, Methods: methods})
}
sort.Slice(out, func(i, j int) bool { return out[i].To < out[j].To })
return out
}
// withShares adds this module's shares of the requirement to the values it gives the provider. A module whose own
// `contributes` already names `shares` is refused: the field is the only way to say it (ADR 0468 §8).
func withShares(m Manifest, requirement string, values map[string]any) (map[string]any, error) {
shares := m.RequireShares[requirement]
if _, named := values[SharesValue]; named && requirement == BucketRequirement {
return nil, fmt.Errorf("%s contributing to %s names `shares`; write it on its requirement instead (novox/hq ADR 0468 §8)",
m.Module, requirement)
}
if len(shares) == 0 {
return values, nil
}
out := make(map[string]any, len(values)+1)
for k, v := range values {
out[k] = v
}
listed := []any{}
for _, s := range sortedShares(shares) {
methods := []any{}
for _, method := range s.Methods {
methods = append(methods, method)
}
listed = append(listed, map[string]any{"to": s.To, "methods": methods})
}
out[SharesValue] = listed
return out, nil
}
+185
View File
@@ -0,0 +1,185 @@
package catalogue
import (
"encoding/json"
"reflect"
"strings"
"testing"
)
// Issue 1591: `shares` on a requirement of s3-bucket (novox/hq ADR 0468 §8).
const filesWithShares = `{"module":"files","version":"1","requires":["shell",
{"name":"s3-bucket","shares":[{"to":"seat:node-files","methods":["put"]},{"to":"print-and-scan","methods":["get","put"]}]}]}`
func minioLike() Manifest {
return Manifest{Module: "minio", Version: "1", Provides: Offers("s3-bucket"),
Receives: map[string]string{"s3-bucket": "/grants/mesh.json"}}
}
func givenToMinio(t *testing.T, out []map[string]any) []Contribution {
t.Helper()
for _, r := range out {
if r["path"] != "/grants/mesh.json" {
continue
}
var parsed struct {
Given []Contribution `json:"given"`
}
if err := json.Unmarshal([]byte(r["content"].(string)), &parsed); err != nil {
t.Fatal(err)
}
return parsed.Given
}
t.Fatalf("minio was given no file: %v", out)
return nil
}
func sharesOf(t *testing.T, values map[string]any) string {
t.Helper()
raw, err := json.Marshal(values["shares"])
if err != nil {
t.Fatal(err)
}
return string(raw)
}
const sortedSharesJSON = `[{"methods":["get","put"],"to":"print-and-scan"},{"methods":["put"],"to":"seat:node-files"}]`
// 1591/AC1: parsed, the shares are on the manifest; stored, `requires` is plain names and the side key holds them.
func TestSharesParseAndAreStoredBesideRequires(t *testing.T) {
m := mustParse(t, filesWithShares)
want := []BucketShare{{To: "seat:node-files", Methods: []string{"put"}}, {To: "print-and-scan", Methods: []string{"get", "put"}}}
if !reflect.DeepEqual(m.RequireShares["s3-bucket"], want) {
t.Fatalf("the shares read as %+v", m.RequireShares)
}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
t.Fatal(err)
}
if string(keys["requires"]) != `["shell","s3-bucket"]` {
t.Errorf("the stored requires is not plain names: %s", keys["requires"])
}
if !strings.Contains(string(keys[RequireSharesKey]), `"seat:node-files"`) {
t.Errorf("the side key does not hold the shares: %s", raw)
}
var again Manifest
if err := json.Unmarshal(raw, &again); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(again, m) {
t.Fatalf("not lossless:\n%+v\n%+v", again, m)
}
}
// 1591/AC2: a malformed share is refused, naming the entry and ADR 0468 §8.
func TestAMalformedShareIsRefused(t *testing.T) {
cases := map[string]string{
"unknown method": `{"name":"s3-bucket","shares":[{"to":"a","methods":["delete"]}]}`,
"empty to": `{"name":"s3-bucket","shares":[{"to":"","methods":["get"]}]}`,
"malformed to": `{"name":"s3-bucket","shares":[{"to":"Seat:X y","methods":["get"]}]}`,
"empty methods": `{"name":"s3-bucket","shares":[{"to":"a","methods":[]}]}`,
"duplicate to": `{"name":"s3-bucket","shares":[{"to":"a","methods":["get"]},{"to":"a","methods":["put"]}]}`,
"duplicate verb": `{"name":"s3-bucket","shares":[{"to":"a","methods":["get","get"]}]}`,
"other require": `{"name":"route","shares":[{"to":"a","methods":["get"]}]}`,
}
for name, entry := range cases {
_, err := ParseManifest([]byte(`{"module":"files","version":"1","requires":[` + entry + `]}`))
if err == nil {
t.Errorf("%s: accepted", name)
continue
}
if !strings.Contains(err.Error(), "ADR 0468 §8") || !strings.Contains(err.Error(), "requires") {
t.Errorf("%s: the error does not name the entry and the ADR: %v", name, err)
}
}
// The stored form is held to the same rules.
_, err := ParseManifest([]byte(`{"module":"files","version":"1","requires":["s3-bucket"],
"requires-shares":{"s3-bucket":[{"to":"a","methods":[]}]}}`))
if err == nil || !strings.Contains(err.Error(), "ADR 0468 §8") {
t.Errorf("the stored form was accepted: %v", err)
}
}
// 1591/AC3: the values a consumer gives carry the shares sorted by `to`, remote (grant) and co-located alike.
func TestTheGrantCarriesTheSharesSorted(t *testing.T) {
m := mustParse(t, filesWithShares)
remote := Resolution{Node: "laptop", Modules: []Manifest{m}}
values, asks, err := remote.ContributionsFrom("s3-bucket", "files", SettingsBy{})
if err != nil || !asks {
t.Fatalf("asks %v, err %v", asks, err)
}
if got := sharesOf(t, values); got != sortedSharesJSON {
t.Fatalf("the grant values carry %s", got)
}
// As the provider's file shows it, through a grant.
provider, err := Resolve(shelf(minioLike()), []string{"minio"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err := provider.Declaration(Rendering{Grants: []Grant{{Provision: "s3-bucket", Consumer: "laptop", From: "files", Values: values}}})
if err != nil {
t.Fatal(err)
}
given := givenToMinio(t, out)
if len(given) != 1 || sharesOf(t, given[0].Values) != sortedSharesJSON {
t.Fatalf("the remote consumer's shares did not reach the file: %+v", given)
}
// Co-located: the consumer contributes to s3-bucket too, and no grant is passed.
co := mustParse(t, `{"module":"files","version":"1","contributes":{"s3-bucket":{"name":"files"}},
"requires":[{"name":"s3-bucket","shares":[{"to":"seat:node-files","methods":["put"]},{"to":"print-and-scan","methods":["put","get"]}]}]}`)
both, err := Resolve(shelf(minioLike(), co), []string{"files"}, workstation(), World{})
if err != nil {
t.Fatal(err)
}
out, err = both.Declaration(Rendering{})
if err != nil {
t.Fatal(err)
}
given = givenToMinio(t, out)
if len(given) != 1 || sharesOf(t, given[0].Values) != sortedSharesJSON || given[0].Values["name"] != "files" {
t.Fatalf("the co-located consumer's shares did not reach the file: %+v", given)
}
}
// 1591/AC4: only the shares change, so the module is not recreated and held for no one; the provider's file changes.
func TestChangingOnlyTheSharesDoesNotRecreateTheConsumer(t *testing.T) {
was := mustParse(t, filesWithShares)
now := mustParse(t, `{"module":"files","version":"1","requires":["shell",
{"name":"s3-bucket","shares":[{"to":"print-and-scan","methods":["get"]}]}]}`)
if said := Recreates(was, now).Say("files"); said != "" {
t.Errorf("a change of shares recreates the module: %s", said)
}
if said := Holds(was, now); said != "" {
t.Errorf("a change of shares holds the module: %s", said)
}
before := Resolution{Node: "laptop", Modules: []Manifest{was}}
after := Resolution{Node: "laptop", Modules: []Manifest{now}}
a, _, _ := before.ContributionsFrom("s3-bucket", "files", SettingsBy{})
b, _, _ := after.ContributionsFrom("s3-bucket", "files", SettingsBy{})
if sharesOf(t, a) == sharesOf(t, b) {
t.Fatalf("the provider's values did not change: %s", sharesOf(t, a))
}
}
// 1591/AC5: a consumer whose contributes to s3-bucket names `shares` itself is refused.
func TestAConsumerCannotContributeSharesItself(t *testing.T) {
_, err := ParseManifest([]byte(`{"module":"files","version":"1",
"contributes":{"s3-bucket":{"shares":"x"}},
"requires":[{"name":"s3-bucket","shares":[{"to":"a","methods":["get"]}]}]}`))
if err == nil || !strings.Contains(err.Error(), "ADR 0468 §8") {
t.Errorf("both written, accepted: %v", err)
}
// Without the field the grant is still refused, so the provider is never told something the manifest did not say.
m := Manifest{Module: "files", Requires: []string{"s3-bucket"},
Contributes: map[string]map[string]any{"s3-bucket": {"shares": "x"}}}
r := Resolution{Node: "laptop", Modules: []Manifest{m}}
if _, _, err := r.ContributionsFrom("s3-bucket", "files", SettingsBy{}); err == nil {
t.Error("a contributed `shares` reached the grant")
}
}
+15 -31
View File
@@ -22,16 +22,12 @@ func catalogueRoot(t *testing.T) string {
}
func TestEveryCatalogueManifestParses(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
named, routed := 0, 0
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
t.Errorf("%s: %v", filepath.Base(filepath.Dir(p)), err)
t.Errorf("%s: %v", filepath.Base(filepath.Dir(r.path)), err)
continue
}
for _, l := range m.Listens {
@@ -54,16 +50,12 @@ func TestEveryCatalogueManifestParses(t *testing.T) {
// definition names a domain or a public address the mesh acts on, and every value that must for now
// carries its reason (novox/hq ADR 0155, issue 134). The list it prints is the one that shrinks.
func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
var named []string
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
t.Errorf("%s: %v", p, err)
t.Errorf("%s: %v", r.path, err)
continue
}
named = append(named, InstallationProblems(m)...)
@@ -78,15 +70,11 @@ func TestNoCatalogueManifestNamesAnInstallation(t *testing.T) {
// backup line by hand, every directory a container writes is declared, and every irreplaceable item is
// backed up — so a store added is a store backed up, without a list of stores to keep in step.
func TestEveryCatalogueModuleDeclaresItsData(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
shelf := Shelf{}
granting := 0
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
continue // TestEveryCatalogueManifestParses says why
}
@@ -108,16 +96,12 @@ func TestEveryCatalogueModuleDeclaresItsData(t *testing.T) {
// overflow fails here, in the pull request that introduces it, rather than on the provider's machine
// the first time a real machine's name meets the module's (issue 263).
func TestEveryCatalogueIdentityFitsWhatItRequires(t *testing.T) {
root := catalogueRoot(t)
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil || len(found) == 0 {
t.Fatalf("no manifests under %s: %v", root, err)
}
_, found := catalogueManifests(t)
shelf := Shelf{}
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
for _, r := range found {
m, err := r.manifest, r.err
if err != nil {
t.Fatalf("%s: %v", p, err)
t.Fatalf("%s: %v", r.path, err)
}
shelf[m.Module] = m
}
+44
View File
@@ -0,0 +1,44 @@
package catalogue
import (
"path/filepath"
"sync"
"testing"
)
// readManifest is one manifest of the catalogue as ReadTreeManifest read it.
type readManifest struct {
path string // the module.json
manifest Manifest
err error
}
var (
readOnce sync.Once
readAll []readManifest
readRoot string
)
// catalogueManifests is every manifest of the catalogue (catalogueRoot), read through ReadTreeManifest once
// per test binary and not once per test: the checks over the real catalogue each read and parsed every
// manifest again, and together took 1.5 s of mesh-sdk's unit bound of 2 s (issue 1756). What it returns is
// shared and read-only: a test that changes a manifest copies it first.
func catalogueManifests(t *testing.T) (string, []readManifest) {
t.Helper()
root := catalogueRoot(t)
readOnce.Do(func() {
readRoot = root
found, err := filepath.Glob(filepath.Join(root, "modules", "*", "module.json"))
if err != nil {
return
}
for _, p := range found {
m, err := ReadTreeManifest(root, filepath.Dir(p))
readAll = append(readAll, readManifest{path: p, manifest: m, err: err})
}
})
if len(readAll) == 0 {
t.Fatalf("no manifests under %s", root)
}
return readRoot, readAll
}
+26
View File
@@ -482,3 +482,29 @@ func TestAnOperatorNamedValueAFileFallsBackOnMayHaveADefault(t *testing.T) {
"default": "", "runtime": true}
refused(t, runtime, `"url"`, `a key naming "url" is the operator's value`)
}
// 1658/AC1: a value whose only placement is ${config:key} in an item's text is placed.
func TestConfigCountsAnItemsTextAsAPlacement(t *testing.T) {
m := map[string]any{
"module": "m", "version": "1",
"config": map[string]any{
"k": map[string]any{"type": "string", "description": "a word the item says", "default": "x"},
},
"items": []any{map[string]any{"collection": "a", "content": "k = ${config:k}\n"}},
}
if _, err := parseRaw(t, m); err != nil {
t.Fatalf("a value placed only in an item's text is refused: %v", err)
}
}
// 1658/AC2: a declared value placed nowhere, with an item that does not say it, is still refused.
func TestConfigStillRefusesAValueNoItemPlaces(t *testing.T) {
m := map[string]any{
"module": "m", "version": "1",
"config": map[string]any{
"k": map[string]any{"type": "string", "description": "a word nothing says", "default": "x"},
},
"items": []any{map[string]any{"collection": "a", "content": "k = fixed\n"}},
}
refused(t, m, `"k"`, "placed nowhere")
}
+15 -2
View File
@@ -1783,6 +1783,10 @@ func (r Resolution) contributions(settings SettingsBy, grants []Grant,
if err != nil {
return nil, err
}
// What the module shares of its bucket reaches the provider with the rest (novox/hq ADR 0468 §8).
if values, err = withShares(m, to, values); err != nil {
return nil, err
}
out[to] = append(out[to], Contribution{From: m.Module, Values: values})
}
// Several contributions to one requirement (ADR 0094's sibling for `contributes`): an
@@ -2169,11 +2173,19 @@ func (r Resolution) ContributionsFrom(requirement, module string, settings Setti
mine = append(mine, g.Values)
}
}
var asker Manifest
for _, m := range r.Modules {
if m.Module == module {
asker = m
}
}
if len(mine) == 1 {
// Its shares are already in, put there where its contribution was composed (contributions).
return mine[0], true, nil
}
if len(mine) > 1 {
return map[string]any{}, true, nil
values, err := withShares(asker, requirement, map[string]any{})
return values, err == nil, err
}
// It contributes no payload — but a require-only consumer of a parameterless provision (one whose
// `serves` names no consumer-supplied key: `redis-cache`, `amqp`) still ASKS for it and must be
@@ -2186,7 +2198,8 @@ func (r Resolution) ContributionsFrom(requirement, module string, settings Setti
}
for _, req := range m.Requires {
if req == requirement {
return map[string]any{}, true, nil
values, err := withShares(m, requirement, map[string]any{})
return values, err == nil, err
}
}
}
+10
View File
@@ -0,0 +1,10 @@
package catalogue
import "testing"
// 1694/AC5: the capability a node-engine reports for files by digest (novox/hq ADR 0486 §5) is a known one.
func TestFileReferencesIsAKnownCapability(t *testing.T) {
if !oneOf(KnownCapabilities, "file-references") {
t.Fatal("file-references is not a known capability")
}
}
+16
View File
@@ -74,6 +74,22 @@ func filesVerbs() []Verb {
map[string]string{"path": "an absolute path", "owner": "an account's name or number",
"group": "a group's name or number (optional)", "recursive": "\"true\": also everything under a directory"},
[]string{"path", "owner"}, []string{"group", "recursive"}, "chown", "chgrp"),
Verb{Name: "upload", Description: "Send one file of this node into a bucket of the object store, over a short signed link " +
"and never over the bus; answers the key (the file's SHA-256 when none is given), the size and the SHA-256. " +
"At most 100 MiB. A place a secret is kept, a known credential file, or a link to one is refused by name, " +
"before anything is sent. Read " + filesAsAccount + ".",
Input: schema(map[string]string{"path": "an absolute path to a file", "bucket": "the bucket the file goes into",
"key": "the object's key (optional; the file's SHA-256 when not given)"}, []string{"path", "bucket"})},
// download is a plain Verb, not through write(): write() types its optional arguments as true/false
// switches, which a hash and a reason are not (issue 1719).
Verb{Name: "download", Description: "Write one object of a bucket to a file of this node, over a short signed link and " +
"never over the bus; the old file goes to the node's trash. At most 100 MiB. The bytes must hash to sha256, or to " +
"the key when the key is a SHA-256; otherwise no file is written. Refuses a path a module owns " +
"unless override names a reason; done as the node's operator account, through sudo -n where its rights do not reach.",
Input: schema(map[string]string{"bucket": "the bucket", "key": "the object's key", "path": "an absolute path",
"sha256": "the SHA-256 the bytes must have (optional; the key when that is a SHA-256)",
"reason": "why the file is changed, for the audit trail",
"override": overrideWord}, []string{"bucket", "key", "path", "reason"})},
Verb{Name: "trash", Description: "What the node's trash holds: each deleted or replaced item with its id, " +
"original path and when it went.",
Input: schema(map[string]string{}, nil), Replaces: []string{"ls ~/.local/share/Trash"}},
+94 -2
View File
@@ -2,6 +2,7 @@ package catalogue
import (
"encoding/json"
"sort"
"strings"
"testing"
)
@@ -22,7 +23,8 @@ func TestNodeFilesIsANodeSeatWithEveryVerbOfADR0394(t *testing.T) {
if v.Optional {
t.Errorf("%s.%s is optional", FilesSeat, v.Name)
}
if len(v.Replaces) == 0 {
// upload and download (ADR 0468 §9, issue 1718) move bytes through the object store: no command is theirs.
if len(v.Replaces) == 0 && v.Name != "upload" && v.Name != "download" {
t.Errorf("%s.%s says nothing it replaces", FilesSeat, v.Name)
}
for _, c := range v.Replaces {
@@ -30,7 +32,7 @@ func TestNodeFilesIsANodeSeatWithEveryVerbOfADR0394(t *testing.T) {
}
}
for _, name := range []string{"stat", "readlink", "list", "read", "search", "account", "write", "copy", "move",
"delete", "mkdir", "chmod", "chown", "trash", "restore"} {
"delete", "mkdir", "chmod", "chown", "upload", "download", "trash", "restore"} {
if !have[name] {
t.Errorf("%s serves no verb %s", FilesSeat, name)
}
@@ -109,3 +111,93 @@ func TestNodeFilesRestoreTakesIdAndReason(t *testing.T) {
}
t.Error("no restore verb")
}
// 1718/AC1: node-files names upload and download, in mesh-files' order, before trash.
func TestNodeFilesNamesUploadAndDownloadBeforeTrash(t *testing.T) {
s, _ := SeatNamed(FilesSeat)
var names []string
for _, v := range s.Serves {
names = append(names, v.Name)
}
joined := strings.Join(names, ",")
if !strings.Contains(joined, "chown,upload,download,trash") {
t.Errorf("1718/AC1: node-files' verbs are %s, want upload and download between chown and trash", joined)
}
}
// 1718/AC2: a holder serving every verb including upload and download is accepted; one naming a verb outside the
// contract is still refused, and one leaving upload out is refused.
func TestAHolderOfNodeFilesServesUploadAndDownloadAndNothingElse(t *testing.T) {
s, _ := SeatNamed(FilesSeat)
var verbs []string
for _, v := range s.Serves {
verbs = append(verbs, v.Name)
}
hold := func(serves []string) error {
return CanHold(Manifest{Module: "mesh-files", Version: "1",
Claims: []Claim{{Name: FilesSeat, Scope: ScopeNode, Serves: serves}}}, s)
}
if err := hold(verbs); err != nil {
t.Errorf("1718/AC2: a holder serving every verb was refused: %v", err)
}
if err := hold(append(append([]string{}, verbs...), "sync")); err == nil || !strings.Contains(err.Error(), "does not promise") {
t.Errorf("1718/AC2: a verb outside the contract was accepted: %v", err)
}
var without []string
for _, v := range verbs {
if v != "upload" {
without = append(without, v)
}
}
if err := hold(without); err == nil || !strings.Contains(err.Error(), "upload") {
t.Errorf("1718/AC2: a holder leaving upload out was accepted: %v", err)
}
}
// 1718/AC3: upload requires path and bucket; download requires bucket, key, path and reason; sha256, key (upload)
// and override are plain strings, not true/false switches.
func TestNodeFilesUploadAndDownloadInputs(t *testing.T) {
s, _ := SeatNamed(FilesSeat)
want := map[string]struct {
required string
strs []string
}{
"upload": {"path,bucket", []string{"path", "bucket", "key"}},
"download": {"bucket,key,path,reason", []string{"bucket", "key", "path", "sha256", "reason", "override"}},
}
seen := 0
for _, v := range s.Serves {
w, ok := want[v.Name]
if !ok {
continue
}
seen++
req, _ := v.Input["required"].([]string)
got := append([]string{}, req...)
sortStrings(got)
exp := strings.Split(w.required, ",")
sortStrings(exp)
if strings.Join(got, ",") != strings.Join(exp, ",") {
t.Errorf("1718/AC3: %s requires %v, want %s", v.Name, req, w.required)
}
props, _ := v.Input["properties"].(map[string]any)
if len(props) != len(w.strs) {
t.Errorf("1718/AC3: %s has %d inputs, want %v", v.Name, len(props), w.strs)
}
for _, name := range w.strs {
p, has := props[name].(map[string]any)
if !has {
t.Errorf("1718/AC3: %s has no input %s", v.Name, name)
continue
}
if _, enum := p["enum"]; enum || p["type"] != "string" {
t.Errorf("1718/AC3: %s input %s is %v, want a plain string", v.Name, name, p)
}
}
}
if seen != 2 {
t.Errorf("1718/AC3: found %d of upload and download", seen)
}
}
func sortStrings(s []string) { sort.Strings(s) }
+149
View File
@@ -0,0 +1,149 @@
package catalogue
import (
"reflect"
"strings"
"testing"
)
// A container's supplementary groups may be an operator's value (novox/hq ADR 0480 clause 7, ADR 0348 §11,
// issue 1636): `groups: ["${config:media-gid}"]`, each element a whole placeholder or a whole number, filled at
// composition with the number the config value holds.
func aContainerWithGroups(groups ...any) map[string]any {
return map[string]any{"id": "server", "type": "container", "name": "n8n", "image": "x@sha256:" + strings.Repeat("a", 64),
"groups": groups}
}
// 1636/AC3: a manifest container with groups: ["${setting:media-gid}"] resolves to the number the setting says.
func TestAContainersGroupIsFilledFromTheSetting(t *testing.T) {
r := aContainerWithGroups("${setting:media-gid}", float64(2001))
layers := []Layer{{From: "ace", Values: map[string]any{"media-gid": float64(2000)}}}
if err := settingInto(r, layers, "n8n"); err != nil {
t.Fatal(err)
}
if want := []any{2000, 2001}; !reflect.DeepEqual(r["groups"], want) {
t.Fatalf("composed as %#v, want %#v", r["groups"], want)
}
// A value the operator wrote as text is read as the number it spells.
r = aContainerWithGroups("${setting:media-gid}")
if err := settingInto(r, []Layer{{From: "ace", Values: map[string]any{"media-gid": "2000"}}}, "n8n"); err != nil {
t.Fatal(err)
}
if want := []any{2000}; !reflect.DeepEqual(r["groups"], want) {
t.Fatalf("composed as %#v, want %#v", r["groups"], want)
}
}
// 1636/AC3: an unset value refuses the module by name, and the key and the remedy are named; nothing is dropped.
func TestAnUnsetGroupRefusesTheModuleByName(t *testing.T) {
r := aContainerWithGroups("${setting:media-gid}")
err := settingInto(r, nil, "n8n")
if err == nil || !strings.Contains(err.Error(), "n8n") || !strings.Contains(err.Error(), "media-gid") {
t.Fatalf("want a refusal naming n8n and media-gid, got %v", err)
}
if got, _ := r["groups"].([]any); len(got) != 1 || got[0] != "${setting:media-gid}" {
t.Fatalf("the resource was changed: %#v", r["groups"])
}
}
// 1636/AC2: a value that is not a whole number above zero cannot be a group.
func TestAGroupThatIsNotAWholeNumberAboveZeroIsRefused(t *testing.T) {
for _, bad := range []any{"media", "-1", "0", float64(0), float64(2000.5), "2000:2000", "", true} {
r := aContainerWithGroups("${setting:media-gid}")
err := settingInto(r, []Layer{{From: "ace", Values: map[string]any{"media-gid": bad}}}, "n8n")
if err == nil || !strings.Contains(err.Error(), "media-gid") {
t.Errorf("value %#v: want a refusal naming the key, got %v", bad, err)
}
}
}
// A manifest in the config form registers with the group a config value fills, and the value is placed (a container's
// groups count as a place for it); a placeholder in any other shape is refused where the manifest is read.
func TestAManifestWhoseContainerTakesItsGroupFromConfigRegisters(t *testing.T) {
m := map[string]any{
"module": "n8n", "version": "1",
"config": map[string]any{"media-gid": map[string]any{"type": "number", "minimum": 1, "required": true,
"description": "the group that owns the media library on this machine"}},
"resources": []any{aContainerWithGroups("${config:media-gid}")},
}
if _, err := parseRaw(t, m); err != nil {
t.Fatalf("refused: %v", err)
}
for _, shape := range []string{"x${config:media-gid}", "${config:media-gid}:1", "${config:other}"} {
m["resources"] = []any{aContainerWithGroups(shape)}
if _, err := parseRaw(t, m); err == nil {
t.Errorf("groups %q registered", shape)
}
}
}
// 1636/AC3 composed for a machine, the way a push writes it: n8n's group is the number the machine's config value
// says, written as a number, and the module is left out by name when the machine says none.
func TestAComposedMachineGetsTheGroupTheConfigValueNames(t *testing.T) {
manifest, err := parseRaw(t, map[string]any{
"module": "n8n", "version": "1",
"config": map[string]any{"media-gid": map[string]any{"type": "number", "minimum": 1, "required": true,
"description": "the group that owns the media library on this machine"}},
"resources": []any{aContainerWithGroups("${config:media-gid}")},
})
if err != nil {
t.Fatal(err)
}
r := anAdoptedAnchor()
r.Modules = append(r.Modules, manifest)
with := anchorRendering(false)
with.Settings["n8n"] = []Layer{{From: "anchor", Values: map[string]any{"media-gid": float64(2000)}}}
composed, err := r.Compose(with)
if err != nil {
t.Fatal(err)
}
if why, left := composed.LeftOut["n8n"]; left {
t.Fatalf("left out: %s", why)
}
if got := byID(composed.Resources)["n8n.server"]["groups"]; !reflect.DeepEqual(got, []any{2000}) {
t.Fatalf("composed groups %#v", got)
}
without := anchorRendering(false)
composed, err = r.Compose(without)
if err == nil {
why, left := composed.LeftOut["n8n"]
if !left || !strings.Contains(why, "media-gid") {
t.Fatalf("a machine that says no group kept n8n or left it out without the key: %q (%v)", why, left)
}
} else if !strings.Contains(err.Error(), "media-gid") {
t.Fatalf("refused without the key: %v", err)
}
}
// The node-engine reads a container's groups only when it reports the capability (novox/hq ADR 0480 clause 7, issue 1638):
// a module with a container that names groups is hosted on a machine that reports container-groups, and left out of one
// that does not, the capability named, so the field is never sent to an engine that would refuse it.
func TestAModuleWhoseContainerNamesGroupsIsHeldOnAMachineThatDoesNotReadThem(t *testing.T) {
n8n := Manifest{Module: "n8n", Resources: []map[string]any{aContainerWithGroups("${config:media-gid}")}}
plain := Manifest{Module: "plain", Resources: []map[string]any{
{"id": "server", "type": "container", "name": "p", "image": "x@sha256:" + strings.Repeat("b", 64)}}}
for _, m := range []Manifest{n8n, plain} {
if got := impliedCapabilities(m); (m.Module == "n8n") != reflect.DeepEqual(got, []string{ContainerGroups}) {
t.Errorf("%s implies %v", m.Module, got)
}
}
shelf := shelf(n8n, plain)
reads := Node{Name: "ace", Capabilities: map[string]bool{"container-runtime": true, ContainerGroups: true}}
got, err := Resolve(shelf, []string{"n8n", "plain"}, reads, World{})
if err != nil || len(got.Unhostable) != 0 || len(got.Modules) != 2 {
t.Fatalf("a machine that reads groups: %v, held %+v, hosted %v", err, got.Unhostable, names(got))
}
old := Node{Name: "g14", Capabilities: map[string]bool{"container-runtime": true}}
got, err = Resolve(shelf, []string{"n8n", "plain"}, old, World{})
if err != nil {
t.Fatal(err)
}
if len(got.Unhostable) != 1 || got.Unhostable[0].Module != "n8n" || !reflect.DeepEqual(got.Unhostable[0].Missing, []string{ContainerGroups}) {
t.Fatalf("n8n was not held, naming the capability: %+v", got.Unhostable)
}
if h := names(got); len(h) != 1 || h[0] != "plain" {
t.Errorf("the module without groups is not the one hosted: %v", h)
}
}
+47 -2
View File
@@ -494,6 +494,9 @@ type Manifest struct {
// by UnmarshalJSON from the long form of a `requires` entry, {name, optional}, or from the stored
// form's `optional-requires` list; never written by hand.
OptionalRequires map[string]bool `json:"-"`
// RequireShares is who else may use the bucket a module requires, by requirement (BucketShare, novox/hq ADR 0468
// §8): filled from the long form's `shares`, or from the stored form's `requires-shares`; never written by hand.
RequireShares map[string][]BucketShare `json:"-"`
// Emits are the events this module publishes, named **locally**: `order.placed`, not a subject
// and not a routing key. The mesh derives where it lands (design 29 §1), so reorganising the
@@ -1538,8 +1541,9 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
delete(keys, "contributes")
}
var optionalRequires map[string]bool
var requireShares map[string][]BucketShare
if requires, ok := keys["requires"]; ok && string(requires) != "null" {
names, optional, err := readRequires(requires)
names, optional, shares, err := readRequires(requires)
if err != nil {
return err
}
@@ -1549,6 +1553,23 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
}
keys["requires"] = plainNames
optionalRequires = optional
requireShares = shares
}
// The stored form's shares (RequireSharesKey).
if stored, ok := keys[RequireSharesKey]; ok {
if string(stored) != "null" {
stood, err := readRequireShares(stored)
if err != nil {
return err
}
for requirement, shares := range stood {
if requireShares == nil {
requireShares = map[string][]BucketShare{}
}
requireShares[requirement] = shares
}
}
delete(keys, RequireSharesKey)
}
// The stored form: `requires` plain, and which are optional beside it (OptionalRequiresKey).
if stored, ok := keys[OptionalRequiresKey]; ok {
@@ -1599,6 +1620,7 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
m.unknown = unknown
m.forms = forms
m.OptionalRequires = optionalRequires
m.RequireShares = requireShares
if len(plain) > 0 {
m.Secrets = plain
}
@@ -1611,6 +1633,21 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
if len(contributesMany) > 0 {
m.ContributesMany = contributesMany
}
if err := m.checkSharesNotContributed(); err != nil {
return err
}
return nil
}
// checkSharesNotContributed refuses a module whose own `contributes` to s3-bucket names `shares` (ADR 0468 §8).
func (m Manifest) checkSharesNotContributed() error {
if len(m.RequireShares[BucketRequirement]) == 0 {
return nil
}
if _, named := m.Contributes[BucketRequirement][SharesValue]; named {
return fmt.Errorf("contributes.%s names `shares` and so does its requirement; keep one, the requirement's (novox/hq ADR 0468 §8)",
BucketRequirement)
}
return nil
}
@@ -1655,7 +1692,7 @@ func (m Manifest) marshalFields() ([]byte, error) {
if err != nil {
return nil, err
}
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 && len(m.OptionalRequires) == 0 {
if len(m.SecretsMany) == 0 && len(m.ContributesMany) == 0 && len(m.OptionalRequires) == 0 && len(m.RequireShares) == 0 {
return raw, nil
}
var keys map[string]json.RawMessage
@@ -1671,6 +1708,13 @@ func (m Manifest) marshalFields() ([]byte, error) {
}
keys[OptionalRequiresKey] = listed
}
if len(m.RequireShares) > 0 {
listed, err := json.Marshal(m.RequireShares)
if err != nil {
return nil, err
}
keys[RequireSharesKey] = listed
}
if len(m.ContributesMany) > 0 {
mergedContributes := map[string]any{}
for to, values := range m.Contributes {
@@ -2170,6 +2214,7 @@ func parseManifest(raw []byte, predefined PredefinedList) (Manifest, error) {
problems = append(problems, replacesProblems(m)...)
problems = append(problems, UnitSettingProblems(m)...)
problems = append(problems, OwnerSettingProblems(m)...)
problems = append(problems, GroupSettingProblems(m)...)
problems = append(problems, endpointNameProblems(m)...)
problems = append(problems, RouteProblems(m)...)
for _, port := range m.Guards {
+4 -10
View File
@@ -4,8 +4,6 @@ import (
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/internal/beside"
)
// A bind mount the module never declared is refused where it is written (novox/hq 04-ISSUES/026,
@@ -71,14 +69,10 @@ func TestTheRuntimeSocketIsGrantedByTheCapabilityAndNotOtherwise(t *testing.T) {
// **Every manifest in the catalogue (internal/beside) passes**, so the rule is not one the catalogue
// is already breaking.
func TestEveryCatalogueManifestDeclaresWhatItMounts(t *testing.T) {
files, _ := filepath.Glob(filepath.Join(beside.Catalogue(t), "*", "module.json"))
if len(files) == 0 {
t.Fatal("no manifests in the catalogue, so this proved nothing")
}
root := beside.Dir(t, "mesh-catalog") // once: it copies the captured tree
for _, file := range files {
if _, err := ReadTreeManifest(root, filepath.Dir(file)); err != nil {
t.Errorf("%s: %v", filepath.Base(filepath.Dir(file)), err)
_, files := catalogueManifests(t)
for _, f := range files {
if f.err != nil {
t.Errorf("%s: %v", filepath.Base(filepath.Dir(f.path)), f.err)
}
}
}
+17 -5
View File
@@ -32,17 +32,20 @@ const OnlyWhenBound = "only-when-bound"
type requirementForm struct {
Name string `json:"name"`
Optional bool `json:"optional,omitempty"`
// Shares is who else may use the consumer's bucket (BucketShare, novox/hq ADR 0468 §8), on s3-bucket only.
Shares []BucketShare `json:"shares,omitempty"`
}
// readRequires reads `requires` in both of its forms — a bare name, or {name, optional} — into the
// names and which of them are optional.
func readRequires(raw json.RawMessage) ([]string, map[string]bool, error) {
func readRequires(raw json.RawMessage) ([]string, map[string]bool, map[string][]BucketShare, error) {
var entries []json.RawMessage
if err := json.Unmarshal(raw, &entries); err != nil {
return nil, nil, fmt.Errorf("requires: a list of names, or of {name, optional}: %w", err)
return nil, nil, nil, fmt.Errorf("requires: a list of names, or of {name, optional}: %w", err)
}
var names []string
optional := map[string]bool{}
var shares map[string][]BucketShare
for i, e := range entries {
var plain string
if err := json.Unmarshal(e, &plain); err == nil {
@@ -53,12 +56,21 @@ func readRequires(raw json.RawMessage) ([]string, map[string]bool, error) {
dec := json.NewDecoder(bytes.NewReader(e))
dec.DisallowUnknownFields()
if err := dec.Decode(&full); err != nil {
return nil, nil, fmt.Errorf("requires[%d]: a name, or {name, optional} (novox/hq ADR 0330): %w", i, err)
return nil, nil, nil, fmt.Errorf("requires[%d]: a name, or {name, optional} (novox/hq ADR 0330): %w", i, err)
}
if strings.TrimSpace(full.Name) == "" {
return nil, nil, fmt.Errorf("requires[%d] names no requirement: {name, optional}", i)
return nil, nil, nil, fmt.Errorf("requires[%d] names no requirement: {name, optional}", i)
}
names = append(names, full.Name)
if err := checkBucketShares(full.Name, full.Shares); err != nil {
return nil, nil, nil, fmt.Errorf("requires[%d]: %w", i, err)
}
if len(full.Shares) > 0 {
if shares == nil {
shares = map[string][]BucketShare{}
}
shares[full.Name] = full.Shares
}
if full.Optional {
optional[full.Name] = true
}
@@ -66,7 +78,7 @@ func readRequires(raw json.RawMessage) ([]string, map[string]bool, error) {
if len(optional) == 0 {
optional = nil
}
return names, optional, nil
return names, optional, shares, nil
}
// OptionalRequiresKey is where a stored manifest keeps which of its requirements are optional: beside
+18 -1
View File
@@ -904,7 +904,7 @@ func isModule(catalogue map[string]Manifest, want string) bool {
// machine can host the module as far as its capabilities go.
func missingCapabilities(m Manifest, node Node) []string {
var missing []string
for _, c := range m.Capabilities {
for _, c := range append(append([]string(nil), m.Capabilities...), impliedCapabilities(m)...) {
if !node.Capabilities[c] {
missing = append(missing, c)
}
@@ -1457,3 +1457,20 @@ func packageNames(m map[string][]string) []string {
sort.Strings(names)
return names
}
// ContainerGroups is the capability a node-engine reports when it reads a container's `groups` and passes each as a
// supplementary group (novox/hq ADR 0480 clause 7, issue 1638). An older engine refuses the key (strict decoding).
const ContainerGroups = "container-groups"
// impliedCapabilities are the capabilities a module needs for what its resources say, beyond those it declares: a
// container that names groups needs an engine that reads them, so the module is left out of a machine whose report
// lacks the capability, naming it, and the field is sent to no engine that would refuse it. None for a module whose
// containers name no groups.
func impliedCapabilities(m Manifest) []string {
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "container" && len(groupsOf(r)) > 0 {
return []string{ContainerGroups}
}
}
return nil
}
+2 -1
View File
@@ -66,7 +66,8 @@ var capabilityName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`)
// names an `if-capability` may give (novox/hq ADR 0255). A name nothing detects would leave its
// contribution out on every machine, silently, so it is refused. A detector added there is added here.
var KnownCapabilities = []string{
"battery", "container-runtime", "firewall", "graphical-session", "overlay", "package-manager",
"battery", "container-groups", "container-runtime", "file-references", "firewall", "graphical-session", "overlay",
"package-manager",
"power-meter", "privileged", "seat", "service-manager", "uplink-dhcpcd", "uplink-networkmanager",
"uplink-systemd-networkd", "user-namespaces", "virtualisation",
}
+111 -1
View File
@@ -85,6 +85,9 @@ func settingInto(resource map[string]any, layers []Layer, module string) error {
if fmt.Sprint(resource["type"]) == "directory" {
return settingIntoOwner(resource, layers, module)
}
if fmt.Sprint(resource["type"]) == "container" {
return settingIntoGroups(resource, layers, module)
}
if fmt.Sprint(resource["type"]) != "file" {
return nil
}
@@ -324,7 +327,7 @@ func settingKeysUsedBy(m Manifest) map[string]bool {
return used
}
// settingKeysAskedOutsideOwners is every key a module's files, units, contributions and served facts
// settingKeysAskedOutsideOwners is every key a module's files, units, items, contributions and served facts
// ask for: what a directory's owner may follow (OwnerSettingProblems).
func settingKeysAskedOutsideOwners(m Manifest) map[string]bool {
used := map[string]bool{}
@@ -343,8 +346,18 @@ func settingKeysAskedOutsideOwners(m Manifest) map[string]bool {
if unit, ok := r["unit"].(string); ok {
note(unit)
}
case "container":
for _, g := range groupsOf(r) {
if s, ok := g.(string); ok {
note(s)
}
}
}
}
// An item's text, filled from its template (WithTemplatesFrom), places a key as a file's does (issue 1658).
for _, it := range m.Items {
note(it.Content)
}
inValues := func(values map[string]any) {
for _, v := range values {
if s, ok := v.(string); ok {
@@ -365,3 +378,100 @@ func settingKeysAskedOutsideOwners(m Manifest) map[string]bool {
}
return used
}
// A container's supplementary groups may be an operator's value (novox/hq ADR 0480 clause 7, ADR 0348 §11, issue 1636):
// n8n joins the media library's group, whose number is the machine's, never the definition's (ADR 0112 §1). Each
// element of `groups` is a whole placeholder or a whole number; a placeholder is filled with the number the
// config value holds, and the list reaches the node-engine as numbers.
// groupSide is one element of a container's groups: a whole placeholder or a whole number.
var groupSide = regexp.MustCompile(`^(?:\$\{setting:[a-z0-9][a-z0-9_.-]*\}|[0-9]+)$`)
// groupsOf is a container resource's groups as the manifest wrote them (text or numbers), none when it names none.
func groupsOf(r map[string]any) []any {
groups, _ := r["groups"].([]any)
return groups
}
// GroupSettingProblems refuses, where a manifest is read, a container whose groups hold anything but whole
// placeholders and whole numbers, or a placeholder whose key the module neither declares nor asks for elsewhere.
func GroupSettingProblems(m Manifest) []string {
var problems []string
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
for _, g := range groupsOf(r) {
s, isText := g.(string)
if !isText {
continue // a number: the declaration judges its value
}
if !groupSide.MatchString(s) {
problems = append(problems, fmt.Sprintf("%s: the container %v's group %q is not a whole "+
"${config:key} or a whole number: a supplementary group is a number, and nothing else is "+
"written into it (novox/hq ADR 0480 clause 7)", m.Module, r["id"], s))
continue
}
if m.Templated() {
continue
}
for _, key := range settingsUsed(s) {
if _, declared := m.Settings[key]; !declared && !m.DeclaresConfig() {
continue // the settings form judges its own keys
}
if _, declared := m.Settings[key]; !declared {
problems = append(problems, fmt.Sprintf("%s: the container %v's group says ${config:%s}, and the module "+
"declares no such value (novox/hq ADR 0348 §4)", m.Module, r["id"], key))
}
}
}
}
return problems
}
// settingIntoGroups fills ${setting:…} in a container's groups, refusing — with the key and why, the resource left as
// it was — a key nothing sets and a value that is not a whole number above zero. A number is kept; the list that
// results is numbers alone.
func settingIntoGroups(resource map[string]any, layers []Layer, module string) error {
groups := groupsOf(resource)
if len(groups) == 0 {
return nil
}
filled := make([]any, 0, len(groups))
for _, g := range groups {
s, isText := g.(string)
if !isText {
if f, isNumber := g.(float64); isNumber && f == float64(int(f)) {
g = int(f) // a number the manifest wrote, as the numbers a placeholder gives
}
filled = append(filled, g)
continue
}
keys := settingsUsed(s)
if len(keys) != 1 || settingRef.ReplaceAllString(s, "") != "" {
filled = append(filled, g) // not a whole placeholder: the manifest check refuses it
continue
}
key := keys[0]
value, set := settingValue(layers, key)
if !set {
return &UnsetSettingError{Module: module, Setting: key,
where: fmt.Sprintf("a container %v whose groups", resource["id"]), setToday: orNoSettings(layers),
said: fmt.Sprintf(
"%s has a container %v whose groups say ${setting:%s}, and nothing sets %q for it — an operator's "+
"value is the assignment's, never the definition's (novox/hq ADR 0112): "+
"`settings set %s <file>` with {%q: …}%s",
module, resource["id"], key, key, module, key, orNoSettings(layers))}
}
v := plainly(value)
id, whole := wholeID(v)
if !whole || id == 0 {
return fmt.Errorf("%s: the setting %q is %q, which cannot be a group of the container %v: a supplementary "+
"group is a whole number above zero, never a name or other text (novox/hq ADR 0480 clause 7, ADR 0328 §15)",
module, key, v, resource["id"])
}
filled = append(filled, int(id))
}
resource["groups"] = filled
return nil
}
+88
View File
@@ -0,0 +1,88 @@
package catalogue
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0477 §4 (issue 1573): the go and nodejs modules own the toolchain packages and provide
// them with the machine's reach, so a node without one refuses the module that needs it at resolution, naming
// the provision, and does not find out at its first build. The manifests are the catalogue's, as they are.
// builderNode is a machine that can run a build module: the package manager and user namespaces it asks of
// the machine, and the forge account (mesh-scope, provided by the forge) stood in for.
func builderNode() Node {
return Node{Name: "builder", Site: "house",
Capabilities: map[string]bool{"package-manager": true, "user-namespaces": true}}
}
func toolchainShelf(t *testing.T) map[string]Manifest {
t.Helper()
m := map[string]Manifest{
"forge": {Module: "forge", Version: "1", Provides: []Offer{{Name: "forge-account"}, {Name: "npm-package-registry"}}},
}
for _, name := range []string{"go", "nodejs", "mesh-build-go", "mesh-build-node", "dev-tools"} {
m[name] = catalogueManifest(t, name)
}
return m
}
func TestAModuleNeedingAToolchainIsRefusedOnANodeWithoutIt(t *testing.T) {
for _, c := range []struct{ module, provision string }{
{"mesh-build-go", "go-toolchain"},
{"mesh-build-node", "node-toolchain"},
{"dev-tools", "go-toolchain"},
} {
_, err := Resolve(toolchainShelf(t), []string{"forge", c.module}, builderNode(), World{})
if err == nil {
t.Fatalf("%s composed on a node with no toolchain module", c.module)
}
for _, want := range []string{`"` + c.provision + `" is wanted by ` + c.module,
"usable only on the machine that provides it"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("%s: the refusal does not say %q:\n%v", c.module, want, err)
}
}
}
}
func TestAModuleNeedingAToolchainComposesOnANodeWithIt(t *testing.T) {
for _, c := range []struct{ module, owner string }{
{"mesh-build-go", "go"},
{"mesh-build-node", "nodejs"},
{"dev-tools", "go"},
} {
assigned := []string{"forge", c.owner, c.module}
if c.module == "dev-tools" {
assigned = append(assigned, "nodejs")
}
got, err := Resolve(toolchainShelf(t), assigned, builderNode(), World{})
if err != nil {
t.Fatalf("%s with %s did not compose: %v", c.module, c.owner, err)
}
in := map[string]bool{}
for _, n := range names(got) {
in[n] = true
}
if !in[c.module] || !in[c.owner] {
t.Errorf("%s with %s composed %v", c.module, c.owner, names(got))
}
}
}
// Only the owners declare the packages: a build module that declared go again would compose without the
// owner's provision being met by anything, which is the gap the provision closes.
func TestOnlyTheToolchainModulesDeclareTheToolchainPackages(t *testing.T) {
owner := map[string]string{"go": "go", "nodejs": "nodejs", "npm": "nodejs"}
for name, m := range toolchainShelf(t) {
for _, r := range m.Resources {
if r["type"] != "package" {
continue
}
pkg, _ := r["package"].(string)
if o, ok := owner[pkg]; ok && o != name {
t.Errorf("%s declares the package %s, which only %s may", name, pkg, o)
}
}
}
}
+1 -1
View File
@@ -56,7 +56,7 @@ type filler struct {
// seatInto, machineInto and contributionsInto. A pass that comes to read another field adds it here,
// or the sweep refuses the placeholder it would have filled.
var fillers = []filler{
{namespace: "setting", patterns: []*regexp.Regexp{settingRef}, fields: map[string][]string{"file": {"content"}, "service": {"unit"}, "directory": {"owner"}}},
{namespace: "setting", patterns: []*regexp.Regexp{settingRef}, fields: map[string][]string{"file": {"content"}, "service": {"unit"}, "directory": {"owner"}, "container": {"groups"}}},
{namespace: "dir", patterns: []*regexp.Regexp{dirRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "access", patterns: []*regexp.Regexp{accessRef}, fields: map[string][]string{"*": {"path", "content", "volumes", "env", "env-file"}}},
{namespace: "secret", patterns: []*regexp.Regexp{placeholder}, fields: map[string][]string{"file": {"content"}},
+3 -3
View File
@@ -106,7 +106,7 @@ func TestAFailedWriteWaitsNoLongerThanTheDeadline(t *testing.T) {
default:
}
})
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
start := time.Now()
_, err = Ask(ctx, socket, aLineTheSocketCannotHold())
@@ -117,7 +117,7 @@ func TestAFailedWriteWaitsNoLongerThanTheDeadline(t *testing.T) {
if !errors.Is(err, os.ErrDeadlineExceeded) {
t.Fatalf("the error is not the deadline: %v", err)
}
if waited > 3*time.Second {
t.Fatalf("waited %s with a deadline of 1s", waited)
if waited > time.Second {
t.Fatalf("waited %s with a deadline of 50ms", waited)
}
}
+46
View File
@@ -324,6 +324,15 @@ type File struct {
// depend on a field somewhere else.
Bytes string `json:"bytes,omitempty"`
// Digest and Size say the content is not carried but fetched: a file over FileReferenceThreshold travels as
// the sha256 digest of its content, "sha256:<64 hex>", and its size in bytes, in place of Content or Bytes
// (novox/hq ADR 0486 §1). The node-engine fetches the object of that name from FileBucketFor(node), checks what it
// received against Digest and Size, and only then writes the file. Exclusive with Content, Sealed, Bytes,
// VaultSealed and Secrets; sent only to an engine that reports FileReferencesCapability, as an older engine
// refuses a field it does not know, whole.
Digest string `json:"digest,omitempty"`
Size int64 `json:"size,omitempty"`
// Owner is the user this belongs to, by name. Absent means root.
Owner string `json:"owner,omitempty"`
@@ -465,6 +474,10 @@ func (f *File) validate(where string, _ bool) []string {
problems = append(problems, where+": a file the vault sealed whole has no holes for secrets")
}
}
problems = append(problems, f.checkReference(where)...)
if f.Digest != "" || f.Size != 0 {
said = append(said, "digest")
}
if len(said) > 1 {
sort.Strings(said)
problems = append(problems, where+
@@ -1454,6 +1467,14 @@ type Container struct {
// container runs as its image says. Its key is `identity`, the manifest's word; the field is not, because
// a resource's Identity is its id.
RunAs *ContainerIdentity `json:"identity,omitempty"`
// Groups are the supplementary groups the container's process joins, as numbers, each passed to the runtime
// as a group to add (novox/hq ADR 0480 clause 7, issue 1636): n8n writes the media library through the
// library's group, which a stock image knows nothing of. Each is above zero and below the kernel's "no group"
// (ADR 0328 §15: never root). Part of the spec, so a change recreates the container. Absent: none are added.
// An older node-engine refuses the key (strict decoding), so the controller sends it only to a node that
// reports it reads it.
Groups []int64 `json:"groups,omitempty"`
}
// The ways an image is told who to run as (novox/hq ADR 0325 §2).
@@ -1521,6 +1542,30 @@ func (i *ContainerIdentity) problems(where string, env map[string]string) []stri
return problems
}
// groupProblems refuses a supplementary group that is root's or below it, the kernel's "no group" number, or a
// number named twice (novox/hq ADR 0480 clause 7, ADR 0328 §15).
func (c *Container) groupProblems(where string) []string {
var problems []string
seen := map[int64]bool{}
for _, g := range c.Groups {
switch {
case g == 0:
problems = append(problems, fmt.Sprintf("%s: container %s adds group 0, and the mesh never makes or keeps anything "+
"as root: a supplementary group is never root's (novox/hq ADR 0328 §15)", where, c.Name))
case g < 0:
problems = append(problems, fmt.Sprintf("%s: container %s adds group %d, below zero: a group is a whole number "+
"above zero", where, c.Name, g))
case g >= 1<<32-1:
problems = append(problems, fmt.Sprintf("%s: container %s adds group %d, which is the kernel's \"no group\" "+
"number or beyond it", where, c.Name, g))
case seen[g]:
problems = append(problems, fmt.Sprintf("%s: container %s adds group %d twice", where, c.Name, g))
}
seen[g] = true
}
return problems
}
func (c *Container) Identity() string { return c.ID }
func (c *Container) Kind() Type { return TypeContainer }
func (c *Container) Target() string { return c.Name }
@@ -1603,6 +1648,7 @@ func (c *Container) validate(where string, _ bool) []string {
"runs and exits, and joins nothing afterwards")
}
problems = append(problems, c.RunAs.problems(where+": container "+c.Name, c.Env)...)
problems = append(problems, c.groupProblems(where)...)
return append(problems, checkImage(where, c.Image)...)
}
+150
View File
@@ -0,0 +1,150 @@
package declaration
import (
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"fmt"
"regexp"
)
// The three names of a file travelling by digest (novox/hq ADR 0486 §1, §2, §5), shared by the node-engine that
// reads them and the controller that writes them.
const (
// FileReferenceThreshold is the size above which a file travels by digest: a file of this many bytes or fewer
// stays in the declaration as it is.
FileReferenceThreshold = 64 * 1024
// FileBucketPrefix starts the name of a node's object bucket, FileBucketFor(node), which holds the contents of the
// files of that node's declaration, each under the hex of its sha256 digest (ObjectName). One bucket per node,
// so a grant names it and a literal `*` is never needed (ADR 0486 §2, changed by ADR 0488 §1 and §2). The controller alone writes it; the
// node's engine reads an object by name.
FileBucketPrefix = "declaration-files-"
// FileStreamPrefix starts the name of the stream behind such a bucket, which the bus's grants name.
FileStreamPrefix = "OBJ_"
// FileReferencesCapability is what a node-engine reports when it reads a file given as a digest and size.
FileReferencesCapability = "file-references"
)
var nodeToken = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
// FileBucketFor is the name of the object bucket of a node's files, "declaration-files-<node>" (novox/hq ADR 0488
// §4, which replaces the single bucket constant of issue 1694). The node name must be a name the bus accepts in a
// bucket and in a subject token: letters, digits, '_' and '-', so no '.', '*', '>' or space.
func FileBucketFor(node string) (string, error) {
if !nodeToken.MatchString(node) {
return "", fmt.Errorf("node %q; a node's bucket is named by letters, digits, '_' and '-' only", node)
}
return FileBucketPrefix + node, nil
}
// FileStreamFor is the name of the stream behind a node's bucket, "OBJ_" and the bucket's name.
func FileStreamFor(node string) (string, error) {
b, err := FileBucketFor(node)
if err != nil {
return "", err
}
return FileStreamPrefix + b, nil
}
var digestForm = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
// DigestOf is the digest a reference names for a content: "sha256:<64 lower-case hex>".
func DigestOf(content []byte) string {
sum := sha256.Sum256(content)
return "sha256:" + hex.EncodeToString(sum[:])
}
// ObjectName is the name of the object holding the content a digest names: the hex without its "sha256:" prefix.
func ObjectName(digest string) (string, error) {
if !digestForm.MatchString(digest) {
return "", fmt.Errorf("digest %q is not sha256:<64 lower-case hex characters>", digest)
}
return digest[len("sha256:"):], nil
}
// IsReference reports whether the file gives its content as a digest and size instead of carrying it.
func (f *File) IsReference() bool { return f.Digest != "" }
// checkReference judges the digest and size, and what a reference may not be given with.
func (f *File) checkReference(where string) []string {
if f.Digest == "" && f.Size == 0 {
return nil
}
var problems []string
if !digestForm.MatchString(f.Digest) {
problems = append(problems, fmt.Sprintf("%s: digest %q; a file by reference names its content as "+
"sha256:<64 lower-case hex characters>", where, f.Digest))
}
if f.Size <= 0 {
problems = append(problems, fmt.Sprintf("%s: size %d; a file by reference states its size in bytes, "+
"more than zero", where, f.Size))
}
if f.Content != "" || f.Bytes != "" || f.Sealed != "" || f.VaultSealed != nil || len(f.Secrets) > 0 {
problems = append(problems, where+": a file by reference carries its content in the object its digest "+
"names, so it says no content, bytes, sealed, vault-sealed or secrets")
}
if f.Into != "" {
problems = append(problems, where+": a file by reference is written whole; a file written into says its "+
"part in content")
}
return problems
}
// Body is the bytes the file's content stands for: its content, or its bytes decoded.
func (f *File) Body() ([]byte, error) {
if f.Bytes == "" {
return []byte(f.Content), nil
}
b, err := base64.StdEncoding.DecodeString(f.Bytes)
if err != nil {
return nil, fmt.Errorf("file %s: bytes are not base64: %w", f.ID, err)
}
return b, nil
}
// TravelsByDigest reports whether the controller may put this file's content in the bucket: it is plain content
// or bytes (not sealed, not filled with secrets, not written into another file) of more than FileReferenceThreshold
// bytes.
func (f *File) TravelsByDigest() bool {
if f.IsReference() || f.Sealed != "" || f.VaultSealed != nil || len(f.Secrets) > 0 || f.Into != "" {
return false
}
if f.Bytes != "" {
// DecodedLen counts padding, so the exact length is taken off the decoded bytes.
body, err := f.Body()
return err == nil && len(body) > FileReferenceThreshold
}
return len(f.Content) > FileReferenceThreshold
}
// AsReference is the file as it is sent to an engine that reports FileReferencesCapability: a copy with the digest
// and size in place of the content, and the content the controller puts into the bucket under ObjectName(digest).
// A file that does not travel by digest is refused, not passed through, so a caller cannot send a small file empty.
func (f *File) AsReference() (File, []byte, error) {
if !f.TravelsByDigest() {
return File{}, nil, fmt.Errorf("file %s does not travel by digest", f.ID)
}
body, err := f.Body()
if err != nil {
return File{}, nil, err
}
ref := *f
ref.Content, ref.Bytes = "", ""
ref.Digest, ref.Size = DigestOf(body), int64(len(body))
return ref, body, nil
}
// Verify checks what was fetched against the digest and size the file names; the node-engine writes the file only
// after it returns nil. A mismatch is the apply's failure, never a partial apply (ADR 0486 §1).
func (f *File) Verify(body []byte) error {
if !f.IsReference() {
return fmt.Errorf("file %s is not given by reference", f.ID)
}
if int64(len(body)) != f.Size {
return fmt.Errorf("file %s: the object is %d bytes and the declaration says %d", f.ID, len(body), f.Size)
}
if got := DigestOf(body); got != f.Digest {
return fmt.Errorf("file %s: the object hashes to %s and the declaration names %s", f.ID, got, f.Digest)
}
return nil
}
+158
View File
@@ -0,0 +1,158 @@
package declaration
import (
"bytes"
"encoding/base64"
"encoding/json"
"reflect"
"strings"
"testing"
)
func refFile(extra string) string {
return `{"declaration":1,"resources":[{"id":"m.f","type":"file","path":"/etc/m","digest":"sha256:` +
strings.Repeat("a", 64) + `","size":70000` + extra + `}]}`
}
// 1694/AC1: a reference round trips, and an invalid one is refused naming the field.
func TestAFileReferenceRoundTripsAndABadOneNamesItsField(t *testing.T) {
d, err := Parse([]byte(refFile("")))
if err != nil {
t.Fatal(err)
}
f := d.Resources[0].(*File)
if !f.IsReference() || f.Size != 70000 || f.Digest != "sha256:"+strings.Repeat("a", 64) {
t.Fatalf("read: %+v", f)
}
raw, _ := json.Marshal(d.Resources[0])
d2, err := Parse([]byte(`{"declaration":1,"resources":[` + string(raw) + `]}`))
if err != nil || !reflect.DeepEqual(d2.Resources[0], d.Resources[0]) {
t.Fatalf("round trip: %v %s", err, raw)
}
for want, raw := range map[string]string{
"digest": strings.Replace(refFile(""), strings.Repeat("a", 64), "abc", 1),
"size": strings.Replace(refFile(""), `"size":70000`, `"size":0`, 1),
"lower-case": strings.Replace(refFile(""), strings.Repeat("a", 64), strings.Repeat("A", 64), 1),
"content": refFile(`,"content":"x"`),
"bytes": refFile(`,"bytes":"eA=="`),
"whole": refFile(`,"into":"block"`),
} {
_, err := Parse([]byte(raw))
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("want a refusal naming %q, got %v", want, err)
}
}
// a size alone is not a reference
if _, err := Parse([]byte(`{"declaration":1,"resources":[{"id":"m.f","type":"file","path":"/e","size":5}]}`)); err == nil {
t.Error("a size without a digest was accepted")
}
}
// 1694/AC2: a declaration as today's SDK writes it reads unchanged and its bytes carry no new key.
func TestAnInlineFileIsReadAndWrittenAsBefore(t *testing.T) {
const in = `{"id":"m.f","type":"file","path":"/e","content":"hello"}`
d, err := Parse([]byte(`{"declaration":1,"resources":[` + in + `]}`))
if err != nil {
t.Fatal(err)
}
out, _ := json.Marshal(d.Resources[0])
if string(out) != in {
t.Fatalf("signed bytes changed: %s", out)
}
if d.Resources[0].(*File).IsReference() {
t.Fatal("inline read as a reference")
}
}
// 1694/AC3: the controller's helper makes a reference of a big file and the engine's check refuses a changed byte.
func TestABigFileBecomesAReferenceAndAChangedByteFailsTheCheck(t *testing.T) {
big := strings.Repeat("x", FileReferenceThreshold+1)
f := &File{ID: "m.f", Type: TypeFile, Path: "/e", Content: big}
if !f.TravelsByDigest() {
t.Fatal("a file over the threshold stays inline")
}
ref, body, err := f.AsReference()
if err != nil || ref.Content != "" || ref.Size != int64(len(big)) || string(body) != big {
t.Fatalf("%v %+v", err, ref)
}
if p := ref.validate("f", false); len(p) > 0 {
t.Fatalf("the reference is refused: %v", p)
}
if name, err := ObjectName(ref.Digest); err != nil || len(name) != 64 {
t.Fatalf("object name %q %v", name, err)
}
if err := ref.Verify(body); err != nil {
t.Fatal(err)
}
bad := bytes.Clone(body)
bad[7] ^= 1
if err := ref.Verify(bad); err == nil || !strings.Contains(err.Error(), "hashes to") {
t.Fatalf("a changed byte: %v", err)
}
if err := ref.Verify(body[1:]); err == nil || !strings.Contains(err.Error(), "bytes") {
t.Fatalf("a short object: %v", err)
}
}
// 1694/AC4: what stays inline, and bytes files travel decoded.
func TestWhichFilesTravelByDigest(t *testing.T) {
small := strings.Repeat("x", FileReferenceThreshold)
big := strings.Repeat("x", FileReferenceThreshold+1)
for name, f := range map[string]File{
"at the threshold": {Content: small},
"sealed": {Sealed: "s"},
"secrets": {Content: big, Secrets: map[string]string{"a": "b"}},
"into": {Content: big, Into: IntoBlock},
} {
if f.TravelsByDigest() {
t.Errorf("%s travels by digest", name)
}
if _, _, err := f.AsReference(); err == nil {
t.Errorf("%s was made a reference", name)
}
}
exact := File{ID: "m.e", Bytes: base64.StdEncoding.EncodeToString(bytes.Repeat([]byte{7}, FileReferenceThreshold))}
if exact.TravelsByDigest() {
t.Error("a bytes file of exactly the threshold travels by digest")
}
if _, _, err := exact.AsReference(); err == nil {
t.Error("a bytes file of exactly the threshold was made a reference")
}
raw := bytes.Repeat([]byte{0, 1, 2}, FileReferenceThreshold)
f := File{ID: "m.b", Bytes: base64.StdEncoding.EncodeToString(raw)}
ref, body, err := f.AsReference()
if err != nil || !bytes.Equal(body, raw) || ref.Size != int64(len(raw)) || ref.Digest != DigestOf(raw) {
t.Fatalf("bytes file: %v", err)
}
}
func TestTheSharedNames(t *testing.T) {
if FileBucketPrefix != "declaration-files-" || FileReferencesCapability != "file-references" || FileReferenceThreshold != 65536 {
t.Fatal("a name changed")
}
if _, err := ObjectName("sha256:zz"); err == nil {
t.Fatal("bad digest named an object")
}
}
// 1694/AC6: each node has a bucket of its own, so a grant names it and never needs a `*` (ADR 0486 §2).
func TestEachNodeHasABucketOfItsOwn(t *testing.T) {
for _, n := range []string{"ace", "g14", "novox", "shanks"} {
b, err := FileBucketFor(n)
if err != nil || b != "declaration-files-"+n {
t.Errorf("bucket for %q: %q %v", n, b, err)
}
s, err := FileStreamFor(n)
if err != nil || s != "OBJ_declaration-files-"+n {
t.Errorf("stream for %q: %q %v", n, s, err)
}
}
for _, bad := range []string{"", "a.b", "a*", "*", ">", "a b", "a>", "a/b", "ä"} {
if _, err := FileBucketFor(bad); err == nil {
t.Errorf("bucket for %q accepted", bad)
}
if _, err := FileStreamFor(bad); err == nil {
t.Errorf("stream for %q accepted", bad)
}
}
}
+55
View File
@@ -0,0 +1,55 @@
package declaration
import (
"encoding/json"
"strings"
"testing"
)
// A container may be given supplementary groups (novox/hq ADR 0480 clause 7, issue 1636): numbers, each above zero.
// 1636/AC1: a declaration with groups decodes, is written back with them, and a declaration without them has none.
func TestAContainerWithGroupsDecodesAndIsWrittenBack(t *testing.T) {
d, err := Parse([]byte(`{"declaration":1,"resources":[
{"id":"n8n.server","type":"container","name":"n8n","image":"` + anImage + `","groups":[2000,2001]},
{"id":"n8n.plain","type":"container","name":"plain","image":"` + anImage + `"}]}`))
if err != nil {
t.Fatal(err)
}
c := d.Resources[0].(*Container)
if len(c.Groups) != 2 || c.Groups[0] != 2000 || c.Groups[1] != 2001 {
t.Fatalf("groups read as %v", c.Groups)
}
raw, err := json.Marshal(c)
if err != nil || !strings.Contains(string(raw), `"groups":[2000,2001]`) {
t.Fatalf("written as %s (%v)", raw, err)
}
plain, _ := json.Marshal(d.Resources[1].(*Container))
if strings.Contains(string(plain), "groups") {
t.Fatalf("a container without groups is written with the key: %s", plain)
}
if g := d.Resources[1].(*Container).Groups; g != nil {
t.Fatalf("a container without groups reads %v", g)
}
}
// 1636/AC2: zero, a number below it, one the kernel keeps for "no group" and a number twice are refused, naming why.
func TestAContainersGroupsAreWholeNumbersAboveZeroAndNotRepeated(t *testing.T) {
for want, groups := range map[string]string{
"never root": `[0]`,
"below zero": `[-5]`,
"no group": `[4294967295]`,
"twice": `[2000,2000]`,
} {
_, err := Parse([]byte(`{"declaration":1,"resources":[{"id":"m.c","type":"container","name":"c","image":"` +
anImage + `","groups":` + groups + `}]}`))
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("groups %s: want a refusal saying %q, got %v", groups, want, err)
}
}
// Text where a number stands is refused by the decoder itself.
if _, err := Parse([]byte(`{"declaration":1,"resources":[{"id":"m.c","type":"container","name":"c","image":"` +
anImage + `","groups":["media"]}]}`)); err == nil {
t.Error("a group named by text was accepted")
}
}
+10
View File
@@ -20,6 +20,16 @@ type BuildRecord struct {
At time.Time `json:"at,omitzero"`
SourceFingerprint string `json:"source-fingerprint,omitempty"`
SourceInputs *SourceInputs `json:"source-inputs,omitempty"`
// Made is what the build made (Build.Made, records.go:110); the digest is the tail of an artifact's Reference.
Made []Artifact `json:"made,omitempty"`
}
// Artifact is one thing a build made: mesh-controller internal/chain/records.go Artifact, json keys name, kind and
// reference, nothing else (novox/hq ADR 0478 clause 8).
type Artifact struct {
Name string `json:"name"`
Kind string `json:"kind"`
Reference string `json:"reference"`
}
// BuildsBucket is the bucket the builds are kept in: mesh-controller internal/broker/controller_buckets.go:101,
+26
View File
@@ -127,3 +127,29 @@ func TestBuildKeys(t *testing.T) { // 1223/AC3
t.Errorf("NewestKey = %q", got)
}
}
// The fixture's made artifacts are the controller's Artifact (mesh-controller internal/chain/records.go, json tags name,
// kind, reference; the controller writes no digest key, the digest is the tail of the reference). The kinds and references
// are those of the live record mesh-controller.builds printed for build-agent build-1792540091397811913 (2026-10-21 01:49).
func TestBuildRecordMade(t *testing.T) { // 1588/AC2
var r BuildRecord
if err := json.Unmarshal(loadBuildFixture(t), &r); err != nil {
t.Fatal(err)
}
want := []Artifact{
{Name: "server", Kind: "image", Reference: "artifact-store://build-agent/server@sha256:a79c13f1c727ff94959e80824cc704ea58b80d80ecdd00fd447698acebb35090"},
{Name: "files", Kind: "archive", Reference: "artifact-store://build-agent/files/blobs/sha256:bccd3f47851436d1b45a3063cc2d7e34c5c109b2d030739bfbb4b9781703f9a9"},
}
if len(r.Made) != len(want) {
t.Fatalf("made = %+v, want %+v", r.Made, want)
}
for i := range want {
if r.Made[i] != want[i] {
t.Errorf("made[%d] = %+v, want %+v", i, r.Made[i], want[i])
}
}
var none BuildRecord
if err := json.Unmarshal([]byte(`{"id":"b","repository":"a/b"}`), &none); err != nil || none.Made != nil {
t.Errorf("a record without made: %+v, %v", none, err)
}
}
+88
View File
@@ -0,0 +1,88 @@
package buildrun
import (
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ChooseNodes picks up to want build nodes for a language (ADR 0478 clause 3): the nodes whose toolchains key reports the
// version the nodes agree on, the ones with room under their build capacity first (places), then the fewest lanes the
// part has running or chosen there, a tie broken by the node's name. A place is reserved on each node chosen, in the
// same step as the choice, so builds choosing at once do not choose the same full node while another has room; the
// caller takes the place (Places.Acquire) or gives it up (Places.Cancel). A node at its capacity is still chosen when
// the others are too (fewer than want have room): its lane waits for a place. A node qualifies only when qualifies is true
// of it (online and its build module healthy; nil means every node does). It answers the toolchain all the reporting nodes
// agree on, qualifying or not, as the indexer computes it. When nothing is reported, or the nodes report different
// versions (so the indexer agreed none), it fails naming the language and why; when no reporting node qualifies it fails
// naming the nodes. A node whose build module refuses on its toolchain refuses the step, and the build fails naming it.
func ChooseNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, delivery.Toolchain2, error) {
tc, none := delivery.AgreedToolchain(entries)
if none != "" {
return nil, delivery.Toolchain2{}, fmt.Errorf("no %s build node qualifies: %s", language, none)
}
nodes, err := choose(language, entries, places, qualifies, want)
return nodes, tc, err
}
// ChooseAnyNodes is ChooseNodes for artifacts that need no toolchain, such as an archive (ADR 0478 clause 1): the
// build module's nodes agree on nothing, so any node of the build module that qualifies is chosen, by capacity and load only.
func ChooseAnyNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
return choose(language, entries, places, qualifies, want)
}
func choose(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
if places == nil {
places = &Places{}
}
nodes := make([]string, 0, len(entries))
var reporting []string
seen := map[string]bool{}
for _, e := range entries {
if seen[e.Node] {
continue
}
seen[e.Node] = true
reporting = append(reporting, e.Node)
if places.CapacityOf(e.Node) > 0 && (qualifies == nil || qualifies(e.Node)) { // a capacity of 0 keeps the part off a node
nodes = append(nodes, e.Node)
}
}
if len(nodes) == 0 {
sort.Strings(reporting)
return nil, fmt.Errorf("no %s build node qualifies: %s report the toolchain, and none is online with a healthy build module and a build capacity", language, strings.Join(reporting, ", "))
}
return places.choose(nodes, want), nil
}
// DigestsDifferPrefix opens the error of a build whose nodes built an artifact to different digests (ADR 0478 clause 4):
// such a key is not built again as a new revision, since the same source on the same toolchains gave two answers and
// building again settles nothing; it waits for another fingerprint.
const DigestsDifferPrefix = "digests differ:"
// Agree is the one digest every node built an artifact to (ADR 0478 clause 4), or an error naming each node and its
// digest when they differ. One node's digest agrees with itself.
func Agree(artifact string, byNode map[string]string) (string, error) {
nodes := make([]string, 0, len(byNode))
for n := range byNode {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var first string
for i, n := range nodes {
if i == 0 {
first = byNode[n]
continue
}
if byNode[n] != first {
msg := fmt.Sprintf("%s the nodes built %s to different digests:", DigestsDifferPrefix, artifact)
for _, n := range nodes {
msg += fmt.Sprintf(" %s %s;", n, byNode[n])
}
return "", fmt.Errorf("%s", msg[:len(msg)-1])
}
}
return first, nil
}
+99
View File
@@ -0,0 +1,99 @@
package buildrun
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const digest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
func hex64(c byte) string { return strings.Repeat(string(c), 64) }
func tcs(language, version string, nodes ...string) []delivery.NodeToolchain {
var out []delivery.NodeToolchain
for _, n := range nodes {
out = append(out, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: language, Version: version, Recipe: "1"}})
}
return out
}
func TestChooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName(t *testing.T) { // 1607/AC4, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "novox", "ace", "g14", "shanks")
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 3, "ace": 1, "g14": 1, "shanks": 0}), nil, 2)
if err != nil {
t.Fatal(err)
}
if len(nodes) != 2 || nodes[0] != "shanks" || nodes[1] != "ace" {
t.Errorf("nodes = %v, want [shanks ace]: fewest steps first, a tie by name", nodes)
}
if tc.Version != "go1.27.1" || tc.Recipe != "1" || tc.Language != "go" {
t.Errorf("toolchain = %+v", tc)
}
}
func TestChooseWithOneNodeAnswersOneNode(t *testing.T) { // 1607/AC4
nodes, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "novox"), nil, nil, 2)
if err != nil || len(nodes) != 1 || nodes[0] != "novox" {
t.Errorf("nodes = %v, err %v, want [novox]", nodes, err)
}
}
func TestChooseFailsNamingTheLanguageAndTheVersionsWhenNoNodeQualifies(t *testing.T) { // 1607/AC4
_, _, err := ChooseNodes("typescript", nil, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "typescript") || !strings.Contains(err.Error(), delivery.ToolchainNoReport) {
t.Errorf("no report: error = %v, want it to name typescript and %q", err, delivery.ToolchainNoReport)
}
entries := append(tcs("go", "go1.27.1", "novox"), tcs("go", "go1.26.0", "ace")...)
_, _, err = ChooseNodes("go", entries, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), "go1.26.0") || !strings.Contains(err.Error(), "go") {
t.Errorf("disagreeing nodes: error = %v, want both versions named", err)
}
}
func TestAgreeNeedsEqualDigestsAndNamesBothNodesWhenTheyDiffer(t *testing.T) { // 1607/AC4, ADR 0478 clause 4
d, err := Agree("tool", map[string]string{"ace": digest, "novox": digest})
if err != nil || d != digest {
t.Errorf("Agree = %q, %v, want %q", d, err, digest)
}
_, err = Agree("tool", map[string]string{"ace": digest, "novox": "sha256:" + hex64('9')})
if err == nil || !strings.Contains(err.Error(), "ace") || !strings.Contains(err.Error(), "novox") || !strings.Contains(err.Error(), digest) || !strings.Contains(err.Error(), hex64('9')) || !strings.Contains(err.Error(), "tool") {
t.Errorf("differing digests: error = %v, want the artifact, both nodes and both digests", err)
}
if d, err := Agree("tool", map[string]string{"ace": digest}); err != nil || d != digest {
t.Errorf("one node: %q, %v", d, err)
}
}
func TestChooseSkipsNodesThatAreNotOnlineAndFailsNamingThemWhenNoneIs(t *testing.T) { // 1607 review 3, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "ace", "g14", "novox")
online := func(n string) bool { return n != "g14" } // a sleeping laptop
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 5}), online, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "ace" || nodes[1] != "novox" || tc.Version != "go1.27.1" {
t.Errorf("nodes = %v, toolchain %+v, err %v, want [ace novox] with g14 left out although it runs nothing", nodes, tc, err)
}
_, tc, err = ChooseNodes("go", entries, nil, func(string) bool { return false }, 2)
if err == nil || !strings.Contains(err.Error(), "online") || !strings.Contains(err.Error(), "g14") || !strings.Contains(err.Error(), "go") {
t.Errorf("none online: error = %v, want it to name the nodes that report and that none is online", err)
}
if tc.Version != "go1.27.1" {
t.Errorf("none online: the agreed toolchain %+v is still answered", tc)
}
}
// placesRunning is Places with these steps counted as running, for a test of the choice.
func placesRunning(running map[string]int) *Places {
return &Places{running: running, changed: make(chan struct{})}
}
func TestChoosePrefersNodesBelowTheirCapacity(t *testing.T) { // round 3
entries := tcs("go", "go1.27.1", "ace", "novox", "shanks")
// shanks has capacity 3 with 2 running: room; ace has the default 1 with 1 running: none; novox has none running.
p := placesRunning(map[string]int{"shanks": 2, "ace": 1})
p.Capacity = map[string]int{"shanks": 3}
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "novox" || nodes[1] != "shanks" {
t.Errorf("nodes = %v, err %v, want [novox shanks]: the nodes with room, ace at capacity left out", nodes, err)
}
}
+5
View File
@@ -0,0 +1,5 @@
// Package buildrun holds the parts every module that runs steps on the build modules shares: the caller, the steps
// watch and the runner of a long step (ADR 0469), the sets of presence, toolchains and healthy build modules, the
// places a node offers by capacity, and the choice of build nodes (ADR 0478). The builder and the checker of the
// delivery use them; they moved here from the builder's own package so neither keeps a copy (issue 1671, ADR 0484).
package buildrun
+56
View File
@@ -0,0 +1,56 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HaveSet holds what each node's node-engine says of each module on it (node-engine_have, key `<node>.<module>`): the
// builder reads it to know whether a node's build module is running and healthy, since a build module whose installed
// toolchain is not the one its recipe builds with is unhealthy (mesh-build-go build_go_health) and its steps would
// refuse (ADR 0478 clause 3). A module the node-engine has written no key of is not healthy. Safe for use from the watch
// and from a build at once.
type HaveSet struct {
mu sync.Mutex
healthy map[string]bool // by `<node>.<module>`
}
// Put takes a change of the have bucket in and answers whether a module's healthy state changed. A node's own key
// (`<node>.node`) and a value that is not a have are left out.
func (s *HaveSet) Put(key string, value []byte, deleted bool) bool {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
return false
}
ok := false
if !deleted {
h, err := delivery.DecodeHave(value)
if err != nil {
return false
}
ok = h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK
}
k := node + "." + module
s.mu.Lock()
defer s.mu.Unlock()
if s.healthy[k] == ok {
return false
}
if s.healthy == nil {
s.healthy = map[string]bool{}
}
if ok {
s.healthy[k] = true
} else {
delete(s.healthy, k)
}
return true
}
// Healthy is whether the module on the node was last seen running and healthy.
func (s *HaveSet) Healthy(node, module string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.healthy[node+"."+module]
}
+7
View File
@@ -0,0 +1,7 @@
package buildrun
// The build modules the mesh runs, by the language each builds (ADR 0469).
const (
GoBuildModule = "mesh-build-go"
NodeBuildModule = "mesh-build-node"
)
+171
View File
@@ -0,0 +1,171 @@
package buildrun
import (
"context"
"fmt"
"sort"
"strconv"
"strings"
"sync"
)
// DefaultCapacity is the build capacity of a node that has no setting: one build step at a time.
const DefaultCapacity = 1
// ParseCapacity reads a capacity setting: `node=steps` pairs separated by commas, such as
// `shanks=3,ace=2`. A node without a pair has DefaultCapacity. A pair with no node, a count that is not a whole number of
// at least 0, and a node named twice are refused. A count of 0 keeps the part off the node: it is never chosen.
func ParseCapacity(s string) (map[string]int, error) {
out := map[string]int{}
for _, pair := range strings.Split(s, ",") {
pair = strings.TrimSpace(pair)
if pair == "" {
continue
}
node, count, ok := strings.Cut(pair, "=")
node, count = strings.TrimSpace(node), strings.TrimSpace(count)
n, err := strconv.Atoi(count)
if !ok || node == "" || err != nil || n < 0 {
return nil, fmt.Errorf("the capacity %q is not <node>=<steps at once, 0 or more>", pair)
}
if _, dup := out[node]; dup {
return nil, fmt.Errorf("the capacity names %s twice", node)
}
out[node] = n
}
return out, nil
}
// Places counts the build steps a part runs on each node and holds a lane back while its node is at its
// capacity (the operator's decision of 2026-10-21; ADR 0478 clause 3). A lane is the fetch and the builds of one build
// module on one node, run one step at a time, so a lane holds one place while it runs. A node is chosen for a lane by
// reserving a place on it at the moment of the choice (ChooseNodes), so that builds choosing at once see each other's
// choices and none waits on a full node while another has room. A lane waits for its place by watching the ends of the
// other lanes, not by a timer: ending a lane wakes the waiting ones.
type Places struct {
// Capacity is the number of steps at once by node; a node not in it has DefaultCapacity.
Capacity map[string]int
mu sync.Mutex
running map[string]int // lanes holding a place
reserved map[string]int // lanes chosen for the node that do not hold a place yet
changed chan struct{} // closed, and replaced, each time a lane ends
}
// CapacityOf is how many steps the part runs on the node at once: DefaultCapacity for a node the setting does not
// name, and 0 for one it names so, where the part never builds.
func (p *Places) CapacityOf(node string) int {
if n, ok := p.Capacity[node]; ok && n >= 0 {
return n
}
return DefaultCapacity
}
// Running is the steps in flight now, by node.
func (p *Places) Running() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.running))
for n, c := range p.running {
if c > 0 {
out[n] = c
}
}
return out
}
// Reserved is the lanes chosen for a node that hold no place yet, by node.
func (p *Places) Reserved() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.reserved))
for n, c := range p.reserved {
if c > 0 {
out[n] = c
}
}
return out
}
func (p *Places) init() {
if p.running == nil {
p.running = map[string]int{}
}
if p.reserved == nil {
p.reserved = map[string]int{}
}
if p.changed == nil {
p.changed = make(chan struct{})
}
}
// Acquire takes a place on the node, waiting while the node is at its capacity, and answers the function that gives it
// back. A place reserved for the node by a choice is the one taken. It returns an error when ctx ends first, and then
// gives up the reservation too.
func (p *Places) Acquire(ctx context.Context, node string) (release func(), err error) {
for {
p.mu.Lock()
p.init()
if p.running[node] < p.CapacityOf(node) {
p.running[node]++
if p.reserved[node] > 0 {
p.reserved[node]--
}
p.mu.Unlock()
var once sync.Once
return func() {
once.Do(func() {
p.mu.Lock()
p.running[node]--
close(p.changed)
p.changed = make(chan struct{})
p.mu.Unlock()
})
}, nil
}
wait := p.changed
p.mu.Unlock()
select {
case <-wait:
case <-ctx.Done():
p.Cancel(node)
return nil, fmt.Errorf("waiting for a place on %s: %w", node, ctx.Err())
}
}
}
// Cancel gives up a place reserved for the node that will not be taken.
func (p *Places) Cancel(node string) {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
if p.reserved[node] > 0 {
p.reserved[node]--
}
}
// choose orders the nodes (those with room first, counting the lanes running and the lanes reserved, then the fewest of
// them, then by name), keeps the first want and reserves a place on each, all at once.
func (p *Places) choose(nodes []string, want int) []string {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
load := func(n string) int { return p.running[n] + p.reserved[n] }
sort.SliceStable(nodes, func(i, j int) bool {
ri, rj := load(nodes[i]) < p.CapacityOf(nodes[i]), load(nodes[j]) < p.CapacityOf(nodes[j])
if ri != rj {
return ri
}
if load(nodes[i]) != load(nodes[j]) {
return load(nodes[i]) < load(nodes[j])
}
return nodes[i] < nodes[j]
})
if len(nodes) > want {
nodes = nodes[:want]
}
for _, n := range nodes {
p.reserved[n]++
}
return nodes
}
+126
View File
@@ -0,0 +1,126 @@
package buildrun
import (
"context"
"strings"
"testing"
"time"
)
func TestParseCapacityReadsNodeEqualsCountPairs(t *testing.T) {
c, err := ParseCapacity(" shanks=3, ace=2 ")
if err != nil || c["shanks"] != 3 || c["ace"] != 2 || len(c) != 2 {
t.Errorf("capacity = %v, %v", c, err)
}
if c, err := ParseCapacity(""); err != nil || len(c) != 0 {
t.Errorf("empty: %v, %v", c, err)
}
for _, bad := range []string{"shanks", "shanks=x", "=2", "a=1,a=2", "shanks=-1"} {
if _, err := ParseCapacity(bad); err == nil {
t.Errorf("%q accepted", bad)
}
}
}
func acquireAsync(p *Places, node string) chan func() {
got := make(chan func(), 1)
go func() {
release, err := p.Acquire(context.Background(), node)
if err != nil {
close(got)
return
}
got <- release
}()
return got
}
func TestThreeStepsOnANodeOfCapacityTwoRunTwoAtOnceAndTheThirdStartsWhenOneEnds(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"ace": 2}}
a, b := acquireAsync(p, "ace"), acquireAsync(p, "ace")
ra, rb := <-a, <-b
third := acquireAsync(p, "ace")
select {
case <-third:
t.Fatal("a third step started on a node of capacity 2 with two running")
case <-time.After(10 * time.Millisecond):
}
if p.Running()["ace"] != 2 {
t.Errorf("running = %v, want 2 on ace", p.Running())
}
ra()
select {
case r := <-third:
r()
case <-time.After(2 * time.Second):
t.Fatal("the third step did not start when one ended")
}
rb()
}
func TestANodeWithNoSettingRunsOneStepAtATime(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"shanks": 3}}
first := <-acquireAsync(p, "g14")
second := acquireAsync(p, "g14")
select {
case <-second:
t.Fatal("two steps on a node without a setting")
case <-time.After(10 * time.Millisecond):
}
first()
r := <-second
r()
if p.CapacityOf("g14") != 1 || p.CapacityOf("shanks") != 3 {
t.Errorf("capacities = %d, %d", p.CapacityOf("g14"), p.CapacityOf("shanks"))
}
}
func TestAWaitingStepStopsWhenItsContextEnds(t *testing.T) { // round 3
p := &Places{}
r := <-acquireAsync(p, "ace")
defer r()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { _, err := p.Acquire(ctx, "ace"); done <- err }()
cancel()
select {
case err := <-done:
if err == nil {
t.Error("Acquire returned no error for a cancelled wait")
}
case <-time.After(2 * time.Second):
t.Fatal("a cancelled wait did not return")
}
}
// The operator's decision of 2026-10-21 13:40: the builder builds on shanks and ace only; a capacity of 0 keeps it off a node.
func TestACapacityOfZeroKeepsThePartOffTheNodeAndANodeNotNamedKeepsOne(t *testing.T) { // issue 1661
c, err := ParseCapacity("shanks=3,ace=2,g14=0,novox=0")
if err != nil || c["g14"] != 0 || c["novox"] != 0 || c["shanks"] != 3 || len(c) != 4 {
t.Fatalf("capacity = %v, %v", c, err)
}
p := &Places{Capacity: c}
if p.CapacityOf("g14") != 0 || p.CapacityOf("novox") != 0 || p.CapacityOf("ace") != 2 || p.CapacityOf("other") != DefaultCapacity {
t.Errorf("capacities = %d %d %d %d, want 0 0 2 and the default for a node not named", p.CapacityOf("g14"), p.CapacityOf("novox"), p.CapacityOf("ace"), p.CapacityOf("other"))
}
if _, err := ParseCapacity("shanks=-1"); err == nil {
t.Error("a negative capacity accepted")
}
entries := tcs("go", "go1.27.1", "ace", "g14", "novox", "shanks", "other")
for i := 0; i < 6; i++ {
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil {
t.Fatal(err)
}
for _, n := range nodes {
if n == "g14" || n == "novox" {
t.Fatalf("choice %d: %v includes a node of capacity 0", i, nodes)
}
}
}
// Only nodes of capacity 0 report: nothing qualifies, and the build says why.
_, _, err = ChooseNodes("go", tcs("go", "go1.27.1", "g14", "novox"), p, nil, 2)
if err == nil || !strings.Contains(err.Error(), "capacity") {
t.Errorf("only capacity-0 nodes: error = %v, want it to name the capacity", err)
}
}
+53
View File
@@ -0,0 +1,53 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// PresenceSet holds which nodes are online, as the watchman writes it (mesh-delivery-watchman_presence, key `<node>`, the
// node-engine's connection): a node asleep or off the bus is offline and its build module cannot answer. A node the
// watchman has no key of is not online. Safe for use from the watch and from a build at once.
type PresenceSet struct {
mu sync.Mutex
online map[string]bool
}
// Put takes a change of the presence bucket in and answers whether a node's online state changed. A part's key and a
// value that is not a presence are left out.
func (s *PresenceSet) Put(key string, value []byte, deleted bool) bool {
node, part, _, err := delivery.ParsePresenceKey(key)
if err != nil || part != "" {
return false
}
on := false
if !deleted {
p, err := delivery.DecodePresence(value)
if err != nil {
return false
}
on = p.Online
}
s.mu.Lock()
defer s.mu.Unlock()
if s.online[node] == on {
return false
}
if s.online == nil {
s.online = map[string]bool{}
}
if on {
s.online[node] = true
} else {
delete(s.online, node)
}
return true
}
// Online is whether the watchman last saw the node online.
func (s *PresenceSet) Online(node string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.online[node]
}
+43
View File
@@ -0,0 +1,43 @@
package buildrun
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func presenceValue(t *testing.T, online bool) []byte {
t.Helper()
b, err := delivery.EncodePresence(delivery.Presence{Online: online})
if err != nil {
t.Fatal(err)
}
return b
}
func TestPresenceSetKnowsOnlyNodesTheWatchmanSawOnline(t *testing.T) { // 1607 review 3
var s PresenceSet
if s.Online("ace") {
t.Error("a node the watchman never wrote is online")
}
if !s.Put("ace", presenceValue(t, true), false) || !s.Online("ace") {
t.Error("ace online was not taken in")
}
if s.Put("ace", presenceValue(t, true), false) {
t.Error("the same presence again counted as a change")
}
if !s.Put("ace", presenceValue(t, false), false) || s.Online("ace") {
t.Error("ace going offline was not taken in")
}
s.Put("g14", presenceValue(t, true), false)
if !s.Put("g14", nil, true) || s.Online("g14") {
t.Error("a deleted key is still online")
}
// A part's instance (part.<part>.<instance>) is not a node.
if s.Put("part.mesh-delivery-builder.novox", presenceValue(t, true), false) || s.Online("part") {
t.Error("a part's key was taken for a node")
}
if s.Put("novox", []byte("junk"), false) {
t.Error("junk counted")
}
}
+67
View File
@@ -0,0 +1,67 @@
package buildrun
import (
"context"
"encoding/json"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Caller calls a tool of a module on one named machine (mesh-sdk process.Conn.CallOn; ADR 0469 clause 3).
type Caller interface {
CallOn(node, key string, body any) (json.RawMessage, error)
}
// Steps waits for the end of a build module's long step: the value of its key `<node>.<id>` in the module's steps state
// (ADR 0469 clause 2) once its state is not running. It is a watch of that key, not a poll; it returns when the step
// ends or ctx does.
type Steps interface {
Wait(ctx context.Context, module, key string) (delivery.Step, error)
}
// Runner runs a build module's long steps: the call answers at once with the step's key, and the result is read from that
// key when the step ends (ADR 0469 clause 2).
type Runner struct {
Caller Caller
Steps Steps
// Bound is how long a step may run before the builder stops waiting for it.
Bound time.Duration
}
// Run calls tool of the build module on node with args, waits for the step's end and answers its result. A step that
// failed or was refused is an error naming the node, the state and the reason.
func (r Runner) Run(ctx context.Context, module, node, tool string, args map[string]any) (json.RawMessage, error) {
raw, err := r.Caller.CallOn(node, module+"."+tool, args)
if err != nil {
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
var answer struct {
Step string `json:"step"`
Key string `json:"key"`
State string `json:"state"`
}
if err := json.Unmarshal(raw, &answer); err != nil {
return nil, fmt.Errorf("%s on %s answered what is not a step: %w", tool, node, err)
}
if answer.Key == "" {
return nil, fmt.Errorf("%s on %s answered no step key: %s", tool, node, raw)
}
if n, _, err := delivery.ParseStepKey(answer.Key); err != nil || n != node {
return nil, fmt.Errorf("%s on %s answered the step key %q, which is not a key of that node", tool, node, answer.Key)
}
wait, cancel := context.WithTimeoutCause(ctx, r.Bound, fmt.Errorf("the step %s did not end within %s", answer.Key, r.Bound))
defer cancel()
step, err := r.Steps.Wait(wait, module, answer.Key)
if err != nil {
if cause := context.Cause(wait); cause != nil && ctx.Err() == nil {
err = cause
}
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
if step.State != delivery.StepPassed {
return nil, fmt.Errorf("%s on %s: the step %s %s: %s", tool, node, answer.Key, step.State, step.Reason)
}
return step.Result, nil
}
+116
View File
@@ -0,0 +1,116 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
type call struct {
node, key string
body any
}
// fakeCaller answers a long tool as the build modules do: at once, with the step and its key.
type fakeCaller struct {
calls []call
answer string
err error
}
func (f *fakeCaller) CallOn(node, key string, body any) (json.RawMessage, error) {
f.calls = append(f.calls, call{node, key, body})
if f.err != nil {
return nil, f.err
}
return json.RawMessage(f.answer), nil
}
// fakeSteps holds the steps values by `<module>/<key>`; Wait on a key it does not hold blocks until the context ends.
type fakeSteps struct {
mu sync.Mutex
steps map[string]delivery.Step
// during, when set, is called as a wait starts and the function it answers as the wait ends.
during func(key string) func()
}
func (f *fakeSteps) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
f.mu.Lock()
s, ok := f.steps[module+"/"+key]
during := f.during
f.mu.Unlock()
if ok {
if during != nil {
defer during(key)()
}
return s, nil
}
<-ctx.Done()
return delivery.Step{}, context.Cause(ctx)
}
func stepAt(node, id, state, reason string, result string) delivery.Step {
s := delivery.Step{ID: id, Tool: "build_go_fetch", Node: node, State: state, Reason: reason}
if result != "" {
s.Result = json.RawMessage(result)
}
return s
}
func TestRunCallsTheToolOnTheNodeAndReadsTheStepsKey(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", delivery.StepPassed, "", `{"workspace":"w1","commit":"c"}`)}}
res, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", map[string]any{"repository": "novox/r", "commit": "c"})
if err != nil {
t.Fatal(err)
}
if len(caller.calls) != 1 || caller.calls[0].node != "ace" || caller.calls[0].key != "mesh-build-go.build_go_fetch" {
t.Errorf("calls = %+v, want one on ace for mesh-build-go.build_go_fetch", caller.calls)
}
var fr struct{ Workspace string }
if err := json.Unmarshal(res, &fr); err != nil || fr.Workspace != "w1" {
t.Errorf("result = %s (%v), want the step's result", res, err)
}
}
func TestRunFailsOnAFailedOrRefusedStepNamingItsReason(t *testing.T) { // 1607/AC5
for _, state := range []string{delivery.StepFailed, delivery.StepRefused} {
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", state, "go is go1.26.0, the recipe builds with go1.27.1", "")}}
_, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), state) || !strings.Contains(err.Error(), "ace") {
t.Errorf("%s: error = %v, want the node, the state and the reason", state, err)
}
}
}
func TestRunStopsWaitingAtTheBound(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
_, err := Runner{Caller: caller, Steps: &fakeSteps{}, Bound: 20 * time.Millisecond}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "ace.abc1") || !strings.Contains(err.Error(), "did not end") {
t.Errorf("error = %v, want the step key and that it did not end within its bound", err)
}
}
func TestRunRefusesAnAnswerWithNoKeyAndAKeyOfAnotherNode(t *testing.T) { // 1607/AC5
for _, answer := range []string{`{"state":"running"}`, `{"step":"abc1","key":"novox.abc1","state":"running"}`, `not json`} {
_, err := Runner{Caller: &fakeCaller{answer: answer}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if err == nil {
t.Errorf("answer %s: no error", answer)
}
}
}
func TestRunReturnsTheCallsOwnError(t *testing.T) { // 1607/AC5
boom := errors.New("no responders")
_, err := Runner{Caller: &fakeCaller{err: boom}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if !errors.Is(err, boom) || !strings.Contains(err.Error(), "ace") {
t.Errorf("error = %v, want the call's error naming the node", err)
}
}
+64
View File
@@ -0,0 +1,64 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// LanguageOf is the language each build module's toolchains state holds (the indexer's ToolchainModules, reversed).
var LanguageOf = map[string]string{GoBuildModule: "go", NodeBuildModule: "typescript"}
// ToolchainSet holds what the build modules' toolchains buckets hold: each node's entry of a language, as the watches
// put them in. Safe for use from the watches and from a build at once.
type ToolchainSet struct {
mu sync.Mutex
by map[string]map[string]delivery.Toolchain // language -> node -> entry
}
// Put takes a change of a build module's toolchains bucket in: a put of a node's entry (key <node>.<language>) or its
// delete. It answers whether the language's entries changed (a node's version or recipe, or a node appearing or going).
// A key or a value that is not an entry of the module's language is left out.
func (s *ToolchainSet) Put(module, key string, value []byte, deleted bool) bool {
language, ok := LanguageOf[module]
if !ok {
return false
}
node, keyLanguage, err := delivery.ParseToolchainKey(key)
if err != nil || keyLanguage != language {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if deleted {
if _, had := s.by[language][node]; !had {
return false
}
delete(s.by[language], node)
return true
}
tc, err := delivery.DecodeToolchain(value)
if err != nil || tc.Language != language {
return false
}
if s.by == nil {
s.by = map[string]map[string]delivery.Toolchain{}
}
if s.by[language] == nil {
s.by[language] = map[string]delivery.Toolchain{}
}
cur, had := s.by[language][node]
s.by[language][node] = tc
return !had || cur.Version != tc.Version || cur.Recipe != tc.Recipe
}
// Entries are the nodes' entries of a language, in no particular order.
func (s *ToolchainSet) Entries(language string) []delivery.NodeToolchain {
s.mu.Lock()
defer s.mu.Unlock()
var out []delivery.NodeToolchain
for node, tc := range s.by[language] {
out = append(out, delivery.NodeToolchain{Node: node, Toolchain: tc})
}
return out
}
+51
View File
@@ -0,0 +1,51 @@
package buildrun
import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func tcValue(t *testing.T, language, version string) []byte {
t.Helper()
b, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: "1", At: time.Now()})
if err != nil {
t.Fatal(err)
}
return b
}
func TestToolchainSetKeepsEachNodesEntryByLanguageAndSaysWhenOneChanged(t *testing.T) { // 1607/AC6
var s ToolchainSet
if !s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("a first entry did not count as a change")
}
if s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("the same entry again counted as a change")
}
s.Put(GoBuildModule, "ace.go", tcValue(t, "go", "go1.27.1"), false)
s.Put(NodeBuildModule, "ace.typescript", tcValue(t, "typescript", "tsc5"), false)
if got := s.Entries("go"); len(got) != 2 {
t.Errorf("go entries = %+v, want two", got)
}
if got := s.Entries("typescript"); len(got) != 1 || got[0].Node != "ace" || got[0].Version != "tsc5" {
t.Errorf("typescript entries = %+v", got)
}
if !s.Put(GoBuildModule, "ace.go", nil, true) || len(s.Entries("go")) != 1 {
t.Error("a delete did not remove the node's entry")
}
}
func TestToolchainSetLeavesOutWhatIsNotAToolchainOfTheModulesLanguage(t *testing.T) { // 1607/AC6
var s ToolchainSet
if s.Put(GoBuildModule, "novox.typescript", tcValue(t, "typescript", "x"), false) {
t.Error("a typescript toolchain in the go build module's bucket counted")
}
if s.Put(GoBuildModule, "novox.go", []byte("not json"), false) || s.Put("other-module", "novox.go", tcValue(t, "go", "x"), false) {
t.Error("garbage or an unknown module counted")
}
if len(s.Entries("go")) != 0 {
t.Error("an entry was kept")
}
}
+65 -4
View File
@@ -9,9 +9,11 @@ import (
)
// The checker's keys (novox/hq ADR 0447 clauses 1 to 3, ADR 0444, ADR 0446 clause 5, ADR 0453 clause 4): the verdicts of
// its checks and the passing verdicts it keeps per package and level. The checker writes them from R4c, dark; the
// forge bridge writes the status `mesh/check` from a verdict (unrequired until R4f), the gate reads it from R4f, and
// the view and the watchman read them (a running check carries a pulse).
// its checks and the passing verdicts it keeps per package and level. The checker writes them from R4c, dark, and
// writes the status `mesh/check` it decides as an owed act in its own bucket (CheckerOwedBucket, the gate's owed shape);
// the forge bridge, the only part that calls the forge, performs that act (unrequired until R4f), once per revision of
// the key (ADR 0484 clause 2). The gate reads the status from R4f, and the view and the watchman read the verdicts (a
// running check carries a pulse).
//
// **The verdict's key is the group and the heads, not the fingerprints** (the lead's limit for R4a). Research 063
// section 01 drew `verdicts.<g>.check.<fpset>`; ADR 0447 clause 1 judges a pull request's head merged onto its
@@ -88,6 +90,25 @@ func ParseCheckerVerdictKey(key string) (group, headsHash string, err error) {
type LevelResult struct {
Run int `json:"run"`
Kept int `json:"kept"`
// Skipped is the tests the run skipped, so a pass never hides them (issue 1251, ADR 0484 clause 7).
Skipped int `json:"skipped,omitempty"`
}
// The states of a Composition (ADR 0484 clause 6): the composition check is the assigner's, so until it delivers a
// verdict says CompositionNotRun, and a line never says the composition passed on that.
const (
CompositionNotRun = "not-run"
CompositionPass = "pass"
CompositionFail = "fail"
)
// Composition is what the check of the machines' composition made of a verdict: its State (not-run, pass or fail), the
// Words of a refusal, the mesh-sdk commit it composed with and the machines composed (ADR 0482 clause 9, ADR 0484 clause 6).
type Composition struct {
State string `json:"state"`
Words string `json:"words,omitempty"`
SDK string `json:"sdk,omitempty"`
Machines []string `json:"machines,omitempty"`
}
// Took is the time of each part of a check in integer milliseconds (ADR 0447 clause 3): waiting in the queue, cloning,
@@ -123,6 +144,33 @@ type Verdict struct {
Took Took `json:"took,omitzero"`
Machines []string `json:"machines,omitempty"`
SDK string `json:"sdk,omitempty"`
// Composition is the composition check's result; absent in a value written before it (ADR 0484 clause 6).
Composition *Composition `json:"composition,omitempty"`
// OverBound names the levels whose running went over their bound; it does not change the State (ADR 0453 clause 9).
OverBound []string `json:"over-bound,omitempty"`
}
func (c *Composition) check() error {
if c == nil {
return nil
}
switch c.State {
case CompositionNotRun:
if c.Words != "" || c.SDK != "" || len(c.Machines) > 0 {
return fmt.Errorf("delivery: the verdict's composition is not run and says nothing of words, the mesh-sdk commit or machines")
}
case CompositionPass:
if c.SDK == "" {
return fmt.Errorf("delivery: the verdict's composition passed and names no mesh-sdk commit")
}
case CompositionFail:
if c.Words == "" {
return fmt.Errorf("delivery: the verdict's composition failed and gives no words")
}
default:
return fmt.Errorf("delivery: the verdict's composition state %q is not %s, %s or %s", c.State, CompositionNotRun, CompositionPass, CompositionFail)
}
return nil
}
func (v Verdict) check() error {
@@ -141,10 +189,23 @@ func (v Verdict) check() error {
if err := checkLevel(level); err != nil {
return err
}
if r.Run < 0 || r.Kept < 0 {
if r.Run < 0 || r.Kept < 0 || r.Skipped < 0 {
return fmt.Errorf("delivery: the verdict's %s level counts a negative number of tests", level)
}
}
if err := v.Composition.check(); err != nil {
return err
}
seen := map[string]bool{}
for _, level := range v.OverBound {
if err := checkLevel(level); err != nil {
return fmt.Errorf("delivery: the verdict's over-bound names: %w", err)
}
if seen[level] {
return fmt.Errorf("delivery: the verdict's over-bound names the level %s twice", level)
}
seen[level] = true
}
for part, ms := range map[string]int64{"queue": v.Took.Queue, "clone": v.Took.Clone, "unit": v.Took.Unit,
"integration-setup": v.Took.IntegrationSetup, "integration-run": v.Took.IntegrationRun, "composition": v.Took.Composition} {
if ms < 0 {
+370
View File
@@ -1,6 +1,7 @@
package delivery
import (
"encoding/json"
"errors"
"strings"
"testing"
@@ -220,3 +221,372 @@ func TestCheckerVersions(t *testing.T) {
}
}
}
// oldVerdictJSON is a verdict as the SDK before issue 1672 wrote it (copied from EncodeVerdict of passVerdict's shape
// at the commit before this change): no skipped count, no composition, no over-bound names.
const oldVerdictJSON = `{"v":1,"group":"feat/x","heads":"4b5491557853","trunk":{"novox/mesh-control":"bbbb","novox/mesh-sdk":"aaaa"},"state":"pass","levels":{"integration":{"run":3,"kept":40},"unit":{"run":12,"kept":301}},"took":{"queue":1200,"clone":800,"unit":950,"integration-setup":2100,"integration-run":7400,"composition":3300},"machines":["ace","g14","novox","shanks"],"sdk":"v0.1.14-0.20261019091011-aa68b67389c9"}`
// oldVerdict is the reader of today's shape: the struct as it stood before the new fields.
type oldVerdict struct {
V int `json:"v"`
Group string `json:"group"`
Heads string `json:"heads"`
State string `json:"state"`
Levels map[string]struct{ Run, Kept int }
Machines []string
SDK string
}
// 1672/AC1: the skipped count, the composition and the over-bound names round trip and appear in the bytes.
func TestVerdictCarriesSkippedCompositionAndOverBound(t *testing.T) {
v := passVerdict()
v.Levels[LevelUnit] = LevelResult{Run: 12, Kept: 301, Skipped: 4}
v.Composition = &Composition{State: CompositionPass, SDK: v.SDK, Machines: []string{"ace", "novox"}}
v.OverBound = []string{LevelIntegration}
data, err := EncodeVerdict(v)
if err != nil {
t.Fatal(err)
}
for _, field := range []string{`"skipped":4`, `"composition":{"state":"pass"`, `"over-bound":["integration"]`} {
if !strings.Contains(string(data), field) {
t.Errorf("the value lacks %s: %s", field, data)
}
}
got, err := DecodeVerdict(data)
v.V = Version
if err != nil || !SameVerdict(got, v) || got.Levels[LevelUnit].Skipped != 4 || got.Composition == nil || got.Composition.State != CompositionPass {
t.Errorf("round trip = %+v, %v", got, err)
}
notRun := passVerdict()
notRun.Composition = &Composition{State: CompositionNotRun}
if _, err := EncodeVerdict(notRun); err != nil {
t.Errorf("a composition not yet run refused: %v", err)
}
failed := passVerdict()
failed.State, failed.Failed = VerdictFail, FailedFirst{Machine: "ace", Words: "x"}
failed.Composition = &Composition{State: CompositionFail, Words: "path /etc/x is owned by two modules", SDK: failed.SDK, Machines: []string{"ace"}}
if _, err := EncodeVerdict(failed); err != nil {
t.Errorf("a failed composition refused: %v", err)
}
}
// 1672/AC1: an invalid new value is refused, naming the field.
func TestVerdictNewFieldsChecked(t *testing.T) {
cases := map[string]struct {
edit func(*Verdict)
words []string
}{
"a negative skipped count": {func(v *Verdict) { v.Levels[LevelUnit] = LevelResult{Run: 1, Skipped: -1} }, []string{"unit", "negative"}},
"an unknown composition": {func(v *Verdict) { v.Composition = &Composition{State: "maybe"} }, []string{"composition", "maybe"}},
"a composition with no state": {func(v *Verdict) { v.Composition = &Composition{} }, []string{"composition"}},
"a failed composition without words": {func(v *Verdict) { v.Composition = &Composition{State: CompositionFail} }, []string{"composition", "words"}},
"a passed composition without the sdk": {func(v *Verdict) { v.Composition = &Composition{State: CompositionPass, Machines: []string{"ace"}} }, []string{"composition", "sdk"}},
"a not-run composition naming machines": {func(v *Verdict) { v.Composition = &Composition{State: CompositionNotRun, Machines: []string{"ace"}} }, []string{"composition", "not run"}},
"an unknown over-bound level": {func(v *Verdict) { v.OverBound = []string{"e2e"} }, []string{"e2e"}},
"an over-bound level named twice": {func(v *Verdict) { v.OverBound = []string{LevelUnit, LevelUnit} }, []string{"over-bound", "unit"}},
}
for name, c := range cases {
v := passVerdict()
c.edit(&v)
_, err := EncodeVerdict(v)
if err == nil {
t.Errorf("%s accepted", name)
continue
}
for _, w := range c.words {
if !strings.Contains(err.Error(), w) {
t.Errorf("%s: error %q does not name %q", name, err, w)
}
}
}
}
// 1672/AC2: a value written by the SDK before this change decodes with the new reader, with the new fields empty, and
// the new SDK writes the same bytes for it; a value with the new fields decodes under a reader of today's shape.
func TestVerdictOldAndNewReadersAgree(t *testing.T) {
v, err := DecodeVerdict([]byte(oldVerdictJSON))
if err != nil {
t.Fatalf("the new reader of an old value: %v", err)
}
if v.Composition != nil || v.OverBound != nil || v.Levels[LevelUnit].Skipped != 0 || v.Levels[LevelUnit].Run != 12 {
t.Errorf("an old value grew fields: %+v", v)
}
again, err := EncodeVerdict(v)
if err != nil || string(again) != oldVerdictJSON {
t.Errorf("the new SDK writes %s, %v for an old value, want today's bytes", again, err)
}
n := passVerdict()
n.Levels[LevelUnit] = LevelResult{Run: 12, Kept: 301, Skipped: 4}
n.Composition = &Composition{State: CompositionNotRun}
n.OverBound = []string{LevelUnit}
data, err := EncodeVerdict(n)
if err != nil {
t.Fatal(err)
}
var ov oldVerdict
if err := json.Unmarshal(data, &ov); err != nil || ov.Group != "feat/x" || ov.State != VerdictPass || ov.Levels[LevelUnit].Run != 12 || ov.SDK == "" {
t.Errorf("the old reader of a new value: %+v, %v", ov, err)
}
}
func sampleCheckerOwed() Owed {
return Owed{Act: OwedStatus, Context: StatusContextCheck, State: "success", Target: "https://mesh.example/delivery/check/36",
Text: "pass 4b5491557853 on aaaa: unit 12 run 301 kept, integration 3 run 40 kept; composition: not yet run"}
}
// 1673/AC1: the checker's owed bucket is the checker module plus _owed, and its key is the gate's owed key with the act
// status=mesh/check.
func TestCheckerOwedKeyAndBucket(t *testing.T) {
if CheckerOwedBucket != "mesh-delivery-checker_owed" || CheckerOwedHistory != 5 || PreviewsBucket != "mesh-delivery-checker_previews" || PreviewsHistory != 5 {
t.Errorf("buckets %s %d %s %d", CheckerOwedBucket, CheckerOwedHistory, PreviewsBucket, PreviewsHistory)
}
key, err := CheckerOwedKey("novox/mesh-sdk", 36, "aaaa")
if err != nil || key != "novox/mesh-sdk.36.aaaa.status=mesh/check" {
t.Fatalf("key = %q, %v", key, err)
}
repo, number, head, err := ParseCheckerOwedKey(key)
if err != nil || repo != "novox/mesh-sdk" || number != 36 || head != "aaaa" {
t.Errorf("parse = %q %d %q %v", repo, number, head, err)
}
other, _ := OwedKey("novox/mesh-sdk", 36, "aaaa", "status=mesh/merge-gate")
for _, bad := range []string{other, "novox/mesh-sdk.36.aaaa.view", "novox/mesh-sdk.36.aaaa.note", "novox/mesh-sdk.36.aaaa.comment.1", ""} {
if _, _, _, err := ParseCheckerOwedKey(bad); err == nil {
t.Errorf("checker owed key %q accepted", bad)
}
}
if _, err := CheckerOwedKey("nope", 36, "aaaa"); err == nil {
t.Error("a bad repository accepted")
}
}
// 1673/AC2: an owed value with the context mesh/check is accepted and an unknown context is refused, in the act and in the value.
func TestOwedAcceptsMeshCheckAndRefusesAnUnknownContext(t *testing.T) {
for _, ctx := range []string{StatusContextCheck, StatusContextMergeGate} {
if kind, got, err := ParseOwedAct("status=" + ctx); err != nil || kind != OwedStatus || got != ctx {
t.Errorf("ParseOwedAct(status=%s) = %q %q %v", ctx, kind, got, err)
}
}
for _, bad := range []string{"status=mesh/other", "status=check", "status=mesh/Check"} {
if _, err := OwedKey("novox/mesh-sdk", 36, "aaaa", bad); err == nil || !strings.Contains(err.Error(), bad[len("status="):]) {
t.Errorf("owed act %q: %v, want a refusal naming the context", bad, err)
}
}
data, err := EncodeCheckerOwed(sampleCheckerOwed())
if err != nil {
t.Fatal(err)
}
if got, err := DecodeCheckerOwed(data); err != nil || !SameOwed(got, sampleCheckerOwed()) && got.Context != StatusContextCheck {
t.Errorf("round trip = %+v, %v", got, err)
}
unknown := sampleCheckerOwed()
unknown.Context = "mesh/other"
if _, err := EncodeOwed(unknown); err == nil || !strings.Contains(err.Error(), "mesh/other") {
t.Errorf("an owed status of the context mesh/other: %v", err)
}
// The checker's bucket holds only its own status.
gate := sampleCheckerOwed()
gate.Context = StatusContextMergeGate
for name, o := range map[string]Owed{"the gate's context": gate, "a view": {Act: OwedView, Text: "x"}, "a note": {Act: OwedNote, Text: "x", Ref: "mesh-plan"}} {
if _, err := EncodeCheckerOwed(o); err == nil {
t.Errorf("the checker's bucket takes %s", name)
}
}
for _, state := range []string{"pending", "success", "failure"} {
o := sampleCheckerOwed()
o.State = state
if _, err := EncodeCheckerOwed(o); err != nil {
t.Errorf("state %s: %v", state, err)
}
}
}
func samplePreview() Preview {
return Preview{Repository: "novox/mesh-delivery", Head: "aaaa", Trunk: "bbbb", Merge: "cccc",
Modules: map[string]PreviewModule{
"modules/mesh-delivery-checker": {Tree: "t1", Manifest: strings.Repeat("1", 64), Locks: map[string]string{"go.sum": strings.Repeat("2", 64)}},
"modules/mesh-delivery-view": {Tree: "t2", Manifest: strings.Repeat("3", 64)},
}}
}
// 1673/AC3: the previews key is <repository>.<head>.<trunk> and parses back.
func TestPreviewKey(t *testing.T) {
key, err := PreviewKey("novox/mesh-delivery", "aaaa", "bbbb")
if err != nil || key != "novox/mesh-delivery.aaaa.bbbb" {
t.Fatalf("key = %q, %v", key, err)
}
repo, head, trunk, err := ParsePreviewKey(key)
if err != nil || repo != "novox/mesh-delivery" || head != "aaaa" || trunk != "bbbb" {
t.Errorf("parse = %q %q %q %v", repo, head, trunk, err)
}
for _, bad := range [][3]string{{"nope", "aaaa", "bbbb"}, {"novox/x", "", "bbbb"}, {"novox/x", "aaaa", ""}, {"novox/x", "a.b", "bbbb"}, {"novox/x", "aaaa", "b b"}} {
if _, err := PreviewKey(bad[0], bad[1], bad[2]); err == nil {
t.Errorf("PreviewKey%v accepted", bad)
}
}
for _, bad := range []string{"", "novox/x.aaaa", "novox/x.aaaa.bbbb.cccc", "nope.aaaa.bbbb", "novox/x..bbbb", "novox/x.aaaa."} {
if _, _, _, err := ParsePreviewKey(bad); err == nil || !strings.Contains(err.Error(), "\""+bad+"\"") {
t.Errorf("ParsePreviewKey(%q) = %v, want a refusal naming it", bad, err)
}
}
}
// 1673/AC4: a preview round trips, and an invalid one is refused naming the field.
func TestPreviewRoundTripAndRefusals(t *testing.T) {
data, err := EncodePreview(samplePreview())
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(string(data), `{"v":1,`) {
t.Errorf("at %s", data)
}
got, err := DecodePreview(data)
want := samplePreview()
want.V = Version
if err != nil || !SamePreview(got, want) {
t.Errorf("round trip = %+v, %v", got, err)
}
if _, err := EncodePreview(Preview{Repository: "novox/x", Head: "aaaa", Trunk: "bbbb", Merge: "cccc"}); err != nil {
t.Errorf("a preview of a head that changes no module refused: %v", err)
}
cases := map[string]struct {
edit func(*Preview)
words []string
}{
"no repository": {func(p *Preview) { p.Repository = "" }, []string{"repository"}},
"a bad repository": {func(p *Preview) { p.Repository = "nope" }, []string{"nope"}},
"no head": {func(p *Preview) { p.Head = "" }, []string{"head"}},
"no trunk": {func(p *Preview) { p.Trunk = "" }, []string{"trunk"}},
"no merge": {func(p *Preview) { p.Merge = "" }, []string{"merge"}},
"a module with no tree": {func(p *Preview) {
p.Modules["modules/mesh-delivery-view"] = PreviewModule{Manifest: strings.Repeat("3", 64)}
}, []string{"modules/mesh-delivery-view", "tree"}},
"a bad manifest digest": {func(p *Preview) {
p.Modules["modules/mesh-delivery-view"] = PreviewModule{Tree: "t2", Manifest: "xyz"}
}, []string{"modules/mesh-delivery-view", "xyz"}},
"an empty directory": {func(p *Preview) { p.Modules[""] = PreviewModule{Tree: "t", Manifest: strings.Repeat("3", 64)} }, []string{"directory"}},
"a lock with no hash": {func(p *Preview) {
p.Modules["modules/mesh-delivery-view"] = PreviewModule{Tree: "t2", Manifest: strings.Repeat("3", 64), Locks: map[string]string{"go.sum": ""}}
}, []string{"modules/mesh-delivery-view", "go.sum"}},
}
for name, c := range cases {
p := samplePreview()
c.edit(&p)
_, err := EncodePreview(p)
if err == nil {
t.Errorf("%s accepted", name)
continue
}
for _, w := range c.words {
if !strings.Contains(err.Error(), w) {
t.Errorf("%s: error %q does not name %q", name, err, w)
}
}
}
if _, err := DecodePreview([]byte(`{"v":2,"repository":"novox/x"}`)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("a preview of version 2: %v", err)
}
if _, err := DecodePreview([]byte(`{"v":1,"repository":"novox/x","head":"a","trunk":"b"}`)); err == nil {
t.Error("decoded a preview with no merge")
}
}
// 1673/AC5: a reader of today's checker shapes ignores a field it does not know, and the checker's owed value is the
// gate's shape, so the gate's reader reads it (as indexer2_test.go does for the indexer's values).
func TestCheckerOwedIsReadableByTheGatesReaderAndAnOldOwedReaderIgnoresNewFields(t *testing.T) {
data, err := EncodeCheckerOwed(sampleCheckerOwed())
if err != nil {
t.Fatal(err)
}
var old struct {
V int `json:"v"`
Act string `json:"act"`
Context string `json:"context"`
State string `json:"state"`
}
if err := json.Unmarshal(data, &old); err != nil || old.Act != OwedStatus || old.Context != "mesh/check" {
t.Errorf("the old reader: %+v %v", old, err)
}
withNew := strings.Replace(string(data), `{"v":1,`, `{"v":1,"a-field-of-tomorrow":7,`, 1)
if _, err := DecodeOwed([]byte(withNew)); err != nil {
t.Errorf("an owed value with a field of tomorrow: %v", err)
}
p, _ := EncodePreview(samplePreview())
withNew = strings.Replace(string(p), `{"v":1,`, `{"v":1,"a-field-of-tomorrow":7,`, 1)
if _, err := DecodePreview([]byte(withNew)); err != nil {
t.Errorf("a preview with a field of tomorrow: %v", err)
}
}
// 1673/AC2 (review): the gate's recheck owes a pending status=mesh/repo-check (mesh-catalog mesh-delivery-gate owed.go,
// checks.go ctxRepo); its key and value round trip, and the checker's bucket still refuses it.
func TestTheGatesRecheckStatusRoundTripsAndTheCheckersBucketRefusesIt(t *testing.T) {
key, err := OwedKey("novox/mesh-sdk", 36, "aaaa", "status=mesh/repo-check")
if err != nil || key != "novox/mesh-sdk.36.aaaa.status=mesh/repo-check" {
t.Fatalf("key = %q, %v", key, err)
}
if _, _, _, act, err := ParseOwedKey(key); err != nil || act != "status=mesh/repo-check" {
t.Errorf("parse = %q, %v", act, err)
}
o := Owed{Act: OwedStatus, Context: StatusContextRepoCheck, State: "pending", Target: "https://mesh.example/delivery/36", Text: "checking again"}
data, err := EncodeOwed(o)
if err != nil {
t.Fatal(err)
}
if got, err := DecodeOwed(data); err != nil || !SameOwed(got, o) && got.Context != o.Context {
t.Errorf("round trip = %+v, %v", got, err)
}
if _, err := EncodeCheckerOwed(o); err == nil {
t.Error("the checker's bucket takes the gate's recheck status")
}
if _, _, _, err := ParseCheckerOwedKey(key); err == nil {
t.Error("a checker owed key of the recheck status accepted")
}
}
// 1445/AC5 (layout): a head that does not merge is a preview naming its conflicting files, with no merge commit and no
// modules; a preview with conflicts and a merge or modules is refused; a preview of today's shape (no conflicts)
// decodes with none, and a reader of today's shape reads one with conflicts.
func TestPreviewOfAHeadInConflict(t *testing.T) {
c := Preview{Repository: "novox/mesh-delivery", Head: "aaaa", Trunk: "bbbb", Conflicts: []string{"a.txt", "modules/m/module.json"}}
data, err := EncodePreview(c)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"conflicts":["a.txt","modules/m/module.json"]`) || strings.Contains(string(data), `"merge"`) {
t.Errorf("at %s", data)
}
got, err := DecodePreview(data)
c.V = Version
if err != nil || !SamePreview(got, c) {
t.Errorf("round trip = %+v, %v", got, err)
}
withMerge := c
withMerge.Merge = "cccc"
withModules := c
withModules.Modules = samplePreview().Modules
emptyPath := c
emptyPath.Conflicts = []string{""}
for name, p := range map[string]Preview{"a merge commit": withMerge, "modules": withModules, "an empty conflicting path": emptyPath} {
if _, err := EncodePreview(p); err == nil || !strings.Contains(err.Error(), "onflict") {
t.Errorf("a preview in conflict with %s: %v", name, err)
}
}
// today's bytes: a clean preview has no conflicts and decodes with none.
clean, _ := EncodePreview(samplePreview())
if strings.Contains(string(clean), "conflicts") {
t.Errorf("a clean preview carries conflicts: %s", clean)
}
if p, err := DecodePreview(clean); err != nil || p.Conflicts != nil {
t.Errorf("a clean preview decodes with conflicts %v, %v", p.Conflicts, err)
}
var old struct {
V int `json:"v"`
Repository string `json:"repository"`
Head string `json:"head"`
Trunk string `json:"trunk"`
Merge string `json:"merge"`
}
if err := json.Unmarshal(data, &old); err != nil || old.Head != "aaaa" || old.Merge != "" {
t.Errorf("the old reader: %+v %v", old, err)
}
}
+26 -1
View File
@@ -28,6 +28,18 @@ const (
AnswersHistory = 5
)
// The contexts of the statuses the mesh owes the forge: the gate's, the delivery's view of a head and of a group
// (design 47), and the checker's, which is not required until R4f (ADR 0484 clause 2, ADR 0453 clause 4).
const (
StatusContextMergeGate = "mesh/merge-gate"
StatusContextDelivery = "mesh/delivery"
StatusContextDeliveryGroup = "mesh/delivery-group"
StatusContextRepoCheck = "mesh/repo-check" // the gate's pending status of a recheck (mesh-delivery.recheck)
StatusContextCheck = "mesh/check"
)
var statusContexts = []string{StatusContextMergeGate, StatusContextDelivery, StatusContextDeliveryGroup, StatusContextRepoCheck, StatusContextCheck}
// AskKindMerge is the kind of the gate's ask for a group's merge, the first part of its key.
const AskKindMerge = "merge"
@@ -131,6 +143,16 @@ func AskingID(askKey string, asking int) string {
return AskID(askKey + "." + strconv.Itoa(asking))
}
// checkStatusContext refuses a status context the mesh does not owe the forge.
func checkStatusContext(context string) error {
for _, c := range statusContexts {
if context == c {
return nil
}
}
return fmt.Errorf("delivery: the status context %q is not one of %s", context, strings.Join(statusContexts, ", "))
}
func checkOwedAct(act string) (kind, context string, err error) {
kind, context, hasContext := strings.Cut(act, "=")
switch kind {
@@ -138,7 +160,7 @@ func checkOwedAct(act string) (kind, context string, err error) {
if !hasContext {
return "", "", fmt.Errorf("delivery: the owed act %q is a status without its context, status=<context>", act)
}
if err := checkName("status context", context); err != nil {
if err := checkStatusContext(context); err != nil {
return "", "", err
}
return kind, context, nil
@@ -345,6 +367,9 @@ func (o Owed) check() error {
if o.Context == "" {
return fmt.Errorf("delivery: an owed status has no context")
}
if err := checkStatusContext(o.Context); err != nil {
return err
}
if o.Target == "" {
return fmt.Errorf("delivery: an owed status has no target")
}
+181
View File
@@ -0,0 +1,181 @@
package delivery
import (
"encoding/json"
"fmt"
"reflect"
"strings"
)
// The checker's other buckets (ADR 0484 clauses 2 and 3): owed, the acts it owes the forge, which the forge bridge
// performs once per revision of a key, and previews, what the fetch step answered for a head merged onto a trunk commit.
const (
// CheckerOwedBucket is the checker's bucket of owed acts, in the gate's owed shape: a status with the context
// StatusContextCheck for each head it judged. One part writes each key: the checker.
CheckerOwedBucket = CheckerModule + "_owed"
// CheckerOwedHistory is the revisions the checker's owed bucket keeps per key.
CheckerOwedHistory = 5
// PreviewsBucket is the checker's bucket of merge previews; the indexer reads it for the candidates' fingerprints.
PreviewsBucket = CheckerModule + "_previews"
// PreviewsHistory is the revisions the previews bucket keeps per key.
PreviewsHistory = 5
)
// CheckerOwedKey is the key, in CheckerOwedBucket, of the status the checker owes on a pull request at a head: the
// gate's OwedKey with the act `status=mesh/check`, the only act the checker owes.
func CheckerOwedKey(repository string, number int, head string) (string, error) {
return OwedKey(repository, number, head, OwedStatus+"="+StatusContextCheck)
}
// ParseCheckerOwedKey is the inverse of CheckerOwedKey; it refuses an owed key of any other act.
func ParseCheckerOwedKey(key string) (repository string, number int, head string, err error) {
repository, number, head, act, err := ParseOwedKey(key)
if err != nil {
return "", 0, "", err
}
if act != OwedStatus+"="+StatusContextCheck {
return "", 0, "", fmt.Errorf("delivery: %q is not a checker owed key: the checker owes only status=%s, not %q", key, StatusContextCheck, act)
}
return repository, number, head, nil
}
// EncodeCheckerOwed is o as JSON at Version; it refuses what EncodeOwed refuses and any act but a status with the
// context StatusContextCheck.
func EncodeCheckerOwed(o Owed) ([]byte, error) {
if err := checkCheckerOwed(o); err != nil {
return nil, err
}
return EncodeOwed(o)
}
func checkCheckerOwed(o Owed) error {
if o.Act != OwedStatus || o.Context != StatusContextCheck {
return fmt.Errorf("delivery: the checker owes only a status with the context %s, not the act %q with the context %q", StatusContextCheck, o.Act, o.Context)
}
return nil
}
// DecodeCheckerOwed reads an owed act of the checker's bucket.
func DecodeCheckerOwed(data []byte) (Owed, error) {
o, err := DecodeOwed(data)
if err != nil {
return Owed{}, err
}
if err := checkCheckerOwed(o); err != nil {
return Owed{}, err
}
return o, nil
}
// PreviewKey is the key, in PreviewsBucket, of a head merged onto a trunk commit: `<repository>.<head>.<trunk>`.
func PreviewKey(repository, head, trunk string) (string, error) {
repo, err := escapeRepository(repository)
if err != nil {
return "", err
}
if err := checkName("head", head); err != nil {
return "", err
}
if err := checkName("trunk commit", trunk); err != nil {
return "", err
}
return repo + "." + head + "." + trunk, nil
}
// ParsePreviewKey is the inverse of PreviewKey.
func ParsePreviewKey(key string) (repository, head, trunk string, err error) {
parts := strings.Split(key, ".")
if len(parts) != 3 {
return "", "", "", fmt.Errorf("delivery: %q is not a preview key, <repository>.<head>.<trunk>", key)
}
repository, err = unescapeRepository(parts[0])
if err != nil {
return "", "", "", fmt.Errorf("delivery: %q is not a preview key: %w", key, err)
}
if _, err := PreviewKey(repository, parts[1], parts[2]); err != nil {
return "", "", "", fmt.Errorf("delivery: %q is not a preview key: %w", key, err)
}
return repository, parts[1], parts[2], nil
}
// PreviewModule is what the preview holds of one module directory: the tree sha of the directory, the digest of its
// manifest (the key in the manifests bucket) and the hex sha256 of its lock files by path relative to it.
type PreviewModule struct {
Tree string `json:"tree"`
Manifest string `json:"manifest"`
Locks map[string]string `json:"locks,omitempty"`
}
// Preview is the value of a preview key: the repository, the head and the trunk commit it was merged onto, the merge
// commit and, for each module directory of the merge, a PreviewModule. A head that does not merge is a preview too: it
// names the files in Conflicts and has no merge commit and no modules (issue 1445).
type Preview struct {
V int `json:"v"`
Repository string `json:"repository"`
Head string `json:"head"`
Trunk string `json:"trunk"`
Merge string `json:"merge,omitempty"`
Conflicts []string `json:"conflicts,omitempty"`
Modules map[string]PreviewModule `json:"modules,omitempty"`
}
func (p Preview) check() error {
if _, err := PreviewKey(p.Repository, p.Head, p.Trunk); err != nil {
return err
}
for _, f := range p.Conflicts {
if f == "" {
return fmt.Errorf("delivery: a preview names a conflicting file with an empty path")
}
}
if len(p.Conflicts) > 0 {
if p.Merge != "" || len(p.Modules) > 0 {
return fmt.Errorf("delivery: a preview of a head in conflict has no merge commit and no modules")
}
return nil
}
if p.Merge == "" {
return fmt.Errorf("delivery: a preview names no merge commit")
}
for dir, m := range p.Modules {
if dir == "" {
return fmt.Errorf("delivery: a preview names a module with an empty directory")
}
if m.Tree == "" {
return fmt.Errorf("delivery: the preview's module %q has no tree", dir)
}
if _, err := ManifestsKey(m.Manifest); err != nil {
return fmt.Errorf("delivery: the preview's module %q: %w", dir, err)
}
for path, hash := range m.Locks {
if path == "" || hash == "" {
return fmt.Errorf("delivery: the lock %q of the preview's module %q has an empty path or hash", path, dir)
}
}
}
return nil
}
// EncodePreview is p as JSON at Version.
func EncodePreview(p Preview) ([]byte, error) {
if err := p.check(); err != nil {
return nil, err
}
p.V = Version
return json.Marshal(p)
}
// DecodePreview reads a preview.
func DecodePreview(data []byte) (Preview, error) {
var p Preview
if err := decode(data, &p); err != nil {
return Preview{}, err
}
if err := p.check(); err != nil {
return Preview{}, err
}
return p, nil
}
// SamePreview says whether two previews are the same value.
func SamePreview(a, b Preview) bool { return reflect.DeepEqual(a, b) }
+17
View File
@@ -389,8 +389,20 @@ type Module struct {
Src uint64 `json:"src,omitempty"`
Repository string `json:"repository"`
Commit string `json:"commit"`
// Path is the module's directory in its repository, trimmed of slashes, and RootPath (".") when its manifest is the
// repository's root (novox/hq ADR 0478 clause 9, issue 1615): the build tools need it. Empty means not known: a value
// written before the indexer learned paths has none.
Path string `json:"path,omitempty"`
// Toolchains are the toolchains FP2 was computed with, by language (go, typescript): the version installed and the
// build module's recipe version the build nodes agreed on then (novox/hq ADR 0477 clause 2, ADR 0478 clause 3). A builder
// builds the fingerprint only with these. Empty when FP2 is empty or the module has no toolchain; a value written
// before has none (issue 1615).
Toolchains map[string]ToolchainUsed `json:"toolchains,omitempty"`
}
// RootPath is Module.Path of a module whose manifest is its repository's root.
const RootPath = "."
// KeptInputs2 are SourceInputs2 without the module's name (the key has it) and without toolchains (those come from
// the build modules' toolchain keys when the fingerprint is computed): what a Module keeps of a merge.
type KeptInputs2 struct {
@@ -418,6 +430,11 @@ func (m Module) check() error {
case m.FP2 != "" && m.None2 != "":
return fmt.Errorf("delivery: a module has both fp2 and none2")
}
for language, t := range m.Toolchains {
if t.Version == "" || t.Recipe == "" {
return fmt.Errorf("delivery: the module's toolchain of %q needs its version and its recipe version", language)
}
}
if m.Inputs2 != nil && m.Manifest == "" {
return fmt.Errorf("delivery: a module keeps new-form inputs but has no manifest digest")
}
+51
View File
@@ -374,3 +374,54 @@ func TestForgeDirs(t *testing.T) {
t.Error("dirs without dirs-known accepted")
}
}
func TestModulePath(t *testing.T) { // 1588/AC1
data, err := EncodeModule(Module{FP: "src1:x", Repository: "a/b", Commit: "c", Path: "modules/build-agent"})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"path":"modules/build-agent"`) {
t.Errorf("encoded %s has no path", data)
}
m, err := DecodeModule(data)
if err != nil || m.Path != "modules/build-agent" {
t.Errorf("round trip = %+v, %v", m, err)
}
// A value written before has no path: it decodes, checks, and encodes without the key.
old, err := DecodeModule([]byte(`{"v":1,"fp":"src1:x","repository":"a/b","commit":"c"}`))
if err != nil || old.Path != "" {
t.Fatalf("old shape = %+v, %v", old, err)
}
again, _ := EncodeModule(old)
if strings.Contains(string(again), "path") {
t.Errorf("a module without a path encodes %s", again)
}
}
func TestModuleRootPathAndToolchains(t *testing.T) { // 1615/AC1, AC2
tc := map[string]ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "2"}}
data, err := EncodeModule(Module{FP: "src1:x", Repository: "a/b", Commit: "c", Path: RootPath, Toolchains: tc})
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(data), `"path":"."`) || !strings.Contains(string(data), `"toolchains":{"go":{"version":"go1.27.1","recipe":"1"}`) {
t.Errorf("encoded %s", data)
}
m, err := DecodeModule(data)
if err != nil || m.Path != RootPath || !reflect.DeepEqual(m.Toolchains, tc) {
t.Errorf("round trip = %+v, %v", m, err)
}
// A value written before has none: it decodes and encodes without the key.
old, err := DecodeModule([]byte(`{"v":1,"fp":"src1:x","repository":"a/b","commit":"c","path":"modules/a"}`))
if err != nil || old.Toolchains != nil {
t.Fatalf("old shape = %+v, %v", old, err)
}
if again, _ := EncodeModule(old); strings.Contains(string(again), "toolchains") {
t.Errorf("a module without toolchains encodes %s", again)
}
for _, bad := range []map[string]ToolchainUsed{{"go": {Recipe: "1"}}, {"go": {Version: "v"}}} {
if _, err := EncodeModule(Module{FP: "src1:x", Repository: "a/b", Commit: "c", Toolchains: bad}); err == nil {
t.Errorf("toolchains %v accepted", bad)
}
}
}
+1 -1
View File
@@ -1 +1 @@
{"id":"build-1792395447088936520","repository":"http://novox.internal:20000/novox/mesh-controller.git","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","manifest-digest":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","source-inputs":{"Module":"build-agent","Tree":"11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca","Bases":["sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc","sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c"],"Contexts":{"server":"narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"},"Toolchains":null,"Unpinned":""},"made":[{"name":"server","kind":"image","reference":"registry.example/build-agent@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"}],"at":"2026-10-19T09:38:00Z"}
{"id":"build-1792395447088936520","repository":"http://novox.internal:20000/novox/mesh-controller.git","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","manifest-digest":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","source-inputs":{"Module":"build-agent","Tree":"11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca","Bases":["sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc","sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7236c349a0c"],"Contexts":{"server":"narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"},"Toolchains":null,"Unpinned":""},"made":[{"name":"server","kind":"image","reference":"artifact-store://build-agent/server@sha256:a79c13f1c727ff94959e80824cc704ea58b80d80ecdd00fd447698acebb35090"},{"name":"files","kind":"archive","reference":"artifact-store://build-agent/files/blobs/sha256:bccd3f47851436d1b45a3063cc2d7e34c5c109b2d030739bfbb4b9781703f9a9"}],"at":"2026-10-19T09:38:00Z"}
+2
View File
@@ -1,3 +1,5 @@
//go:build integration
package forgekey
import (
+277
View File
@@ -0,0 +1,277 @@
// Package objectlink signs and streams a link to one object in the mesh's object store (novox/hq ADR 0468 §3).
//
// Bytes move over HTTP between a party and the store, on a signed link of at most ten minutes for one object and one
// method, signed for the name the `s3-bucket` binding gives; a document over 100 MiB is refused. Presign makes the
// link (Signature Version 4, query form, path style, payload unsigned, `host` the only signed header); Put and Get
// move the bytes without holding them in memory and count and hash them as they go.
//
// Standard library only: the SDK requires nothing of the mesh's forge.
package objectlink
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/xml"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"sort"
"strings"
"time"
)
const (
// MaxDocument is the largest document a link moves (ADR 0468 §3): the default limit of Put and Get.
MaxDocument int64 = 100 << 20
// MaxExpires is the longest a link lives (ADR 0468 §3).
MaxExpires = 10 * time.Minute
defaultRegion = "us-east-1"
algorithm = "AWS4-HMAC-SHA256"
)
// Request is one link to sign.
type Request struct {
// Method is GET or PUT.
Method string
// Endpoint is the store as the party reaches it, "http://host:port". Its host is signed exactly as written.
Endpoint string
Bucket string
Key string
// AccessKey and SecretKey are the credential the s3-bucket binding gives.
AccessKey string
SecretKey string
// Region defaults to us-east-1.
Region string
// Expires is how long the link lives: at least one second, at most ten minutes.
Expires time.Duration
// Now is the signing time; the clock's when zero.
Now time.Time
}
// Presign returns the signed link for one object and one method.
func Presign(r Request) (string, error) {
if r.Method != http.MethodGet && r.Method != http.MethodPut {
return "", fmt.Errorf("objectlink: method %q: a link is for GET or PUT (ADR 0468 §3)", r.Method)
}
if r.Expires < time.Second || r.Expires > MaxExpires {
return "", fmt.Errorf("objectlink: a link lives at least 1s and at most %s, not %s (ADR 0468 §3)", MaxExpires, r.Expires)
}
if r.Bucket == "" || r.Key == "" || r.AccessKey == "" || r.SecretKey == "" {
return "", errors.New("objectlink: a bucket, a key and a credential are needed to sign a link")
}
u, err := url.Parse(r.Endpoint)
if err != nil || u.Host == "" || (u.Scheme != "http" && u.Scheme != "https") {
return "", fmt.Errorf("objectlink: endpoint %q is not http(s)://host[:port]", r.Endpoint)
}
region := r.Region
if region == "" {
region = defaultRegion
}
now := r.Now
if now.IsZero() {
now = time.Now()
}
path := "/" + encode(r.Bucket, false) + "/" + encode(r.Key, false)
query := sign(r.Method, u.Host, path, r.AccessKey, r.SecretKey, region, now.UTC(), int64(r.Expires/time.Second))
return u.Scheme + "://" + u.Host + path + "?" + query, nil
}
// sign returns the query string of a presigned link, signature last. The path is already encoded.
func sign(method, host, path, accessKey, secretKey, region string, now time.Time, expires int64) string {
date := now.Format("20060102")
stamp := now.Format("20060102T150405Z")
scope := date + "/" + region + "/s3/aws4_request"
params := map[string]string{
"X-Amz-Algorithm": algorithm,
"X-Amz-Credential": accessKey + "/" + scope,
"X-Amz-Date": stamp,
"X-Amz-Expires": fmt.Sprint(expires),
"X-Amz-SignedHeaders": "host",
}
names := make([]string, 0, len(params))
for k := range params {
names = append(names, k)
}
sort.Strings(names)
var query []string
for _, k := range names {
query = append(query, encode(k, true)+"="+encode(params[k], true))
}
canonicalQuery := strings.Join(query, "&")
canonical := strings.Join([]string{method, path, canonicalQuery, "host:" + host + "\n", "host", "UNSIGNED-PAYLOAD"}, "\n")
digest := sha256.Sum256([]byte(canonical))
toSign := strings.Join([]string{algorithm, stamp, scope, hex.EncodeToString(digest[:])}, "\n")
key := mac([]byte("AWS4"+secretKey), date)
key = mac(key, region)
key = mac(key, "s3")
key = mac(key, "aws4_request")
return canonicalQuery + "&X-Amz-Signature=" + hex.EncodeToString(mac(key, toSign))
}
func mac(key []byte, data string) []byte {
h := hmac.New(sha256.New, key)
h.Write([]byte(data))
return h.Sum(nil)
}
// encode is AWS's URI encoding: everything but A-Z a-z 0-9 - _ . ~ as %XX, and the slash kept in a path.
func encode(s string, slash bool) string {
var b strings.Builder
for i := 0; i < len(s); i++ {
c := s[i]
switch {
case c >= 'A' && c <= 'Z', c >= 'a' && c <= 'z', c >= '0' && c <= '9', c == '-', c == '_', c == '.', c == '~':
b.WriteByte(c)
case c == '/' && !slash:
b.WriteByte(c)
default:
fmt.Fprintf(&b, "%%%02X", c)
}
}
return b.String()
}
// Result is what moved: its size and the SHA-256 of its bytes, in hex.
type Result struct {
Size int64
SHA256 string
}
// StoreError is a refusal by the store: its status and the code and message of its XML error body.
type StoreError struct {
Status int
Code string
Message string
}
func (e *StoreError) Error() string {
if e.Code == "" {
return fmt.Sprintf("objectlink: the store answered %d", e.Status)
}
return fmt.Sprintf("objectlink: the store answered %d %s: %s", e.Status, e.Code, e.Message)
}
func storeError(resp *http.Response) error {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
var parsed struct {
Code string `xml:"Code"`
Message string `xml:"Message"`
}
_ = xml.Unmarshal(body, &parsed)
return &StoreError{Status: resp.StatusCode, Code: parsed.Code, Message: parsed.Message}
}
// tooBig is the error of a document past its limit.
func tooBig(limit int64) error {
return fmt.Errorf("objectlink: the document is over %d bytes (ADR 0468 §3)", limit)
}
// exact counts and hashes what passes, and fails a stream that is longer than declared, or ends before it.
type exact struct {
r io.Reader
size int64
n int64
hash interface{ Write([]byte) (int, error) }
}
func (l *exact) Read(p []byte) (int, error) {
n, err := l.r.Read(p)
l.n += int64(n)
if l.n > l.size {
return 0, fmt.Errorf("objectlink: the body is longer than the declared %d bytes", l.size)
}
l.hash.Write(p[:n])
if err == io.EOF && l.n < l.size {
return n, fmt.Errorf("objectlink: the body ended at %d bytes, short of the declared %d", l.n, l.size)
}
return n, err
}
// Put streams body to a link made for PUT, never holding it in memory. The caller gives the body's exact size, which
// is always sent as the Content-Length (a store answers 411 to a PUT without one). A negative size is refused, and so
// is one over limit (MaxDocument when limit is not above zero, ADR 0468 §3), both before any request. A body that
// yields more or fewer bytes than size fails the request, so a PUT whose body does not end as declared stores nothing.
func Put(ctx context.Context, client *http.Client, link string, body io.Reader, size, limit int64) (Result, error) {
if limit <= 0 {
limit = MaxDocument
}
if client == nil {
client = http.DefaultClient
}
if size < 0 {
return Result{}, errors.New("objectlink: put needs the size of the body; a negative or unknown size is refused")
}
if size > limit {
return Result{}, tooBig(limit)
}
hash := sha256.New()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
counted := &exact{r: body, size: size, hash: hash}
req, err := http.NewRequestWithContext(ctx, http.MethodPut, link, io.NopCloser(counted))
if err != nil {
return Result{}, err
}
req.ContentLength = size
if size == 0 {
req.Body = http.NoBody
}
resp, err := client.Do(req)
if err != nil {
cancel()
return Result{}, fmt.Errorf("objectlink: put: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode > 299 {
return Result{}, storeError(resp)
}
if counted.n != size {
return Result{}, fmt.Errorf("objectlink: put: sent %d bytes of the declared %d", counted.n, size)
}
return Result{Size: counted.n, SHA256: hex.EncodeToString(hash.Sum(nil))}, nil
}
// Get streams the object of a link made for GET into w, refusing more than limit bytes (MaxDocument when limit is
// not above zero). When wantSHA256 is given and the bytes hash to another value the call fails: the caller removes
// what it wrote.
func Get(ctx context.Context, client *http.Client, link string, w io.Writer, limit int64, wantSHA256 string) (Result, error) {
if limit <= 0 {
limit = MaxDocument
}
if client == nil {
client = http.DefaultClient
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, link, nil)
if err != nil {
return Result{}, err
}
resp, err := client.Do(req)
if err != nil {
return Result{}, fmt.Errorf("objectlink: get: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode > 299 {
return Result{}, storeError(resp)
}
if resp.ContentLength > limit {
return Result{}, tooBig(limit)
}
hash := sha256.New()
n, err := io.Copy(io.MultiWriter(w, hash), io.LimitReader(resp.Body, limit+1))
if err != nil {
return Result{}, fmt.Errorf("objectlink: get: %w", err)
}
if n > limit {
return Result{}, tooBig(limit)
}
got := hex.EncodeToString(hash.Sum(nil))
if wantSHA256 != "" && !strings.EqualFold(wantSHA256, got) {
return Result{}, fmt.Errorf("objectlink: the document hashes to %s, not the %s expected", got, strings.ToLower(wantSHA256))
}
return Result{Size: n, SHA256: got}, nil
}
+163
View File
@@ -0,0 +1,163 @@
package objectlink
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
)
// Issue 1592: one helper signs and streams a store link (novox/hq ADR 0468 §3).
type roundTrip func(*http.Request) (*http.Response, error)
func (f roundTrip) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
func hexSum(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) }
// 1592/AC1: AWS's documented query-presign example (a GET of test.txt, 86400 s, 2013-05-24) gives the documented signature.
func TestPresignMatchesAWSDocumentedExample(t *testing.T) {
now := time.Date(2013, 5, 24, 0, 0, 0, 0, time.UTC)
query := sign("GET", "examplebucket.s3.amazonaws.com", "/test.txt", "AKIAIOSFODNN7EXAMPLE",
"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "us-east-1", now, 86400)
const want = "aeeed9bbccd4d02ee5c0109b86d86835f995330da4c265957d157751f604d404"
if !strings.HasSuffix(query, "&X-Amz-Signature="+want) {
t.Fatalf("the signature differs from AWS's documented one: %s", query)
}
if !strings.Contains(query, "X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request") {
t.Errorf("the credential is not encoded as documented: %s", query)
}
}
// 1592/AC2: an Expires of 0, below 0 or above 600 s is refused; the signed host is the endpoint's host.
func TestPresignRefusesBadExpiresAndKeepsTheHost(t *testing.T) {
r := Request{Method: "PUT", Endpoint: "http://minio.internal:9000", Bucket: "files", Key: "a b/c.pdf",
AccessKey: "ak", SecretKey: "sk", Expires: 10 * time.Minute, Now: time.Unix(1700000000, 0)}
link, err := Presign(r)
if err != nil {
t.Fatal(err)
}
u, err := url.Parse(link)
if err != nil {
t.Fatal(err)
}
if u.Host != "minio.internal:9000" || u.Path != "/files/a b/c.pdf" || u.Query().Get("X-Amz-Expires") != "600" ||
u.Query().Get("X-Amz-SignedHeaders") != "host" {
t.Errorf("the link is %s", link)
}
for _, e := range []time.Duration{0, -time.Second, 500 * time.Millisecond, 601 * time.Second, time.Hour} {
r.Expires = e
if _, err := Presign(r); err == nil {
t.Errorf("Expires %s was accepted", e)
}
}
}
// 1592/AC3: Put streams and answers size and SHA-256 as the server saw them, always with the declared Content-Length;
// an unknown size, one past the limit, or a body not ending as declared is refused; a 403 with an S3 error names its code.
func TestPutStreamsCountsAndRefuses(t *testing.T) {
var gotBody []byte
var lengths []int64
store := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotBody, _ = io.ReadAll(r.Body)
lengths = append(lengths, r.ContentLength)
if r.URL.Query().Get("deny") != "" {
w.WriteHeader(403)
io.WriteString(w, `<?xml version="1.0"?><Error><Code>SignatureDoesNotMatch</Code><Message>nope</Message></Error>`)
}
}))
defer store.Close()
data := bytes.Repeat([]byte("0123456789"), 100)
// A reader with no Len or Seek: still sent with the declared length.
got, err := Put(context.Background(), store.Client(), store.URL+"/b/k", io.MultiReader(bytes.NewReader(data)), 1000, 0)
if err != nil {
t.Fatal(err)
}
if got.Size != int64(len(gotBody)) || got.SHA256 != hexSum(gotBody) || got.Size != 1000 {
t.Errorf("answered %+v, the server received %d bytes %s", got, len(gotBody), hexSum(gotBody))
}
if _, err := Put(context.Background(), store.Client(), store.URL+"/b/k", bytes.NewReader(data), 1000, 1000); err != nil {
t.Errorf("exactly the limit: %v", err)
}
for _, l := range lengths {
if l != 1000 {
t.Errorf("a request carried Content-Length %d, not the declared 1000", l)
}
}
if len(lengths) != 2 {
t.Fatalf("the server saw %d requests", len(lengths))
}
// Refused before any request: unknown (negative) size, and a size past the limit.
if _, err := Put(context.Background(), store.Client(), store.URL+"/b/k", bytes.NewReader(data), -1, 0); err == nil || !strings.Contains(err.Error(), "size") {
t.Errorf("an unknown size: %v", err)
}
if _, err := Put(context.Background(), store.Client(), store.URL+"/b/k", bytes.NewReader(data), 1000, 999); err == nil || !strings.Contains(err.Error(), "ADR 0468 §3") {
t.Errorf("a size past the limit: %v", err)
}
if len(lengths) != 2 {
t.Errorf("a refused call made a request: the server saw %d", len(lengths))
}
// A body longer or shorter than declared fails, and the store is not told it succeeded.
reads := &http.Client{Transport: roundTrip(func(r *http.Request) (*http.Response, error) {
_, err := io.Copy(io.Discard, r.Body)
if err == nil {
return &http.Response{StatusCode: 200, Body: io.NopCloser(strings.NewReader(""))}, nil
}
return nil, err
})}
for name, declared := range map[string]int64{"longer than declared": 999, "shorter than declared": 1001} {
if _, err := Put(context.Background(), reads, store.URL+"/b/k", bytes.NewReader(data), declared, 0); err == nil {
t.Errorf("a body %s was accepted", name)
}
}
_, err = Put(context.Background(), store.Client(), store.URL+"/b/k?deny=1", bytes.NewReader(data), 1000, 0)
e, ok := err.(*StoreError)
if !ok || e.Status != 403 || e.Code != "SignatureDoesNotMatch" || !strings.Contains(e.Error(), "SignatureDoesNotMatch") {
t.Fatalf("the refusal is %v", err)
}
}
// 1592/AC4: Get writes the body and answers size and SHA-256; a wrong digest and a document past the limit are errors.
func TestGetStreamsChecksAndRefuses(t *testing.T) {
data := bytes.Repeat([]byte("abcdefghij"), 100)
store := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write(data)
}))
defer store.Close()
var out bytes.Buffer
got, err := Get(context.Background(), store.Client(), store.URL+"/b/k", &out, 0, hexSum(data))
if err != nil {
t.Fatal(err)
}
if got.Size != 1000 || got.SHA256 != hexSum(data) || !bytes.Equal(out.Bytes(), data) {
t.Errorf("answered %+v, wrote %d bytes", got, out.Len())
}
if _, err := Get(context.Background(), store.Client(), store.URL+"/b/k", io.Discard, 0, hexSum([]byte("other"))); err == nil {
t.Error("a wrong digest was accepted")
}
if _, err := Get(context.Background(), store.Client(), store.URL+"/b/k", io.Discard, 999, ""); err == nil {
t.Error("a document past the limit was accepted")
}
chunked := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.(http.Flusher).Flush()
w.Write(data)
}))
defer chunked.Close()
if _, err := Get(context.Background(), chunked.Client(), chunked.URL+"/b/k", io.Discard, 999, ""); err == nil {
t.Error("a chunked document past the limit was accepted")
}
if _, err := Get(context.Background(), chunked.Client(), chunked.URL+"/b/k", io.Discard, 1000, ""); err != nil {
t.Errorf("exactly the limit: %v", err)
}
}
+30
View File
@@ -4,8 +4,11 @@ package outward
import (
"encoding/base64"
"encoding/json"
"fmt"
"math/rand"
"os"
"path/filepath"
"testing"
)
@@ -163,6 +166,33 @@ func TestCheckSecretLetsACamelCaseNameThrough(t *testing.T) {
}
}
// 1439/AC1: a name written in lowercase words and digits between `-`, `_` or `/`, every piece short, is a
// name and not a secret: mesh-issues' owners list (the catalogue's 134 defaults plus okioki) was refused
// whole as "a long random-looking string" for modules/asus-zephyrus-g14, i3status-rust and powerlevel10k.
func TestCheckSecretLetsALowercaseNameWithDigitsThrough(t *testing.T) {
for _, text := range []string{
"mesh-catalog modules/asus-zephyrus-g14",
"mesh-catalog modules/i3status-rust",
"mesh-catalog modules/powerlevel10k",
} {
if r, ok := CheckSecret(text); !ok {
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
}
}
raw, err := os.ReadFile(filepath.Join("testdata", "owners-1439.json"))
if err != nil {
t.Fatal(err)
}
var owners []string
if err := json.Unmarshal(raw, &owners); err != nil || len(owners) != 135 {
t.Fatalf("testdata holds %d owners (%v), want the 134 defaults plus okioki", len(owners), err)
}
compact, _ := json.Marshal(owners)
if r, ok := CheckSecret("+ owners: " + string(compact)); !ok {
t.Errorf("CheckSecret withheld the owners list: %s", r)
}
}
// 1439/AC2: what looks random is still refused: a mixed-case run, a base64url token with `-` and `_`, a
// lowercase run of letters and digits with no separator, and a lowercase name with one long random piece.
func TestCheckSecretStillRefusesRandomRunsBesideNames(t *testing.T) {
-37
View File
@@ -1,37 +0,0 @@
package outward
import (
"encoding/json"
"os"
"testing"
)
// TestReplay1439 is issue 1439, replayed (mesh-lab R1439, novox/hq ADR 0237 §5), written only with what the
// package had before its fix: it fails on the commit before mesh-sdk #52 and passes on it.
//
// 1439/AC1: a name written in lowercase words and digits between `-`, `_` or `/`, every piece short, is a
// name and not a secret: mesh-issues' owners list (the catalogue's 134 defaults plus okioki) was refused
// whole as "a long random-looking string" for modules/asus-zephyrus-g14, i3status-rust and powerlevel10k.
func TestReplay1439(t *testing.T) {
for _, text := range []string{
"mesh-catalog modules/asus-zephyrus-g14",
"mesh-catalog modules/i3status-rust",
"mesh-catalog modules/powerlevel10k",
} {
if r, ok := CheckSecret(text); !ok {
t.Errorf("CheckSecret(%q) withheld it: %s", text, r)
}
}
raw, err := os.ReadFile("replay1439_owners.json")
if err != nil {
t.Fatal(err)
}
var owners []string
if err := json.Unmarshal(raw, &owners); err != nil || len(owners) != 135 {
t.Fatalf("testdata holds %d owners (%v), want the 134 defaults plus okioki", len(owners), err)
}
compact, _ := json.Marshal(owners)
if r, ok := CheckSecret("+ owners: " + string(compact)); !ok {
t.Errorf("CheckSecret withheld the owners list: %s", r)
}
}
+62
View File
@@ -0,0 +1,62 @@
package provider
import (
"context"
"encoding/json"
"slices"
"testing"
)
// Issue 1098: an assignment with `ended` set, or with a `state`, is not a standing assignment: the helper reads it
// as a removed key, so the entry of its consumer is removed and the adapter hears it is gone. The fields the
// controller adds change nothing.
func TestAnEndedAssignmentIsReadAsARemovedKey(t *testing.T) {
bus, m := newStandIn(), &maker{}
v := &vault{values: map[string]string{"pair/novox/postgres-database/ace/letta": "s1"}}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
p := postgres(bus, v, m)
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); !slices.Equal(got, []string{"ace.letta.postgres-database"}) {
t.Fatalf("entries %v", got)
}
for _, mark := range []map[string]any{{"ended": "2026-10-18T01:00:00Z"}, {"state": "pending"}} {
raw := bus.get(AssignmentsBucket, "ace.letta")
var a map[string]any
if err := json.Unmarshal(raw, &a); err != nil {
t.Fatal(err)
}
for k, v := range mark {
a[k] = v
}
raw, _ = json.Marshal(a)
_ = bus.Put(context.Background(), AssignmentsBucket, "ace.letta", raw)
out, err := p.Pass(context.Background())
if err != nil {
t.Fatal(err)
}
if got := bus.keys(BucketOf("postgres")); len(got) != 0 || len(out) != 1 || out[0].Status != Removed {
t.Fatalf("%v: entries %v, outcomes %+v", mark, got, out)
}
assign(t, bus, "ace", "letta", bind("postgres-database", "", "novox", "postgres"))
if _, err := p.Pass(context.Background()); err != nil {
t.Fatal(err)
}
}
}
func TestStandingIsFalseOnlyForAnEndedOrPendingValue(t *testing.T) {
for raw, want := range map[string]bool{
`{"node":"a","module":"b"}`: true,
`{"node":"a","module":"b","ended":"","state":"","x":{"y":1}}`: true,
`{"node":"a","module":"b","ended":null}`: true,
`{"node":"a","module":"b","build":"b1","commit":"c","rollout":"r","since":"t","artifact":{"name":"n"}}`: true,
`{"node":"a","module":"b","ended":"2026-10-18T01:00:00Z"}`: false,
`{"node":"a","module":"b","state":"pending"}`: false,
} {
if got := Standing([]byte(raw)); got != want {
t.Errorf("%s: %v, want %v", raw, got, want)
}
}
}
+23
View File
@@ -335,6 +335,9 @@ func (p *Provider) pass(ctx context.Context, assignments, entries map[string][]b
var out []Outcome
unknown := map[string]bool{} // consumers whose assignment was skipped: their entries are not judged
for _, key := range sortedKeys(assignments) {
if !Standing(assignments[key]) {
continue // ended or pending: read as a removed key (issue 1098)
}
var a planned
if err := json.Unmarshal(assignments[key], &a); err != nil || a.Node+"."+a.Module != key {
o := Outcome{Key: key, Status: Failed, Why: fmt.Sprintf("the assignment %s is not the controller's, so "+
@@ -744,3 +747,23 @@ func (b stateBus) Delete(_ context.Context, bucket, key string) error {
}
return s.Delete(key)
}
// Standing is whether an assignment's value is a standing assignment: false when `ended` is set (the module was
// taken off its node, novox/hq ADR 0408 clause 4) or `state` is not empty (a pending one). Such a value is read
// exactly as a removed key. Every other field, known or not, changes nothing (issue 1098).
func Standing(value []byte) bool {
var v struct {
Ended json.RawMessage `json:"ended"`
State json.RawMessage `json:"state"`
}
if json.Unmarshal(value, &v) != nil {
return true // not decided here: the parse that follows refuses it
}
return !fieldSet(v.Ended) && !fieldSet(v.State)
}
// fieldSet is whether a field carries something: not absent, null or empty.
func fieldSet(f json.RawMessage) bool {
s := strings.TrimSpace(string(f))
return s != "" && s != "null" && s != `""`
}
+4 -4
View File
@@ -138,11 +138,11 @@ func text(t *testing.T, a map[string]any) (string, bool) {
return content[0].(map[string]any)["text"].(string), isError
}
// Two writers incrementing one counter a thousand times each end at two thousand: every number handed out
// Two writers incrementing one counter two hundred times each end at four hundred: every number handed out
// once, none lost, each writer's numbers rising — read, then written only at the revision read.
func TestTwoWritersIncrementingOneCounterLoseNoUpdate(t *testing.T) {
r := &casRuntime{keys: map[string]StateEntry{}, writes: map[string]int{}}
const each = 1000
const each = 200 // enough to collide every run (about a third of writes are lost), cheap on a slow builder
got := counting(t, r, func() (any, error) {
var wg sync.WaitGroup
numbers := make([][]uint64, 2)
@@ -179,7 +179,7 @@ func TestTwoWritersIncrementingOneCounterLoseNoUpdate(t *testing.T) {
}
return len(seen), nil
})
if out, isError := text(t, got); isError || out != "2000" {
if out, isError := text(t, got); isError || out != fmt.Sprint(2*each) {
t.Fatalf("two writers counted %s (error: %v)", out, isError)
}
var final uint64
@@ -190,7 +190,7 @@ func TestTwoWritersIncrementingOneCounterLoseNoUpdate(t *testing.T) {
if r.lost == 0 {
t.Fatal("the two writers never collided: the test proved nothing about contention")
}
t.Logf("2000 increments, %d writes lost to the other writer and tried again", r.lost)
t.Logf("%d increments, %d writes lost to the other writer and tried again", 2*each, r.lost)
if r.writes["create"] != 1 || r.writes["update"] != 2*each-1 || r.writes["put"] != 0 {
t.Fatalf("written by %v: one create, then updates at the revision read, never a put", r.writes)
}
+2 -2
View File
@@ -7,10 +7,10 @@ The copy carries the catalogue's private details: domains, first names in module
private addresses. That is acceptable while mesh-sdk is private; if it ever goes public, this copy is
a second place to clean besides the catalogue itself.
Only what go/catalogue's tests read is captured (R4b, issue 1379). Refreshed 2026-10-20 from the mains of both
Only what go/catalogue's tests read is captured (R4b, issue 1379). Refreshed 2026-10-21 from the mains of both
repositories; before that it was mesh-controller's copy, which still held two modules that had left the catalogue.
mesh-catalog bb0fe93bbdddd869ad808210352461efc506c3d3 modules/*/module.json, modules/*/files/**, modules/*/templates/**, predefined.json
mesh-catalog b5dbfbb784507e211372a7c9640d91f7d6dadd84 modules/*/module.json, modules/*/files/**, modules/*/templates/**, predefined.json
mesh-host cdcba9ca73755854d21e4e657d7e46c32b9b5265 internal/apply/placement_guard.go
To move them, from this repository's go/ directory, with the two repositories checked out beside mesh-sdk:
View File
View File
View File
@@ -0,0 +1,9 @@
{
"scanner": "",
"scan-folder": "",
"printer": "",
"printer-certificate": "",
"resolution": 300,
"color-mode": "color",
"crop": true
}
@@ -0,0 +1,9 @@
{
"scheme": "${bound:s3-bucket:scheme}",
"at": "${bound:s3-bucket:at}",
"port": "${bound:s3-bucket:port}",
"bucket": "${bound:s3-bucket:bucket}",
"region": "${bound:s3-bucket:region}",
"key": "${bound:s3-bucket:as}",
"secret": "${secret:s3-bucket}"
}
@@ -0,0 +1,280 @@
{
"module": "brother",
"version": "1",
"requires": [
"s3-bucket"
],
"binds": {
"s3-bucket": "${dir:state}/store-binding.json"
},
"secrets": {
"s3-bucket": "${dir:state}/store.secret"
},
"config": {
"scanner": {
"type": "string",
"description": "The network scanner's eSCL address, such as http://192.0.2.10/eSCL: the scanner's own address and the path its eSCL service answers on; the operator's, with no default.",
"pattern": "^https?://.+",
"trusted": true,
"required": true
},
"scan-folder": {
"type": "string",
"description": "The absolute path of the folder the scans are filed in as PDF files; the module makes it when it files the first scan, and never touches a file in it that it did not write. The operator's, with no default.",
"pattern": "^/.+$",
"trusted": true,
"required": true
},
"printer": {
"type": "string",
"description": "The printer's host, such as 192.0.2.10: where its admin page (HTTPS) and its IPP service answer. The device is the scanner's own; the operator's, with no default.",
"pattern": "^[A-Za-z0-9.:-]+$",
"trusted": true,
"required": true
},
"printer-certificate": {
"type": "string",
"description": "The SHA-256 fingerprint of the certificate the printer's admin page serves, as 32 upper-case hex pairs joined by colons; several, separated by commas, when the printer serves more than one (it serves an ECDSA and an RSA certificate for one name). The module trusts those certificates alone and refuses any other; the operator's, with no default.",
"pattern": "^([0-9A-F]{2}:){31}[0-9A-F]{2}(, *([0-9A-F]{2}:){31}[0-9A-F]{2})*$",
"trusted": true,
"required": true
},
"resolution": {
"type": "number",
"description": "The resolution asked of the scanner, in dpi: 100, 200, 300 or 600. A scanner may give another; the scan is filed at the resolution it gave.",
"enum": [
100,
200,
300,
600
],
"trusted": false,
"default": 300
},
"color-mode": {
"type": "string",
"description": "Whether the scan is in colour or in grays: color or gray.",
"enum": [
"color",
"gray"
],
"trusted": false,
"default": "color"
},
"crop": {
"type": "boolean",
"description": "Whether the document is cut out of the platen's white lid; false files the whole platen. A feeder scan is never cropped.",
"trusted": false,
"default": true
}
},
"own-secrets": {
"admin-password": {
"path": "${dir:state}/admin-password.secret",
"issued-by": "outside",
"prompt": "the admin password of the Brother printer",
"owner": "brother-device"
},
"broker-device": {
"path": "${dir:state}/broker-device.secret",
"owner": "brother-device"
}
},
"streams": [
{
"name": "device-asks",
"events": [
"device.asked"
],
"max-age-days": 1,
"max-mib": 8,
"retention": "work-queue",
"max-deliveries": 3,
"duplicates": 24
}
],
"state": [
{
"name": "device-answers",
"ttl-seconds": 600
}
],
"seats": [
{
"name": "scanner",
"scope": "mesh",
"serves": [
"scan",
"status",
"list",
"document"
]
},
{
"name": "printer",
"scope": "mesh",
"serves": [
"status",
"settings",
"print",
"upload"
]
}
],
"claims": [
{
"name": "scanner",
"scope": "mesh",
"serves": [
"scan",
"status",
"list",
"document"
]
},
{
"name": "printer",
"scope": "mesh",
"serves": [
"status",
"settings",
"print",
"upload"
]
}
],
"emits": [
"scan.added"
],
"tools": [
"scanner.scan",
"scanner.status",
"scanner.list",
"scanner.document",
"printer.status",
"printer.settings",
"printer.print",
"printer.upload",
"brother_health"
],
"resources": [
{
"id": "account",
"type": "user",
"name": "brother-device",
"shell": "/usr/bin/nologin"
},
{
"id": "state",
"type": "directory",
"mode": "0711",
"owner": "${machine:account}",
"place": ".",
"unknown": "program"
},
{
"id": "settings",
"type": "file",
"path": "${dir:state}/settings.json",
"owner": "${machine:account}",
"mode": "0644",
"takes-config": [
"scanner",
"scan-folder",
"printer",
"printer-certificate",
"resolution",
"color-mode",
"crop"
],
"merge": "json",
"template": "files/settings.json",
"trusted": true
},
{
"id": "store",
"type": "file",
"path": "${dir:state}/store.json",
"owner": "brother-device",
"mode": "0600",
"template": "files/store.json"
},
{
"id": "device",
"type": "process",
"name": "brother-device",
"artifact": "tools",
"run": [
"./brother",
"device"
],
"user": "brother-device",
"env": {
"BROTHER_SETTINGS": "${dir:state}/settings.json",
"BROTHER_ADMIN_PASSWORD": "${dir:state}/admin-password.secret",
"BROTHER_BROKER": "${dir:state}/broker-device.secret",
"BROTHER_STORE": "${dir:state}/store.json"
},
"restart-on": [
"settings",
"store"
],
"bus": {
"queues": [
"device-asks"
],
"state": [
"device-answers"
],
"emits": [
"scan.added"
]
},
"health": {
"kind": "unit"
}
},
{
"id": "watch",
"type": "process",
"name": "brother-watch",
"artifact": "tools",
"run": [
"./brother",
"watch"
],
"user": "brother-device",
"health": {
"kind": "tool",
"tool": "brother_health",
"interval": "60s",
"timeout": "10s",
"looks": 2,
"grace": "60s"
}
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/brother",
"binary": "brother",
"loads": [
"brother"
],
"env": {
"BROTHER_SETTINGS": "${dir:state}/settings.json"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
View File
View File
View File
@@ -4,12 +4,11 @@
"capabilities": [
"package-manager"
],
"requires": [
"go-toolchain",
"node-toolchain"
],
"resources": [
{
"id": "go",
"type": "package",
"package": "go"
},
{
"id": "c-compiler",
"type": "package",
@@ -20,16 +19,6 @@
"type": "package",
"package": "make"
},
{
"id": "nodejs",
"type": "package",
"package": "nodejs"
},
{
"id": "npm",
"type": "package",
"package": "npm"
},
{
"id": "python",
"type": "package",
View File
@@ -540,6 +540,8 @@
],
"reads": [
"mesh-controller.assignments",
"ssh-client.keys"
"ssh-client.keys",
"mesh-build-go.keys",
"mesh-build-node.keys"
]
}
@@ -0,0 +1,20 @@
{
"module": "go",
"version": "1",
"capabilities": [
"package-manager"
],
"provides": [
{
"name": "go-toolchain",
"reach": "machine"
}
],
"resources": [
{
"id": "go",
"type": "package",
"package": "go"
}
]
}
View File
View File
View File
View File
@@ -6,7 +6,9 @@
"virtualisation"
],
"requires": [
"npm-package-registry"
"npm-package-registry",
"go-toolchain",
"node-toolchain"
],
"contributes": {
"npm-package-registry": {
@@ -84,16 +86,6 @@
"type": "package",
"package": "iproute2"
},
{
"id": "npm",
"type": "package",
"package": "npm"
},
{
"id": "go",
"type": "package",
"package": "go"
},
{
"id": "incus",
"type": "package",
View File
@@ -0,0 +1,7 @@
Host ${bound:forge-account:ssh_alias}
HostName ${bound:forge-account:ssh_host}
Port ${bound:forge-account:ssh_port}
HostKeyAlias ${bound:forge-account:ssh_alias}
User git
IdentityFile ~/.ssh/mesh_ed25519
IdentitiesOnly yes
@@ -0,0 +1,2 @@
[url "ssh://${bound:forge-account:ssh_alias}/"]
insteadOf = ${bound:forge-account:https_base}
@@ -0,0 +1 @@
${bound:forge-account:ssh_alias} ${bound:forge-account:host_key}
@@ -0,0 +1,148 @@
{
"module": "mesh-build-go",
"version": "1",
"moved": true,
"runs-as": "mesh-build-go",
"capabilities": [
"package-manager",
"user-namespaces"
],
"state": [
"toolchains",
"keys",
"steps"
],
"tools": [
"build_go_fetch",
"build_go_test",
"build_go_digest",
"build_go_build",
"build_go_step",
"build_go_steps",
"build_go_toolchains",
"build_go_status",
"build_go_health"
],
"streams": [
{
"name": "toolchains-go",
"events": ["toolchain.**"]
}
],
"requires": [
"forge-account",
"go-toolchain"
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-build-go",
"runtime-health": {
"kind": "tool",
"tool": "build_go_health",
"interval": "60s",
"timeout": "10s",
"looks": 2,
"grace": "60s"
},
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-build-go",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-build-go"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-build-go",
"place": ".",
"unknown": "program"
},
{
"id": "work",
"type": "directory",
"path": "/var/lib/mesh-build-go-work",
"mode": "0700",
"owner": "mesh-build-go",
"unknown": "program"
},
{
"id": "cache",
"type": "directory",
"path": "/var/lib/mesh-build-go-cache",
"mode": "0700",
"owner": "mesh-build-go",
"unknown": "program"
},
{
"id": "ssh-dir",
"type": "directory",
"path": "${dir:state}/.ssh",
"mode": "0700",
"owner": "mesh-build-go",
"unknown": "program"
},
{
"id": "ssh-config",
"type": "file",
"path": "${dir:ssh-dir}/config",
"owner": "mesh-build-go",
"mode": "0600",
"template": "files/forge"
},
{
"id": "known-hosts",
"type": "file",
"path": "${dir:ssh-dir}/known_hosts",
"owner": "mesh-build-go",
"mode": "0644",
"template": "files/known_hosts"
},
{
"id": "git-config",
"type": "file",
"path": "${dir:state}/.config/git/config",
"owner": "mesh-build-go",
"mode": "0644",
"template": "files/gitconfig"
},
{
"id": "git",
"type": "package",
"package": "git"
},
{
"id": "openssh",
"type": "package",
"package": "openssh"
},
{
"id": "postgresql",
"type": "package",
"package": "postgresql"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-build-go",
"binary": "mesh-build-go",
"loads": [
"mesh-build-go"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,7 @@
Host ${bound:forge-account:ssh_alias}
HostName ${bound:forge-account:ssh_host}
Port ${bound:forge-account:ssh_port}
HostKeyAlias ${bound:forge-account:ssh_alias}
User git
IdentityFile ~/.ssh/mesh_ed25519
IdentitiesOnly yes
@@ -0,0 +1,2 @@
[url "ssh://${bound:forge-account:ssh_alias}/"]
insteadOf = ${bound:forge-account:https_base}
@@ -0,0 +1 @@
${bound:forge-account:ssh_alias} ${bound:forge-account:host_key}
@@ -0,0 +1,169 @@
{
"module": "mesh-build-node",
"version": "1",
"moved": true,
"runs-as": "mesh-build-node",
"capabilities": [
"package-manager",
"user-namespaces"
],
"state": [
"toolchains",
"keys",
"steps"
],
"tools": [
"build_node_fetch",
"build_node_test",
"build_node_build",
"build_node_step",
"build_node_steps",
"build_node_toolchains",
"build_node_status",
"build_node_health"
],
"streams": [
{
"name": "toolchains-node",
"events": [
"toolchain.**"
]
}
],
"requires": [
"forge-account",
"npm-package-registry",
"node-toolchain"
],
"contributes": {
"npm-package-registry": {
"access": "read"
}
},
"binds": {
"npm-package-registry": "${dir:state}/package-registry.json"
},
"secrets": {
"npm-package-registry": "${dir:state}/package-registry.secret"
},
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-build-node",
"runtime-health": {
"kind": "tool",
"tool": "build_node_health",
"interval": "60s",
"timeout": "10s",
"looks": 2,
"grace": "60s"
},
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-build-node",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-build-node"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-build-node",
"place": ".",
"unknown": "program"
},
{
"id": "work",
"type": "directory",
"path": "/var/lib/mesh-build-node-work",
"mode": "0700",
"owner": "mesh-build-node",
"unknown": "program"
},
{
"id": "cache",
"type": "directory",
"path": "/var/lib/mesh-build-node-cache",
"mode": "0700",
"owner": "mesh-build-node",
"unknown": "program"
},
{
"id": "ssh-dir",
"type": "directory",
"path": "${dir:state}/.ssh",
"mode": "0700",
"owner": "mesh-build-node",
"unknown": "program"
},
{
"id": "ssh-config",
"type": "file",
"path": "${dir:ssh-dir}/config",
"owner": "mesh-build-node",
"mode": "0600",
"template": "files/forge"
},
{
"id": "known-hosts",
"type": "file",
"path": "${dir:ssh-dir}/known_hosts",
"owner": "mesh-build-node",
"mode": "0644",
"template": "files/known_hosts"
},
{
"id": "git-config",
"type": "file",
"path": "${dir:state}/.config/git/config",
"owner": "mesh-build-node",
"mode": "0644",
"template": "files/gitconfig"
},
{
"id": "toolchain-dir",
"type": "archive",
"path": "${dir:state}/toolchain",
"owner": "mesh-build-node",
"artifact": "toolchain",
"unknown": "program"
},
{
"id": "git",
"type": "package",
"package": "git"
},
{
"id": "openssh",
"type": "package",
"package": "openssh"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-build-node",
"binary": "mesh-build-node",
"loads": [
"mesh-build-node"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
},
{
"name": "toolchain",
"kind": "archive",
"from": "toolchain"
}
]
}
}
@@ -1,95 +0,0 @@
{
"module": "mesh-delivery",
"version": "1",
"slug": "deliver",
"claims": [
{
"name": "mesh-delivery",
"scope": "mesh",
"serves": [
"deliveries",
"times",
"show",
"groups",
"what-if",
"checks",
"table",
"stalled",
"recheck",
"release",
"stop",
"close",
"retire-history"
]
}
],
"consumes": [
"gitea.pull.updated",
"gitea.pull.merged",
"gitea.pull.closed",
"gitea.forge.status",
"gitea.forge.pull-request.edited",
"gitea.forge.pull-request.closed",
"mesh-controller.checked",
"mesh-controller.plan-moved"
],
"emits": [
"transition",
"group",
"merged"
],
"invokes": [
"seat:mesh-controller.delivery-plan",
"seat:mesh-controller.delivery-check",
"seat:mesh-controller.deliver",
"seat:mesh-controller.delivery-backups-again",
"seat:mesh-controller.delivery-stop",
"seat:mesh-controller.delivery-walks",
"seat:mesh-controller.forget-checks",
"gitea.gitea_note_append",
"gitea.gitea_delivery_view",
"gitea.gitea_commit_status",
"gitea.gitea_commit_statuses",
"gitea.gitea_contains",
"gitea.gitea_open_pulls",
"gitea.gitea_merge_pull_request",
"gitea.gitea_add_comment",
"seat:operator-channel.notify"
],
"uses": [
"operator-channel"
],
"state": [
"deliveries",
"groups",
"asked",
"pulls"
],
"tools": [
"delivery_status"
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": ".",
"unknown": "program"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-delivery",
"binary": "mesh-delivery",
"loads": [
"mesh-delivery"
]
}
]
}
}
@@ -57,7 +57,6 @@
"consumes": [
"mesh-issues.opened",
"mesh-issues.moved",
"mesh-delivery.transition",
"mesh-delivery-gate.transition"
],
"invokes": [
@@ -0,0 +1,3 @@
{
"callers": ["mesh-mail-web"]
}
@@ -0,0 +1 @@
postgresql://${bound:postgres-database:as}:${secret:postgres-database}@${bound:postgres-database:at}:${bound:postgres-database:port}/${bound:postgres-database:as}
@@ -0,0 +1,141 @@
{
"module": "mesh-mail-reader",
"version": "1",
"runs-as": "mesh-mail-reader",
"claims": [
{
"name": "mail-reading",
"scope": "mesh",
"serves": ["entries", "groups", "correct", "lessons", "undo", "status"]
}
],
"consumes": [
"mesh-mail.mail.*.message.added"
],
"invokes": [
"seat:mail-store.read",
"seat:mail-store.accounts",
"claude-code.claude_code_judge",
"claude-code.claude_code_judged",
"messenger.messenger_check",
"seat:operator-channel.tell"
],
"requires": ["postgres-database"],
"contributes": {
"postgres-database": {
"name": "mail-reader"
}
},
"binds": {
"postgres-database": "${dir:state}/database.json"
},
"secrets": {
"postgres-database": "${dir:state}/database.secret"
},
"config": {
"callers": {
"type": "list",
"description": "Who may call one of mail-reading's six verbs (entries, groups, correct, lessons, undo, status): a module's name (on every machine), <machine>.<module>, or person.<name> (novox/hq ADR 0459 §2). mesh-mail-web alone by default.",
"trusted": true,
"default": ["mesh-mail-web"]
}
},
"tools": [
"mail-reading.entries",
"mail-reading.groups",
"mail-reading.correct",
"mail-reading.lessons",
"mail-reading.undo",
"mail-reading.status",
"mail_reader_health"
],
"runtime-health": {
"kind": "tool",
"tool": "mail_reader_health",
"interval": "60s",
"timeout": "10s",
"looks": 2,
"grace": "60s"
},
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-mail-reader",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-mail-reader",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-mail-reader"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-mail-reader",
"place": ".",
"unknown": "program"
},
{
"id": "callers",
"trusted": true,
"type": "file",
"path": "${dir:state}/callers.json",
"mode": "0600",
"owner": "mesh-mail-reader",
"merge": "json",
"template": "files/callers.json",
"takes-config": ["callers"]
},
{
"id": "database-url",
"type": "file",
"path": "${dir:state}/database.url",
"mode": "0600",
"owner": "mesh-mail-reader",
"template": "files/database-url"
},
{
"id": "prepare",
"type": "process",
"name": "mesh-mail-reader-prepare",
"artifact": "tools",
"run": [
"./mesh-mail-reader",
"prepare"
],
"run-once": true,
"env": {
"DATABASE_URL_FILE": "${dir:state}/database.url"
},
"restart-on": [
"database-url"
]
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-mail-reader",
"binary": "mesh-mail-reader",
"loads": [
"mesh-mail-reader"
],
"env": {
"DATABASE_URL_FILE": "${dir:state}/database.url",
"MESH_MAIL_READER_CALLERS": "${dir:state}/callers.json"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -15,7 +15,13 @@
"seat:mail-store.search",
"seat:mail-store.read",
"seat:mail-store.send",
"seat:mail-store.send-status"
"seat:mail-store.send-status",
"seat:mail-reading.entries",
"seat:mail-reading.groups",
"seat:mail-reading.status",
"seat:mail-reading.correct",
"seat:mail-reading.lessons",
"seat:mail-reading.undo"
],
"reads": ["mesh-mail.accounts"],
"requires": ["route", "oidc-client"],
@@ -18,6 +18,11 @@
"scope": "node",
"needs": ["node-notifier"],
"serves": ["status"]
},
{
"name": "mail-reading",
"scope": "mesh",
"serves": ["entries", "groups", "correct", "lessons", "undo", "status"]
}
],
"claims": [
@@ -1,4 +1,5 @@
{
"time-zone": "",
"only-what-needs-me": []
"only-what-needs-me": [],
"messages-per-hour": 100
}

Some files were not shown because too many files have changed in this diff Show More