Compare commits

...
Author SHA1 Message Date
jschoubben a073aeae5e Merge remote-tracking branch 'origin/main' into feat/try-speed 2026-10-25 03:17:34 +01:00
jschoubben f9d11acf23 Merge pull request 'A node's own composition may carry filled paths; a composition that would leave a provisioner file empty is refused (issue 1944)' (#79) from fix/r6-node-composition into main
Reviewed-on: #79
2026-10-25 00:17:36 +00:00
jschoubben 10b6e20232 Let the node's own composition carry its files' filled paths, and refuse a composition that would have a provisioner's file written empty (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/check checking 10b6e202320f on da7f0455759a
2026-10-25 01:46:36 +02:00
jschoubben 99407c37df Merge remote-tracking branch 'origin/main' into feat/try-speed 2026-10-25 01:29:22 +02:00
jschoubben 9b408db24a buildstep: run a check's levels side by side and answer them in level order, so the integration level no longer waits for the unit level and the first failing level is still named first (issue 1921, try speed S10) 2026-10-25 01:22:50 +02:00
jschoubben da7f045575 Merge pull request 'R6: the composition's layout, its signature and digest, the want's digest, the judgement values; the hand tool is mesh-hand-compose (issue 1943)' (#78) from feat/r6 into main
Reviewed-on: #78
2026-10-24 23:17:32 +00:00
jschoubben 571ac521f0 delivery: keep a merged member in its group's candidate, marked merged, so the heads an Approve covers hold until every member merged (issue 1921, issue 1791, try speed S9) 2026-10-25 01:17:16 +02:00
jschoubben 4a12ccb78d buildstep: deepen a merge preview's history from the mirror when it holds the trunk commit, so the preview fetches no history from the forge (issue 1921, try speed S6) 2026-10-25 01:08:39 +02:00
jschoubben cb548a6a58 buildstep: keep what one command of an ask writes into the caches for its later commands, so vet, unit tests and the integration compile build each package once; the shared cache stays unchanged (issue 1921, try speed S2) 2026-10-25 00:58:46 +02:00
jschoubben bc16740feb Run every test step of a check's level after a failure, and let a verdict name every failed test (issue 1921)
A check ask stopped its level at the first test step that did not pass,
so the packages after the changed ones never ran and a check found one
fault per round. Every step now runs; the level still fails, and no
later level runs (ADR 0484 clause 13). The verdict gains Failures, every
failed test with its first line, at most MaxFailures and a count.
2026-10-24 23:41:54 +02:00
jschoubben 748c564cdf Merge main (the R4g finish: the composer command) into feat/r6 (issue 1943)
mesh/delivery delivering: 0 machine step(s) passed
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/check failed: the composition: the composer of the mesh-sdk mesh-controller at bdb8193456e17c9df59215c7f3e69a83a2e5a262 pins did not run: failed …
main's mesh-compose is the composer command (ADR 0482 clause 4) and keeps its name, which composer.Package and the
build steps name; feat/r6's hand path tool, which composes one module for one node and signs it with the operator's key
(ADR 0539 clause 10), moves to cmd/mesh-hand-compose.
2026-10-24 23:01:18 +02:00
jschoubben 312878befb Merge pull request 'R4g finish: the composer's step contract and the reads-outside marker (issues 1621-1623, 1935)' (#77) from feat/r4g-finish into main
Reviewed-on: #77
2026-10-24 20:56:35 +00:00
jschoubben d7a9b9688b Let the judgement beside a composition say unchanged: equal to the declaration the node last applied (issue 1943; ADR 0540 clause 9)
The first want at the switch skips the record hold only where the composition changes nothing on the node.
2026-10-24 21:17:14 +02:00
jschoubben 7f9cc0e55c Keep plan-moved and sent among the controller seat's events until the catalogue's main consumes them no more (issue 1943)
The controller's broker tests read the catalogue's main, whose gate and messenger still consume plan-moved; nothing
emits them (ADR 0539).
2026-10-24 20:38:24 +02:00
jschoubben e45bdc0891 Drop plan-moved and sent from the controller seat's events (issue 1943; the review's notes on 1945)
No plan is kept or sent in release 6 (ADR 0539), and the messenger now hears the delivery view's events instead.
2026-10-24 20:20:48 +02:00
jschoubben a523a8cbb0 Say beside a composition that it changes only the bus's user list (issue 1943; the review's A2, for issue 1906)
A revision that changes only the user list is written into the bus node's want at once, outside the record hold
(ADR 0539 clause 7), so the assigner must tell it apart from a move that recreates or moves data.
2026-10-24 18:15:32 +02:00
jschoubben 5da87a0dda Let the controller say a bucket it deleted at its first start (issue 1943; for issue 1945, B13)
ADR 0539 clause 13 has the controller delete the plan-state buckets once and say each as its event.
2026-10-24 17:03:54 +02:00
jschoubben e64486b0e6 Sign a composition through a signer that keeps its private half (issue 1943, S5)
The controller's identity store signs a message and never hands its key out, as it signed declarations.
2026-10-24 16:42:41 +02:00
jschoubben 64a502d16a Carry the module's upgrade policy and its source beside a composition's judgement (issue 1943; for issue 1906)
The assigner holds a move of a module of policy record, and only the controller knows a person's own choice of
policy (ADR 0539 clause 14).
2026-10-24 16:40:45 +02:00
jschoubben de5b33843f Run the captured declarations' test per node in parallel, inside the unit level's bound (issue 1943) 2026-10-24 16:36:42 +02:00
jschoubben d8da8f1d36 Sign a composition with a trusted key and compose one by hand with mesh-compose (issue 1943, S5)
The node-engine applies only a composition signed by the controller or the operator, and when the controller cannot
compose its own fix the operator composes, signs and writes one at the terminal (ADR 0539 clauses 3 and 10).
2026-10-24 16:36:14 +02:00
jschoubben d506f2196a Refuse the plan verbs by name on the controller and delivery seats (issue 1943, S4)
With no delivery plan nothing starts, stops, retries or reads one; each removed verb stays a row with no input so a
caller is told its replacement, optional on the delivery seat so the gate of before and of release 6 both hold it
(ADR 0539 clauses 15, 17 and 18).
2026-10-24 16:32:11 +02:00
jschoubben d6cef5829b Have a want name its composition and a backup answer; give the node its own want, the seventh rule, the held phase and the backup and record asks (issue 1943, S3)
The node agent applies only the composition its want names, and the assigner writes a want only once that
composition exists (ADR 0539 clauses 1, 6, 14 and 16).
2026-10-24 16:29:24 +02:00
jschoubben 826fd73cf2 Give a composition its value, its canonical digest and its files by digest; key a dotted module escaped (issue 1943, S2; issue 1150)
A node applies one module's composition at a time, so each carries only its own module's resources, checked
resource by resource as the node-engine reads them, and its digest follows its content alone (ADR 0539 clauses 1
and 4). The fixtures are the controller's plan answers for ace and g14, their sealed resources taken out.
2026-10-24 16:26:32 +02:00
jschoubben aee971c198 Name a node's composition bucket, its keys, and the digest with the move's judgement beside it (issue 1943, S1)
The controller writes one module's composition per node and fingerprint, and the assigner reads its digest and
judgement before it writes a want (ADR 0539 clauses 1, 2 and 14).
2026-10-24 16:21:02 +02:00
jschoubben bfc3883af9 Read a manifest's fallback to a predefined key in the composer as the indexer reads it, or against the list a request or -catalogue gives: a run never holds the catalogue's predefined.json, and the composition refused pacman and xterm for it (ADR 0348 §7)
mesh/delivery delivered
mesh/delivery-group group feat/r4g-finish merged: every member is merged
mesh/check checking bfc3883af980 on 8c6e18e82286 | novox/mesh-sdk#77 in group feat/r4g-finish
2026-10-24 14:57:10 +02:00
jschoubben 19c1d35939 Name the pinned mesh-sdk version when an SDK command is built from a reader's tree, not a bare (devel): the compose job builds mesh-compose there and its verdict must name the reader's SDK (ADR 0482 clause 5)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: the composition: the manifest of pacman cannot be read: this manifest cannot be used: | 19c1d35939bc on 8c6e18e82286 | unit 4799 ru…
2026-10-24 14:11:20 +02:00
jschoubben bc0c25e510 Merge main into feat/r4g-finish: the controller seat's verb held and event artifacts-gone (#76)
mesh/delivery-group group feat/r4g-finish rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: ask 1: answer built true key composer-066…
2026-10-24 13:25:44 +02:00
jschoubben 8c6e18e822 Merge pull request 'Give the controller seat the verb held and the event artifacts-gone (issue 1937)' (#76) from fix/1937-done-key-blobs into main
Reviewed-on: #76
2026-10-24 10:35:07 +00:00
jschoubben d9fa5536a8 Mark the packages whose tests read files outside their directory, found by the review (issue 1935) 2026-10-24 04:38:58 +02:00
jschoubben dd0d156b9e Wait for the third run to start, not only for its take, so the capacity test does not race under a loaded suite 2026-10-24 04:25:12 +02:00
jschoubben 51bc6a65b6 Merge remote-tracking branch 'origin/feat/r4g-finish-1935' into join/r4g-finish-sdk 2026-10-24 04:23:53 +02:00
jschoubben 66bf26c0d7 Let the composer read its request from a file, since a run step hands it no standard input (issue 1622) 2026-10-24 04:10:31 +02:00
jschoubben 34f210e5eb Give the controller seat the verb held and the event artifacts-gone, so the builder rebuilds a done key whose blob the artifact store no longer holds (issue 1937)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792875397682979170 (novox/mesh-sdk at 312878be into main, done, …
mesh/delivery-group group fix/1937-done-key-blobs closed: a member was closed before the group merged
mesh/check pass 34f210e5eba1 on e1f393bbb12e | unit 202 run, 3523 kept | integration 2223 run, 2682 kept, 7 skipped | over bound: unit, integration (s…
The store's nightly collector deletes every bare blob no manifest names, and a done key the controller never took in holds
nothing: node-tools, mesh-host and the forge bridge kept done keys naming blobs that answer 404. held lets the builder ask
before it reuses a key; artifacts-gone is said when a take-in finds a gone blob. delivery/held.go is the shared contract.
2026-10-24 04:07:22 +02:00
jschoubben 4784762a70 Name every repository's siblings in beside's comment, mesh-controller beside mesh-host's tests, as the checker places them (issue 1295) 2026-10-24 03:51:35 +02:00
jschoubben 668fc2a722 Let the Go build seat carry a compose ask and its answer, so the assigner composes a group with its mesh-sdk head (issue 1622) 2026-10-24 03:51:35 +02:00
jschoubben 20d18eb4f5 Give mesh-sdk a composer command, so a program linking one mesh-sdk can have another commit's composition judged (issue 1621) 2026-10-24 03:51:35 +02:00
52 changed files with 4432 additions and 374 deletions
+39 -19
View File
@@ -8,6 +8,7 @@ import (
"slices"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -170,16 +171,36 @@ func (s AskSteps) check(ctx context.Context, job delivery.CheckJob) (any, error)
if job.Runner != "" && (job.RunnerWhen == "" || changesUnder(f.Changed, job.RunnerWhen)) {
runner = job.Runner
}
for _, level := range job.Levels {
la := delivery.LevelAnswer{Level: level}
passed, err := s.level(ctx, job, level, f, runner, &la)
answer.Levels = append(answer.Levels, la)
if err != nil {
return nil, err
}
if !passed {
return answer, nil // the first level that did not pass ends the check (ADR 0484 clause 13)
// The levels run side by side (issue 1921, try speed S10; the operator's yes to D1, ADR 0541): the integration level
// does not wait for the unit level. They are answered in their order, so the first level that did not pass is still
// the first failure the asker reads (ADR 0484 clause 13); the levels after it are kept in the answer for the log.
type ran struct {
la delivery.LevelAnswer
passed bool
err error
}
runs := make([]ran, len(job.Levels))
var wg sync.WaitGroup
for i, level := range job.Levels {
wg.Add(1)
go func() {
defer wg.Done()
runs[i].la = delivery.LevelAnswer{Level: level}
runs[i].passed, runs[i].err = s.level(ctx, job, level, f, runner, &runs[i].la)
}()
}
wg.Wait()
failed := false
for _, r := range runs {
if r.err != nil {
if failed {
continue // a level after the first failure that could not run: the failure is the answer
}
answer.Levels = append(answer.Levels, r.la)
return nil, r.err
}
answer.Levels = append(answer.Levels, r.la)
failed = failed || !r.passed
}
return answer, nil
}
@@ -208,6 +229,9 @@ func (s AskSteps) level(ctx context.Context, job delivery.CheckJob, level string
_ = json.Unmarshal(r, &t)
return t.Passed, nil
}
// Every test step of the level runs, also after one that did not pass, so one check names every failing test of the
// level and not only the first (issue 1921); the level as a whole passes only when each of its steps did.
all := true
if s.Digest == nil {
// A module with no digest (Node) tests each module directory of the tree, those whose tests changed first.
for _, priority := range []bool{true, false} {
@@ -219,14 +243,12 @@ func (s AskSteps) level(ctx context.Context, job delivery.CheckJob, level string
if err != nil {
return false, fmt.Errorf("the %s level of %s: %w", level, dir, err)
}
if !passed {
return false, nil
}
all = all && passed
}
}
return true, nil
return all, nil
}
todo, all, keepRunner := s.plan(ctx, job, level, f, runner, withRunner, la)
todo, whole, keepRunner := s.plan(ctx, job, level, f, runner, withRunner, la)
if !keepRunner {
runner = "" // an installed module that ignores the runner judges the level as it judges any other (ADR 0535 §2)
}
@@ -235,7 +257,7 @@ func (s AskSteps) level(ctx context.Context, job delivery.CheckJob, level string
if job.Run != "" {
args["run"] = job.Run
}
if all {
if whole {
if priority {
continue // nothing to put first: the level runs whole after
}
@@ -255,11 +277,9 @@ func (s AskSteps) level(ctx context.Context, job delivery.CheckJob, level string
if err != nil {
return false, fmt.Errorf("the %s level: %w", level, err)
}
if !passed {
return false, nil
}
all = all && passed // the packages left still run (issue 1921)
}
return true, nil
return all, nil
}
// release removes an ask's workspace, saying a failure: the module's sweep removes it later.
+60
View File
@@ -0,0 +1,60 @@
package buildstep
import (
"context"
"encoding/json"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1921/S10 (the operator's yes to D1, ADR 0541): a check's levels run side by side: the integration level has started
// before the unit level ends. The answer holds the levels in their order, so with the unit level failing it still names
// the unit level first; the integration level's answer is kept after it for the log.
func TestACheckRunsItsLevelsSideBySideAndAnswersThemInOrder(t *testing.T) {
integrationStarted := make(chan struct{})
var once sync.Once
var mu sync.Mutex
unitWaited := false
steps := AskSteps{
Fetch: func(context.Context, map[string]any) (any, error) {
return map[string]any{"workspace": "/w", "modules": []string{"."}}, nil
},
Test: func(ctx context.Context, args map[string]any) (any, error) {
switch args["level"] {
case delivery.LevelUnit:
select {
case <-integrationStarted:
mu.Lock()
unitWaited = true
mu.Unlock()
case <-time.After(2 * time.Second):
}
return map[string]any{"passed": false, "level": "unit"}, nil
default:
once.Do(func() { close(integrationStarted) })
return map[string]any{"passed": true, "level": "integration"}, nil
}
},
}
out, err := RunAsk(context.Background(), steps, ask(delivery.CheckJob{Levels: []string{delivery.LevelUnit, delivery.LevelIntegration},
Fetch: map[string]any{"repository": "novox/x", "commit": runCommit}}))
if err != nil {
t.Fatal(err)
}
mu.Lock()
defer mu.Unlock()
if !unitWaited {
t.Error("the integration level did not start while the unit level ran")
}
a := out.(delivery.CheckAnswer)
if len(a.Levels) != 2 || a.Levels[0].Level != delivery.LevelUnit || a.Levels[1].Level != delivery.LevelIntegration {
t.Fatalf("levels %+v, want unit then integration", a.Levels)
}
var first struct{ Passed bool }
if err := json.Unmarshal(a.Levels[0].Tests[0], &first); err != nil || first.Passed {
t.Errorf("the first level's result %s, want the unit level's failure", a.Levels[0].Tests[0])
}
}
+58 -8
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
@@ -16,6 +17,7 @@ const runCommit = "0123456789abcdef0123456789abcdef01234567"
// recorder is a build module's step tools that record each call.
type recorder struct {
mu sync.Mutex // a check's levels run side by side (issue 1921, S10)
calls []string
fetch map[string]any
tests map[string]bool // package or module directory -> passed
@@ -26,7 +28,9 @@ func (r *recorder) steps() AskSteps {
call := func(name string, answer func(args map[string]any) any) StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
b, _ := json.Marshal(args)
r.mu.Lock()
r.calls = append(r.calls, name+" "+string(b))
r.mu.Unlock()
return answer(args), nil
}
}
@@ -109,9 +113,9 @@ func TestRunAskRefusesAnotherToolchain(t *testing.T) {
}
}
// 1887/A3, A5a, issue 1887 design limit 1: a check ask runs the levels of one check after one fetch: it digests the unit
// level, takes the kept verdicts, tests the packages whose tests changed first, and stops at the first failure, running
// no later level.
// 1887/A3, A5a: a check ask runs the levels of one check after one fetch: it digests the unit level, takes the kept
// verdicts, tests the packages whose tests changed first, then the rest also after a failure (issue 1921); the levels run
// side by side and are answered in order, the first that did not pass first (ADR 0484 clause 13; issue 1921 S10).
func TestRunAskChecksTheLevelsOfOneCheck(t *testing.T) {
r := &recorder{
fetch: map[string]any{"workspace": "/w", "changed": []string{"pkg/b/b_test.go"}},
@@ -123,19 +127,26 @@ func TestRunAskChecksTheLevelsOfOneCheck(t *testing.T) {
if err != nil {
t.Fatal(err)
}
got := strings.Join(r.calls, "\n")
var unit []string
for _, c := range r.calls {
if !strings.Contains(c, `"level":"integration"`) {
unit = append(unit, c)
}
}
got := strings.Join(unit, "\n")
want := `fetch {"base":"` + runCommit + `","commit":"` + runCommit + `","repository":"novox/x"}
digest {"level":"unit","workspace":"/w"}
test {"level":"unit","packages":["x/pkg/b"],"workspace":"/w"}`
test {"level":"unit","packages":["x/pkg/b"],"workspace":"/w"}
test {"level":"unit","packages":["x/pkg/a"],"workspace":"/w"}`
if got != want {
t.Fatalf("calls:\n%s\nwant:\n%s", got, want)
}
a := out.(delivery.CheckAnswer)
if len(a.Levels) != 1 {
t.Fatalf("levels %+v, want the unit level alone", a.Levels)
if len(a.Levels) != 2 || a.Levels[1].Level != delivery.LevelIntegration {
t.Fatalf("levels %+v, want the unit level first, the integration level after it", a.Levels)
}
u := a.Levels[0]
if u.Level != delivery.LevelUnit || len(u.Kept) != 1 || u.Kept[0].Package != "x/pkg/k" || u.Kept[0].Tests != 7 || len(u.Tests) != 1 || u.Digest == nil {
if u.Level != delivery.LevelUnit || len(u.Kept) != 1 || u.Kept[0].Package != "x/pkg/k" || u.Kept[0].Tests != 7 || len(u.Tests) != 2 || u.Digest == nil {
t.Fatalf("the unit level %+v", u)
}
}
@@ -203,3 +214,42 @@ func TestRunAskChecksNodeModulesAndConflicts(t *testing.T) {
t.Fatalf("a conflict: %v %v", err, r.calls)
}
}
// Issue 1921: a failure in the packages whose tests changed does not stop the level: the other packages still run, so two
// failing packages, one changed and one not, are two test results of the one level, and the level does not pass. A Node
// check runs every module directory after one fails too.
func TestRunAskRunsEveryPackageAfterAFailure(t *testing.T) {
r := &recorder{
fetch: map[string]any{"workspace": "/w", "changed": []string{"pkg/b/b_test.go"}},
digests: map[string]string{"x/pkg/a": "d-a", "x/pkg/b": "d-b"},
tests: map[string]bool{"x/pkg/a": false, "x/pkg/b": false},
}
out, err := RunAsk(context.Background(), r.steps(), ask(delivery.CheckJob{Levels: []string{delivery.LevelIntegration, delivery.LevelUnit},
Fetch: map[string]any{"repository": "novox/x", "commit": runCommit}}))
if err != nil {
t.Fatal(err)
}
a := out.(delivery.CheckAnswer)
if len(a.Levels) != 2 || len(a.Levels[0].Tests) != 2 || a.Levels[0].Level != delivery.LevelIntegration {
t.Fatalf("levels %+v, want the first level with both packages' results, in the order asked", a.Levels)
}
for i, raw := range a.Levels[0].Tests {
var res struct{ Passed bool }
if err := json.Unmarshal(raw, &res); err != nil || res.Passed {
t.Errorf("result %d %s, want a failure", i, raw)
}
}
r = &recorder{fetch: map[string]any{"workspace": "/w", "modules": []string{"a", "b"}, "changed": []string{"b/test/x.test.ts"}},
tests: map[string]bool{"a": true, "b": false}}
s := r.steps()
s.Digest = nil
out, err = RunAsk(context.Background(), s, ask(delivery.CheckJob{Levels: []string{delivery.LevelUnit, delivery.LevelIntegration},
Fetch: map[string]any{"repository": "novox/x", "commit": runCommit}}))
if err != nil {
t.Fatal(err)
}
if a := out.(delivery.CheckAnswer); len(r.calls) != 5 || len(a.Levels) != 2 || len(a.Levels[0].Tests) != 2 || a.Levels[0].Level != delivery.LevelUnit {
t.Fatalf("calls %v, levels %+v: want both directories of each level, the unit level first", r.calls, a.Levels)
}
}
+114 -10
View File
@@ -3,10 +3,13 @@ package buildstep
import (
"context"
"fmt"
"io/fs"
"os"
"path"
"path/filepath"
"strconv"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
@@ -29,16 +32,25 @@ const Bubblewrap = "bwrap"
// - the Workspace and Writable directories read-write, the ReadOnly ones (the toolchain, the modules the fetch step
// downloaded) read-only;
// - the shared warm Caches read-only under a throwaway writable layer (S18), or read-write for a trunk build alone
// (WriteCaches), so a candidate's code never changes what a later build reads.
// (WriteCaches), so a candidate's code never changes what a later build reads;
// - with a CacheLayer, what one command writes into the caches kept for the later commands of the same ask (issue 1921,
// the try speed-up): the vet, the unit tests and the integration compile of one ask compile each package once.
type Isolation struct {
Workspace string
Writable []string
ReadOnly []string
Caches []string
WriteCaches bool
Env []string
// CacheLayer is a directory of the ask's own, in its workspace and gone with it: each command writes the caches into
// a layer of its own beneath it (overlayfs refuses an upper layer two mounts share, and an ask's commands run side by
// side), which Isolated moves into the ask's kept layer when the command ends; every later command reads the kept
// layer above the shared cache. The shared cache stays unchanged. Empty: what a command writes is thrown away.
CacheLayer string
Env []string
// Hosts is the file bound as /etc/hosts; Isolated writes one naming only the loopback.
Hosts string
// run is the directory of this command's own writable layers under CacheLayer; Isolated makes one per command.
run string
}
// etcNeeded is what of /etc a program in the step may read: who the users are, where the libraries are, the
@@ -91,7 +103,7 @@ func (iso Isolation) command(env []string, name string, args ...string) ([]strin
for _, group := range []struct {
what string
dirs []string
}{{"writable directory", iso.Writable}, {"read-only directory", iso.ReadOnly}, {"cache", iso.Caches}} {
}{{"writable directory", iso.Writable}, {"read-only directory", iso.ReadOnly}, {"cache", iso.Caches}, {"cache layer", layerDirs(iso.CacheLayer)}} {
for _, d := range group.dirs {
if err := checkDir(group.what, d); err != nil {
return nil, err
@@ -113,7 +125,7 @@ func (iso Isolation) command(env []string, name string, args ...string) ([]strin
for _, d := range iso.ReadOnly {
argv = append(argv, "--ro-bind", d, d)
}
argv = append(argv, cacheArgs(iso.Caches, iso.WriteCaches)...)
argv = append(argv, iso.cacheArgs()...)
argv = append(argv, "--bind", iso.Workspace, iso.Workspace)
for _, d := range iso.Writable {
argv = append(argv, "--bind", d, d)
@@ -134,19 +146,107 @@ func (iso Isolation) command(env []string, name string, args ...string) ([]strin
// cacheArgs binds the shared warm caches (S18): for a trunk build read-write, so the next build is warm; for any other
// read-only beneath an overlay whose writes go to a tmpfs of the step's own, so the step writes as a build does and
// nothing it wrote outlives it (bubblewrap's --tmp-overlay, an overlay mount in the step's user namespace).
func cacheArgs(caches []string, write bool) []string {
// nothing it wrote outlives it (bubblewrap's --tmp-overlay, an overlay mount in the step's user namespace). With a
// CacheLayer the ask's kept layer lies above the shared cache and the command's writes go to its own layer on disk
// (--overlay), which Isolated keeps for the ask when the command ends.
func (iso Isolation) cacheArgs() []string {
var argv []string
for _, c := range caches {
if write {
for i, c := range iso.Caches {
switch {
case iso.WriteCaches:
argv = append(argv, "--bind", c, c)
continue
case iso.CacheLayer != "":
upper, work := iso.runLayer(i)
argv = append(argv, "--overlay-src", c, "--overlay-src", keptLayer(iso.CacheLayer, i), "--overlay", upper, work, c)
default:
argv = append(argv, "--overlay-src", c, "--tmp-overlay", c)
}
argv = append(argv, "--overlay-src", c, "--tmp-overlay", c)
}
return argv
}
func layerDirs(layer string) []string {
if layer == "" {
return nil
}
return []string{layer}
}
// keptLayer is where the ask keeps what its commands wrote into the cache i.
func keptLayer(layer string, i int) string { return filepath.Join(layer, "kept", strconv.Itoa(i)) }
// runLayer is the upper and work directory of this command's writes into the cache i. Command, which makes no
// directory, names the layer "run"; Isolated gives each command one of its own.
func (iso Isolation) runLayer(i int) (upper, work string) {
run := iso.run
if run == "" {
run = filepath.Join(iso.CacheLayer, "run")
}
return filepath.Join(run, strconv.Itoa(i), "upper"), filepath.Join(run, strconv.Itoa(i), "work")
}
// layerLocks hold one mutex per cache layer, so two commands ending at once keep their writes one after the other.
var layerLocks sync.Map
// prepareLayer makes the kept layers and a run directory of the command's own under the cache layer.
func (iso *Isolation) prepareLayer() error {
if iso.CacheLayer == "" || iso.WriteCaches || len(iso.Caches) == 0 {
return nil
}
if err := os.MkdirAll(filepath.Join(iso.CacheLayer, "runs"), 0o700); err != nil {
return err
}
run, err := os.MkdirTemp(filepath.Join(iso.CacheLayer, "runs"), "run-")
if err != nil {
return err
}
iso.run = run
for i := range iso.Caches {
upper, work := iso.runLayer(i)
for _, d := range []string{keptLayer(iso.CacheLayer, i), upper, work} {
if err := os.MkdirAll(d, 0o700); err != nil {
return err
}
}
}
return nil
}
// keepLayer moves what the command wrote into the caches into the ask's kept layers, and removes its run directory.
// A file the command wrote replaces the kept one (a cache's newest write wins); a whiteout (a file the command
// deleted) is not kept, since a cache entry deleted by one command is no fault of a later one. A write that cannot be
// kept only means a later command builds it again.
func (iso Isolation) keepLayer() {
if iso.run == "" {
return
}
defer os.RemoveAll(iso.run)
mu, _ := layerLocks.LoadOrStore(iso.CacheLayer, &sync.Mutex{})
mu.(*sync.Mutex).Lock()
defer mu.(*sync.Mutex).Unlock()
for i := range iso.Caches {
upper, _ := iso.runLayer(i)
kept := keptLayer(iso.CacheLayer, i)
filepath.WalkDir(upper, func(p string, d fs.DirEntry, err error) error {
if err != nil || p == upper {
return nil
}
rel, _ := filepath.Rel(upper, p)
to := filepath.Join(kept, rel)
switch {
case d.IsDir():
if fi, err := os.Lstat(to); err == nil && !fi.IsDir() {
os.Remove(to)
}
os.MkdirAll(to, 0o700)
case d.Type().IsRegular():
os.Rename(p, to)
}
return nil
})
}
}
// loopbackHosts is the hosts file of a step: the loopback and nothing else.
const loopbackHosts = "127.0.0.1 localhost\n::1 localhost\n"
@@ -166,6 +266,10 @@ func Isolated(run forgekey.Runner, iso Isolation) forgekey.Runner {
}
iso := iso
iso.Hosts = hosts
if err := iso.prepareLayer(); err != nil {
return forgekey.Ran{Status: 1, Err: "the step's cache layer: " + err.Error()}
}
defer iso.keepLayer()
argv, err := iso.command(env, name, args...)
if err != nil {
return forgekey.Ran{Status: 1, Err: err.Error()}
+36
View File
@@ -64,3 +64,39 @@ func TestARunStepReachesNothingAndLeavesTheCachesUnchanged(t *testing.T) {
t.Fatalf("the step's write to its workspace did not stay (status %d, %s%s): %q %v", r.Status, r.Stderr, r.Err, b, err)
}
}
// 1921/S2: in one ask (one cache layer), a second go build of the same package does not compile it again, and the
// shared cache is unchanged afterwards.
func TestASecondBuildOfTheAskFindsTheFirstsCompiledAndTheSharedCacheIsUnchanged(t *testing.T) {
if _, err := exec.LookPath(Bubblewrap); err != nil {
t.Fatalf("bubblewrap is not installed here (issue 1309): %v", err)
}
goBin, err := exec.LookPath("go")
if err != nil {
t.Fatal(err)
}
root := t.TempDir()
shared := filepath.Join(root, "gobuild")
if err := os.MkdirAll(shared, 0o700); err != nil {
t.Fatal(err)
}
workspace := filepath.Join(root, "ws")
tree := filepath.Join(workspace, "tree")
writeFile(t, tree, "go.mod", "module probe\n\ngo 1.22\n")
writeFile(t, tree, "probe.go", "package probe\n\nfunc Probe() int { return 42 }\n")
run := Isolated(forgekey.ExecRunnerFor(0), Isolation{Workspace: workspace, Caches: []string{shared},
CacheLayer: filepath.Join(workspace, "cachelayer"), ReadOnly: []string{filepath.Dir(filepath.Dir(goBin))}})
env := []string{"PATH=" + filepath.Dir(goBin) + ":/usr/bin:/bin", "GOCACHE=" + shared, "GOTOOLCHAIN=local", "GOFLAGS=-mod=mod"}
for i, want := range []bool{true, false} {
r := run(context.Background(), env, "go", "-C", tree, "build", "-x", "./...")
if r.Status != 0 || r.Err != "" {
t.Fatalf("build %d: status %d %s\n%s", i+1, r.Status, r.Err, r.Stderr)
}
if compiled := strings.Contains(r.Stderr, "/compile "); compiled != want {
t.Errorf("build %d compiled the package: %v, want %v\n%s", i+1, compiled, want, r.Stderr)
}
}
if entries, _ := os.ReadDir(shared); len(entries) != 0 {
t.Errorf("the shared cache was written: %d entries", len(entries))
}
}
+110
View File
@@ -0,0 +1,110 @@
package buildstep
import (
"context"
"os"
"path/filepath"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// overlayOf answers the upper directory a command line gives the cache c, "" when it gives none.
func overlayOf(argv []string, c string) string {
for i := 0; i+3 < len(argv); i++ {
if argv[i] == "--overlay" && argv[i+3] == c {
return argv[i+1]
}
}
return ""
}
// 1921/S2: with a cache layer, a command reads the ask's kept layer above the shared cache and writes into a layer on
// disk of its own, never into the shared cache and never into a throwaway tmpfs.
func TestACommandWithACacheLayerReadsTheKeptLayerAndWritesItsOwn(t *testing.T) {
fixedSystem(t)
iso := Isolation{Workspace: "/w", Caches: []string{"/cache/gomod", "/cache/gobuild"}, CacheLayer: "/w/cachelayer"}
argv, err := iso.Command("go", "vet")
if err != nil {
t.Fatal(err)
}
for i, c := range iso.Caches {
if !holds(argv, "--overlay-src", c, "--overlay-src", keptLayer(iso.CacheLayer, i), "--overlay") {
t.Errorf("the cache %s is not read beneath the ask's kept layer: %v", c, argv)
}
if up := overlayOf(argv, c); up == "" || filepath.Dir(filepath.Dir(filepath.Dir(up))) != "/w/cachelayer" {
t.Errorf("the cache %s is written into %q, not a layer of the ask's own: %v", c, up, argv)
}
if holds(argv, "--tmp-overlay", c) || holds(argv, "--bind", c, c) {
t.Errorf("the cache %s is thrown away or written: %v", c, argv)
}
}
if _, err := (Isolation{Workspace: "/w", CacheLayer: "cachelayer"}).Command("go"); err == nil {
t.Error("a relative cache layer was taken")
}
}
// 1921/S2: what one command wrote into the caches is kept for the ask's later commands, the shared cache stays
// unchanged, two commands at once write into layers of their own (overlayfs refuses an upper layer two mounts share),
// and a command's own layer is gone when it ends.
func TestWhatOneCommandCompiledIsReadByTheNextOfTheSameAsk(t *testing.T) {
fixedSystem(t)
root := t.TempDir()
shared := filepath.Join(root, "gobuild")
if err := os.MkdirAll(shared, 0o700); err != nil {
t.Fatal(err)
}
layer := filepath.Join(root, "ws", "cachelayer")
var mu sync.Mutex
uppers := map[string]bool{}
gate := make(chan struct{})
var started sync.WaitGroup
started.Add(2)
inner := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
up := overlayOf(args, shared)
mu.Lock()
uppers[up] = true
mu.Unlock()
if holds(args, "--", "go", "build") {
started.Done()
<-gate // both commands are running before either ends
os.MkdirAll(filepath.Join(up, "ab"), 0o700)
os.WriteFile(filepath.Join(up, "ab", args[len(args)-1]+"-d"), []byte("compiled"), 0o600)
}
return forgekey.Ran{}
}
run := Isolated(inner, Isolation{Workspace: filepath.Join(root, "ws"), Caches: []string{shared}, CacheLayer: layer})
var wg sync.WaitGroup
for _, pkg := range []string{"one", "two"} {
wg.Add(1)
go func() { defer wg.Done(); run(context.Background(), nil, "go", "build", pkg) }()
}
started.Wait()
close(gate)
wg.Wait()
if len(uppers) != 2 {
t.Fatalf("two commands at once wrote into %d layers, want 2 of their own: %v", len(uppers), uppers)
}
for _, pkg := range []string{"one", "two"} {
if b, err := os.ReadFile(filepath.Join(keptLayer(layer, 0), "ab", pkg+"-d")); err != nil || string(b) != "compiled" {
t.Errorf("what the build of %s compiled is not kept for the ask: %q %v", pkg, b, err)
}
}
for up := range uppers {
if _, err := os.Stat(up); !os.IsNotExist(err) {
t.Errorf("the command's own layer %s outlived it: %v", up, err)
}
}
var next []string
Isolated(func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
next = args
return forgekey.Ran{}
}, Isolation{Workspace: filepath.Join(root, "ws"), Caches: []string{shared}, CacheLayer: layer})(context.Background(), nil, "go", "test")
if !holds(next, "--overlay-src", shared, "--overlay-src", keptLayer(layer, 0)) {
t.Errorf("the next command does not read the kept layer: %v", next)
}
if entries, _ := os.ReadDir(shared); len(entries) != 0 {
t.Errorf("the shared cache was written: %v", entries)
}
}
+15 -6
View File
@@ -24,7 +24,8 @@ type MergeResult struct {
// signature, no hook, nothing of the machine's git configuration that changes a merge. Nothing is pushed.
//
// A merge needs both histories, so it makes the fetch whole (the shallow clone FetchCommit made is deepened) and
// fetches base by its sha. A head that does not merge cleanly is an answer, not an error: the files in conflict, the
// fetches base by its sha: from the mirror WithMirror names, when it holds base (UpdateMirror to base first), else
// from remote. A head that does not merge cleanly is an answer, not an error: the files in conflict, the
// merge undone and the workspace left at base. A head already in base answers a Merge equal to Base and no changed
// paths. An error is that the merge could not be tried: a base or commit that is not a full sha, no base, a workspace
// not at commit, or a fetch or git command that failed. remote and the options are those of FetchCommit; the runner's
@@ -67,12 +68,20 @@ func MergeInto(ctx context.Context, run forgekey.Runner, dir, remote, base, comm
if err != nil {
return res, err
}
fetch := []string{"fetch", "--no-tags", "origin", base}
if strings.TrimSpace(shallow) == "true" {
fetch = []string{"fetch", "--no-tags", "--unshallow", "origin", base}
fetchFrom := func(source string) error {
fetch := []string{"fetch", "--no-tags", source, base}
if strings.TrimSpace(shallow) == "true" {
fetch = []string{"fetch", "--no-tags", "--unshallow", source, base}
}
_, err := git(fetch...)
return err
}
if _, err := git(fetch...); err != nil {
return res, fmt.Errorf("the trunk commit %s: %w", base, err)
// With a mirror brought to the trunk commit, the history is deepened from it, a local fetch (issue 1921, try speed
// S6); a mirror that cannot answer leaves it to the forge, as before.
if o.mirror == "" || fetchFrom(o.mirror) != nil {
if err := fetchFrom("origin"); err != nil {
return res, fmt.Errorf("the trunk commit %s: %w", base, err)
}
}
if _, err := git("checkout", "-q", "--detach", base); err != nil {
return res, fmt.Errorf("the trunk commit %s cannot be checked out: %w", base, err)
+30
View File
@@ -170,3 +170,33 @@ func TestMergeIntoAHeadAlreadyInTheTrunkChangesNothing(t *testing.T) {
t.Errorf("result = %+v, %v", res, err)
}
}
// 1921/S6: with a mirror brought to the head and to the trunk commit, the merge asks the forge for nothing: it succeeds
// with the forge gone.
func TestMergeIntoWithAMirrorAtTheTrunkNeedsNoForge(t *testing.T) {
r := newMergeRig(t)
head := r.head(map[string]string{"b.txt": "head\n"})
base := r.advance(map[string]string{"a.txt": "one\ntwo\nthree\nfour\n"})
mirror := filepath.Join(t.TempDir(), "mirror.git")
if err := os.MkdirAll(mirror, 0o755); err != nil {
t.Fatal(err)
}
for _, c := range []string{head, base} {
if err := UpdateMirror(context.Background(), forgekey.ExecRunner, mirror, r.remote, c, "", AllowFileRemote()); err != nil {
t.Fatal(err)
}
}
if err := FetchCommit(context.Background(), forgekey.ExecRunner, r.ws, r.remote, head, "", AllowFileRemote(), WithMirror(mirror)); err != nil {
t.Fatal(err)
}
if err := os.RemoveAll(r.bare); err != nil { // the forge is gone
t.Fatal(err)
}
res, err := MergeInto(context.Background(), forgekey.ExecRunner, r.ws, r.remote, base, head, AllowFileRemote(), WithMirror(mirror))
if err != nil {
t.Fatalf("the merge asked the forge: %v", err)
}
if len(res.Merge) != 40 || !reflect.DeepEqual(res.Changed, []string{"b.txt"}) {
t.Errorf("result = %+v", res)
}
}
+36
View File
@@ -72,3 +72,39 @@ func TestMergeIntoRunsTheCheckMergesCommands(t *testing.T) {
}
}
}
// 1921/S6: with a mirror that holds the trunk commit, the merge deepens the workspace from the mirror, a local fetch,
// and asks the forge nothing; a mirror that cannot answer falls back to the forge.
func TestMergeIntoWithAMirrorFetchesTheTrunkFromTheMirror(t *testing.T) {
base := strings.Repeat("b", 40)
mirror := t.TempDir()
answer := func(failMirror bool) *fakeGit {
return &fakeGit{rule: func(l string) forgekey.Ran {
switch {
case strings.HasSuffix(l, "rev-parse HEAD"):
return forgekey.Ran{Stdout: sha + "\n"}
case strings.Contains(l, "--is-shallow-repository"):
return forgekey.Ran{Stdout: "true\n"}
case failMirror && strings.Contains(l, " fetch ") && strings.Contains(l, mirror):
return forgekey.Ran{Status: 128, Stderr: "fatal: not our ref"}
}
return forgekey.Ran{}
}}
}
f := answer(false)
dir := t.TempDir()
if _, err := MergeInto(context.Background(), f.run, dir, "ssh://mesh-forge/o/r.git", base, sha, WithMirror(mirror)); err != nil {
t.Fatal(err)
}
all := strings.Join(f.calls, "\n")
if !strings.Contains(all, "fetch --no-tags --unshallow "+mirror+" "+base) || strings.Contains(all, "--unshallow origin") {
t.Errorf("the trunk was not fetched from the mirror alone:\n%s", all)
}
g := answer(true)
if _, err := MergeInto(context.Background(), g.run, t.TempDir(), "ssh://mesh-forge/o/r.git", base, sha, WithMirror(mirror)); err != nil {
t.Fatal(err)
}
if all := strings.Join(g.calls, "\n"); !strings.Contains(all, "fetch --no-tags --unshallow origin "+base) {
t.Errorf("a mirror that could not answer did not fall back to the forge:\n%s", all)
}
}
+2 -5
View File
@@ -127,11 +127,8 @@ func TestPullHoldsAtMostItsCapacity(t *testing.T) {
}
release <- struct{}{}
waitFor(t, "the third taken", func() bool { return src.Waiting(delivery.AskDelivery) == 0 })
mu.Lock()
if started != 3 {
t.Errorf("started %d, want 3", started)
}
mu.Unlock()
// Taken from the source comes a moment before its run starts: wait for the start, not only the take.
waitFor(t, "the third started", func() bool { mu.Lock(); defer mu.Unlock(); return started == 3 })
close(release)
waitFor(t, "all acked", func() bool { a, _, _ := src.Fates(); return len(a) == 3 })
}
+25 -46
View File
@@ -9,7 +9,26 @@ package catalogue
// DeliverySeat is the seat mesh-delivery holds.
const DeliverySeat = "mesh-delivery"
// deliveryVerbs are the delivery seat's tools: six that read, and the acts of a person, a reviewer and healer H2.
// RemovedDeliveryVerbs are the delivery seat's verbs release 6 removed (novox/hq ADR 0539 clause 17): with no plan the
// gate keeps the merge's states and derives nothing after the merge. Each stays in deliveryVerbs as an optional row
// with no input, so the holder of before and the holder of release 6 both hold the seat and a caller is answered by
// name with its replacement, given here.
var RemovedDeliveryVerbs = map[string]string{
"times": "the delivery view (nox-delivery): each machine's want and have, and since when",
"what-if": "the merge check's composition, which composes every machine a change touches",
"release": "the assigner's asks (hold, restore, backup, record), answered on the phone or in the delivery view",
"stop": "a bad fingerprint is held and its canary set back by the assigner (ADR 0429 §6, ADR 0539 clause 18)",
"retire-history": "nothing to retire: the gate keeps no delivery after its merge",
}
// deliveryRemovedInR6 is the row of a verb release 6 removed from the delivery seat.
func deliveryRemovedInR6(name string) Verb {
return Verb{Name: name, Description: "Removed in R6 (ADR 0539): refused by name, naming its replacement: " +
RemovedDeliveryVerbs[name], Input: schema(nil, nil), Optional: true}
}
// deliveryVerbs are the delivery seat's tools: what reads the merges and checks, and the acts of a person and healer H2
// on a merge (ADR 0539 clause 17).
func deliveryVerbs() []Verb {
return []Verb{
{Name: "deliveries", Description: "Every delivery not final, and those that ended in the last day, one line " +
@@ -18,15 +37,7 @@ func deliveryVerbs() []Verb {
Input: schema(map[string]string{"state": "one state, e.g. delivering or held",
"repository": "owner/repository", "group": "a group's id (its branch name)",
"all": "\"true\": the final ones of the last thirty days too"}, nil, "all")},
// Delivery time against the delivery budgets (novox/hq ADR 0282): what probe D16 reads, optional until its
// holder serves it.
{Name: "times", Description: "Delivery time: from a merge to every machine of its walk running the build. " +
"Each class's delivery budget (a leaf module three minutes, a core module ten), the last days' median and " +
"worst, how many within and over, the newest delivery of each class, every delivery over its budget with its " +
"longest phase, and the delivered ones whose time is not known. Given a delivery's id, its time phase by phase.",
Input: schema(map[string]string{"days": "how many days back (default 7)", "id": "one delivery's id: its phases"}, nil),
// Optional until mesh-delivery serves it: its holder lives in the catalogue (design 33 §7).
Optional: true},
deliveryRemovedInR6("times"),
{Name: "show", Description: "One delivery or group whole: its delivery plan (what it builds, what each " +
"machine receives, what is not an ordinary send), every transition with when and why, the machine " +
"steps of its walk, its group and its order.",
@@ -35,12 +46,7 @@ func deliveryVerbs() []Verb {
"(declared, built by, version skew, engine before controller, by name), its composed check and its " +
"state, derived from its members.",
Input: schema(map[string]string{"all": "\"true\": the groups that ended too"}, nil, "all")},
{Name: "what-if", Description: "The delivery plan a change would have, asked of the controller's planner " +
"and kept nowhere: the modules it moves and their dependents in tiers, what each machine would receive, " +
"the steps that are not an ordinary send.",
Input: schema(map[string]string{"repository": "owner/repository",
"paths": "the files it changes, comma-separated, from the repository's root",
"base": "the branch it merges into (default main)"}, []string{"repository", "paths"})},
deliveryRemovedInR6("what-if"),
{Name: "checks", Description: "What the mesh's checks said of a pull request's head or of one commit: each " +
"of the commit's mesh statuses (mesh/check, the one a main requires from R4e, hq's mesh/repo-check, " +
"mesh/delivery, …) with its state, " +
@@ -62,40 +68,13 @@ func deliveryVerbs() []Verb {
"controller is asked for its check. With why.",
Input: schema(map[string]string{"id": "the delivery's id", "why": "why, kept with the transition"},
[]string{"id", "why"})},
{Name: "release", Description: "A person's word that a held delivery goes on: it starts delivering. With why.",
Input: schema(map[string]string{"id": "the delivery's id", "why": "why, kept with the transition"},
[]string{"id", "why"})},
{Name: "stop", Description: "Stop a delivery that is not final, with why; its walk is ended through the " +
"controller, and in a group every member after it is stopped too, naming it.",
Input: schema(map[string]string{"id": "the delivery's id", "why": "why, kept with the transition"},
[]string{"id", "why"})},
deliveryRemovedInR6("release"),
deliveryRemovedInR6("stop"),
{Name: "close", Description: "Healer H2's verb: take the one transition the table names for a delivery held " +
"past its bound. Refused for any other.",
Input: schema(map[string]string{"id": "the delivery's id", "why": "the condition it answers"},
[]string{"id", "why"})},
{Name: "retire-history", Description: "Retire held deliveries that are history, on a person's word, with " +
"why: each made from the forge's word for a merge on the trunk heard long after it happened — replayed " +
"when the holder first held the seat — with no walk of it open. Never released, delivered or put back: " +
"retired, final. One by id, refused when the rule does not take it; or every held one, of one " +
"repository when named, answering how many it retired and naming those it left held. dry \"true\" " +
"answers what it would do and changes nothing.",
Input: schema(map[string]string{"why": "why, kept with every transition",
"id": "one delivery's id; every held one when empty",
"repository": "owner/repository: only its held deliveries",
"dry": "\"true\": what it would retire, changing nothing"}, []string{"why"}, "dry"),
// Added after the seat's holder shipped (novox/hq issue 313): optional until mesh-delivery serves it.
Optional: true},
{Name: "mark-ready", Description: "A reviewer's word that a delivery's head is approved: the holder puts the " +
"label mesh/ready on its pull request and then writes \"ready at <head>\", the order the forge counts, " +
"as the mesh's own forge account, naming who reviewed it and why. Refused when the delivery's commit is " +
"no longer its pull request's head. mesh-delivery then merges it once its checks pass (novox/hq ADR 0364).",
Input: schema(map[string]string{"id": "the delivery's id (owner/repository@commit): the head reviewed",
"reviewed-by": "who reviewed it, as who@node", "why": "the review's verdict in a line"},
[]string{"id", "reviewed-by", "why"}),
// Kept one more release, optional (novox/hq issue 1022): mesh-delivery no longer serves it, but the manifest
// it still has on the mesh does until its catalogue member merges, and a seat refuses a holder that serves
// a verb it does not promise. The next release removes this entry.
Optional: true},
deliveryRemovedInR6("retire-history"),
}
}
-104
View File
@@ -58,107 +58,3 @@ func TestTheDeliverySeatPromisesChecks(t *testing.T) {
t.Fatalf("a holder without %s: %v", before[0], err)
}
}
// The seat retires history on a person's word (novox/hq issue 313): `retire-history`, with why, one delivery by
// id or every held one, dry to see first. Added after the holder shipped, it is optional, so the holder that
// does not serve it yet still holds the seat, and one that does is not refused for a verb the seat lacks.
func TestTheDeliverySeatPromisesRetireHistoryOptionally(t *testing.T) {
seat, _ := SeatNamed(DeliverySeat)
var retire *Verb
for i := range seat.Serves {
if seat.Serves[i].Name == "retire-history" {
retire = &seat.Serves[i]
}
}
if retire == nil || !retire.Optional {
t.Fatalf("the delivery seat promises %v, retire-history optionally", VerbNames(seat.Serves))
}
props, _ := retire.Input["properties"].(map[string]any)
for _, arg := range []string{"why", "id", "repository", "dry"} {
if _, has := props[arg]; !has {
t.Errorf("retire-history takes no %q", arg)
}
}
if req, _ := retire.Input["required"].([]string); !reflect.DeepEqual(req, []string{"why"}) {
t.Errorf("retire-history requires %v, wanted only why", req)
}
var without []string
for _, v := range VerbNames(seat.Serves) {
if v != "retire-history" {
without = append(without, v)
}
}
m := Manifest{Module: "mesh-delivery", Claims: []Claim{{Name: DeliverySeat, Scope: ScopeMesh, Serves: without}}}
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder without retire-history yet: %v", err)
}
m.Claims[0].Serves = VerbNames(seat.Serves)
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder serving retire-history: %v", err)
}
}
// The seat says delivery times (novox/hq ADR 0282, issue 382): `times`, over some days or for one delivery. Added
// after the holder shipped, it is optional, so the holder that does not serve it yet still holds the seat, and one
// that does is not refused for a verb the seat lacks.
func TestTheDeliverySeatPromisesTimesOptionally(t *testing.T) {
seat, _ := SeatNamed(DeliverySeat)
var times *Verb
for i := range seat.Serves {
if seat.Serves[i].Name == "times" {
times = &seat.Serves[i]
}
}
if times == nil || !times.Optional {
t.Fatalf("the delivery seat promises %v, times optionally", VerbNames(seat.Serves))
}
props, _ := times.Input["properties"].(map[string]any)
for _, arg := range []string{"days", "id"} {
if _, has := props[arg]; !has {
t.Errorf("times takes no %q", arg)
}
}
var without []string
for _, v := range VerbNames(seat.Serves) {
if v != "times" {
without = append(without, v)
}
}
m := Manifest{Module: "mesh-delivery", Claims: []Claim{{Name: DeliverySeat, Scope: ScopeMesh, Serves: without}}}
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder without times yet: %v", err)
}
m.Claims[0].Serves = VerbNames(seat.Serves)
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder serving times: %v", err)
}
}
// The seat still promises `mark-ready`, optionally, for one more release (issue 1022); it was added by novox/hq issue 913: `mark-ready`, with the delivery's id, who reviewed it
// and why, so a reviewer marks a pull request through one audited verb, label and comment in the order the forge
// counts, instead of two raw forge calls. Added after the holder shipped, it is optional.
func TestTheDeliverySeatPromisesMarkReadyOptionally(t *testing.T) {
seat, _ := SeatNamed(DeliverySeat)
var mark *Verb
for i := range seat.Serves {
if seat.Serves[i].Name == "mark-ready" {
mark = &seat.Serves[i]
}
}
if mark == nil || !mark.Optional {
t.Fatalf("the delivery seat promises %v, mark-ready optionally", VerbNames(seat.Serves))
}
if req, _ := mark.Input["required"].([]string); !reflect.DeepEqual(req, []string{"id", "reviewed-by", "why"}) {
t.Errorf("mark-ready requires %v, wanted id, reviewed-by and why", req)
}
var without []string
for _, v := range VerbNames(seat.Serves) {
if v != "mark-ready" {
without = append(without, v)
}
}
m := Manifest{Module: "mesh-delivery", Claims: []Claim{{Name: DeliverySeat, Scope: ScopeMesh, Serves: without}}}
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder without mark-ready yet: %v", err)
}
}
+9 -3
View File
@@ -110,12 +110,18 @@ var defaultSeats = append([]Seat{
"rolled-back",
// A pull request's merge check, judged (novox/hq to-be 45 §9).
"checked",
// A walk kept in a new state, for the delivery it walks (novox/hq ADR 0239).
// A walk kept in a new state, and a tier sent to a machine: kept until the catalogue's main consumes them no
// more; nothing emits them (ADR 0539).
"plan-moved",
// A tier sent to a machine, once the send is kept (novox/hq ADR 0406 §5, issue 974).
"sent",
// A key of a record mirror written or removed (novox/hq ADR 0289, design 54 §3; design 57 F3).
"assignment-changed", "declared-config-changed", "predefined-changed"},
"assignment-changed", "declared-config-changed", "predefined-changed",
// A builder's key not taken in because the artifact store does not hold an artifact it names (novox/hq
// issue 1937): the builder builds it again.
"artifacts-gone",
// A bucket of a state this release deleted, deleted at the controller's first start (novox/hq ADR 0539
// clause 13): the merge window's, the silent sends' and the building plans' hold, each said once.
"bucket-deleted"},
Serves: ControllerVerbs},
// The store's first verbs (novox/hq ADR 0159): the smallest set that makes the store askable,
// served by whichever module holds the seat with tools of these names.
+55 -106
View File
@@ -74,6 +74,29 @@ func VerbNames(verbs []Verb) []string {
return out
}
// RemovedControllerVerbs are the controller seat's verbs release 6 removed (novox/hq ADR 0539, ADR 0429 §8): no
// delivery plan, batch, tier or push exists, so nothing starts, stops, retries, follows or reads one. Each stays in
// ControllerVerbs as a row with no input, so a caller is answered by name with its replacement, given here.
var RemovedControllerVerbs = map[string]string{
"plans": "the delivery view (nox-delivery): each machine's want and have, read from the assigner's wants and the " +
"node-engine's haves",
"plan": "the node's compositions: mesh-controller_composed holds each module's digest and judgement per node, and " +
"mesh-hand-compose prints one module's composition for one node",
"push": "a want: the assigner writes want.<node>.<module> by its rules, and by hand the operator's credential writes " +
"it, naming a kept composition (ADR 0539 clause 10)",
"deliver": "nothing to start: a merged change reaches each machine by its want",
"delivery-stop": "a bad fingerprint is held and its canary set back by the assigner (ADR 0429 §6, ADR 0539 clause 18)",
"delivery-backups-again": "the assigner's backup ask, answered on the phone or in the delivery view (ADR 0539 clause 16)",
"delivery-walks": "the delivery view (nox-delivery) and the wants, haves and compositions on the bus",
"delivery-order": "no order to keep: each module reaches its machines by its own want, its providers first by rule 5",
}
// removedInR6 is the row of a verb release 6 removed from the controller's seat: no input, refused by name.
func removedInR6(name string) Verb {
return Verb{Name: name, Description: "Removed in R6 (ADR 0539): refused by name, naming its replacement: " +
RemovedControllerVerbs[name], Input: schema(nil, nil)}
}
// ControllerSeatName is the seat the control plane holds, whose tools are the mesh's own verbs.
const ControllerSeatName = "mesh-controller"
@@ -115,87 +138,23 @@ var ControllerVerbs = []Verb{
"log": "a build's id (as `builds` lists it): print what the build machine said, line by line",
"limit": "how many builds to list (default 20); not with log",
}, nil)},
{Name: "plans", Description: "What the last merges produced and where each stands (novox/hq ADR 0162): " +
"the tiers, the tier a plan is at, what it waits for and since when; one plan whole, given its id; " +
"or, given wait, one plan followed until it moves.",
Input: schema(map[string]string{
"id": "a plan's id (as `plans` lists them): that plan, tier by tier",
"wait": "a plan's id: wait until it moves, then answer it as data — the plan, and what ended the wait " +
"(moved, tier, done, or bound); ask again to follow it further (novox/hq issue 229)",
"until": "with wait: moved (any change, the default), tier (it reaches another tier) or done (it ends)",
"bound": "with wait: how long to wait at most, as 15s or seconds — 20s, the default, is the longest, " +
"inside what a call is answered within",
"stop": "a plan's id: stop it — what was asked still builds, nothing further is asked",
"close": "a plan's id: close a plan that will not move again, as failed by hand (novox/hq issue 254)",
"retry": "a failed plan's id: ask its failed builds again under new ids, and carry the plan on from that tier (novox/hq ADR 0219)",
"go": "a walk's id that waits for its delivery's word: start it by hand, in place of mesh-delivery — with why, " +
"recorded in the hand-act log (novox/hq ADR 0239)",
"repository": "owner/repository: the plan a merge there would produce, saving nothing (what-if); with paths or modules",
"paths": "with repository: the files the merge would change, comma-separated, from the repository's root",
"modules": "with repository: or the modules it would change, comma-separated",
"module-dirs": "with repository and paths: the directories holding a module.json at the commit, comma-separated, " +
"as the forge's announcer says them (novox/hq issue 278); without it, a directory is a module only when the paths hold its manifest",
"data": "with repository and paths: true answers the change plan as data, the document delivery-plan answers " +
"(moved, dependents, new, tiers, machines with what each receives and waits for, steps, summary, class, head, base); " +
"without it the answer is text (novox/hq ADR 0450 §2)",
"head": "with data: the commit at hand",
"base": "with data: the branch it merges into (default main)",
"removed": "with data: the files among paths the change deletes, comma-separated",
"limit": "how many plans to list (default 10); only when listing",
"why": "with stop, close or go: why it is done by hand — required, and recorded in the hand-act log (novox/hq to-be 45 §7)",
"cause": "with stop, close or go: the cause in a word, or a condition's kind (optional)",
}, nil, "data")},
removedInR6("plans"),
// The verbs the delivery's owner asks with (novox/hq ADR 0239): the planner's answer, a group's order and
// composed check, and the start and end of a walk. Callable by a person too; mesh-delivery is who needs them.
{Name: "delivery-plan", Description: "Removed in R4e (ADR 0501 §5 / ADR 0503 §3): refused by name, naming its replacement",
Input: schema(nil, nil)},
{Name: "delivery-order", Description: "A delivery group's order (novox/hq ADR 0419): its members in the order " +
"they are delivered, every pair and why — built by, verb before caller, version skew, engine before " +
"controller — read from the module graph and the manifests' change alone; no `after:` line is read. The " +
"cycle, with every module and edge of it, is the one refusal.",
Input: schema(map[string]string{"members": "the members, as JSON: [{id, repository, base, head, number, paths, " +
"module_dirs, module_dirs_said, removed}]"}, []string{"members"})},
removedInR6("delivery-order"),
{Name: "delivery-check", Description: "Answers that the checker (mesh-delivery-checker) judges a group (novox/hq " +
"ADR 0503 §3); asks nothing. Its inputs stay, since hq's repository check still names it (ADR 0513 §4).",
Input: schema(map[string]string{"group": "the group's id (none for one head checked again)",
"members": "the members, as JSON, as delivery-order takes them",
"ready": "\"true\" when the pull request is marked ready: its check is queued before every other check (novox/hq ADR 0406 §3)"},
[]string{"members"})},
{Name: "deliver", Description: "The delivery's word that a walk waiting for it may start (novox/hq ADR 0239): its " +
"first tier is asked at the controller's next pass. Refused for a walk that waits for nobody.",
Input: schema(map[string]string{"plan": "the walk's id", "why": "what lets it go: its turn, a person's release"},
[]string{"plan"})},
{Name: "delivery-stop", Description: "End a walk on its delivery's word (novox/hq ADR 0239): failed, said as stopped " +
"by whom and why; what it asked still builds and registers, nothing further is asked or sent. Given merges to " +
"leave out, every other merge the walk answered is walked again — in the open batch of its kind, or a new one — " +
"without them, without each later merge of their repository, which contains one, and without each merge moving " +
"a module one moves, each left out too and named (novox/hq ADR 0299); a batch not yet cut drops the merges. The " +
"answer names the batch, the merges walked again, the merges left out, the modules the stopped walk sent that " +
"stay on their machines, and the merges elsewhere that will still deliver a stopped change.",
Input: schema(map[string]string{"plan": "the walk's id", "why": "why", "by": "who stopped the delivery",
"without": "merges to leave out, <repository>@<commit>, comma-separated"},
[]string{"plan", "why"})},
{Name: "delivery-backups-again", Description: "Ask again the backups a walk was held for (novox/hq ADR 0404 " +
"decision 4): a walk whose backup before its send failed or did not answer is held, nothing asked or sent; this " +
"clears those failed backups and the next look at the plans asks them again, and only them. Refused for a walk " +
"with no failed backup. The delivery's release calls it. The answer names the walk and the backups asked again.",
Input: schema(map[string]string{"plan": "the held walk's id", "why": "a person's reason, as release passes it",
"by": "who released the delivery"}, []string{"plan", "why"})},
{Name: "delivery-walks", Description: "The walks the controller keeps (novox/hq ADR 0239): every open one and the " +
"last ended ones, each whole — its tiers, each module's state, first machines and first-node gate with its readings, and its " +
"phases from its merge to every machine running it (novox/hq ADR 0282) — and whether the delivery seat has a " +
"holder on record. Given a plan, that one.",
Input: schema(map[string]string{"plan": "one walk's id", "limit": "how many ended walks beside the open ones (default 50)"},
nil)},
{Name: "plan", Description: "What one machine would run, and why: the declaration the mesh would send it — " +
"or, with files, the files it would be given; or, with diff, what a push would change there against what " +
"it was last sent: resources added, removed and changed, a container's changed fields (it is recreated), " +
"and any mount whose directory on the machine would move (novox/hq ADR 0217).",
Input: schema(map[string]string{
"node": "the machine's name",
"files": "\"true\": the files this machine would be given, instead of the declaration as JSON",
"diff": "\"true\": what a push would change on this machine, against what it was last sent; not with files",
}, []string{"node"}, "files", "diff")},
removedInR6("deliver"),
removedInR6("delivery-stop"),
removedInR6("delivery-backups-again"),
removedInR6("delivery-walks"),
removedInR6("plan"),
{Name: "assign", Description: "Put a module on a machine. Refused with the mesh's own words when it cannot resolve there, " +
"or when a seat its resources are applied through is held by nothing on the machine (novox/hq ADR 0207). " +
"A module not registered yet is looked for (novox/hq issue 325, ADR 0261): while its build runs — a merge " +
@@ -221,22 +180,7 @@ var ControllerVerbs = []Verb{
}, []string{"node", "provision", "from", "module"})},
{Name: "unpin", Description: "Take that choice back, putting the question to the mesh again.",
Input: schema(map[string]string{"node": "the machine's name", "provision": "the provision"}, []string{"node", "provision"})},
{Name: "push", Description: "Send one machine everything it should be. With no machine named it is a push of the " +
"WHOLE mesh — every machine that is behind — and the answer says so first; behind says that outright. " +
"Answers at once that it is running, with a call id: `calls` with that id says what it sent and, for a " +
"machine named, how that machine applied it — applied, refused with its reason, or each failed resource — " +
"once it reports within wait (a push can reload the bus, which then refuses any answer still to come). A push by hand is a repair, " +
"and says why: recorded in the hand-act log (novox/hq to-be 45 §7). A machine whose push would give a " +
"running module's data another directory is held and named, the others sent (novox/hq ADR 0217): read " +
"`plan` with diff, and name the machine with move to send it.",
Input: schema(map[string]string{
"node": "the machine's name; without it, every machine that is behind",
"move": "with node: the modules whose data may move with this push, comma-separated (optional)",
"wait": "with node: how long the call waits for the machine to report how it applied what it was sent, as \"90s\" or \"2m\"; \"0\" waits for nothing; without it, the controller's setting push-wait (optional)",
"behind": "\"true\": every machine that is behind, the whole mesh — the same as naming none, said outright; not with node",
"why": "why this is pushed by hand: recorded in the hand-act log",
"cause": "the cause in a word, or a condition's kind — the word a second push for the same reason uses (optional)",
}, []string{"why"}, "behind")},
removedInR6("push"),
{Name: "rotate", Description: "Replace a credential. A pair credential, by provision (and a consuming machine and module, " +
"else every holder): both ends are re-sent together. Or a module's own secret, by machine, module and " +
"name: made anew and the machine sent, so the module starts again on it — for a secret its definition " +
@@ -457,26 +401,22 @@ var ControllerVerbs = []Verb{
"probe": "one probe's id (D14): its verdict alone, with every finding — the verdict shows at most twenty a probe",
}, nil, "run", "probes", "signals")},
// How a module's new builds reach its machines, and the bus's planned step (novox/hq ADR 0236).
{Name: "upgrade", Description: "How each module's new builds reach its machines (novox/hq ADR 0236): rolled " +
"out one machine first and judged there at the gate, then the rest — or recorded, waiting for a person's " +
"push — with where that comes from (a person, the module, the bus, its irreplaceable data, the default) and " +
"why. With module, that one; with policy, a person's choice for it — roll-out, record (with why) or default " +
"to take the choice back. The bus is never rolled out.",
{Name: "upgrade", Description: "Whether each module's new builds wait for a person (novox/hq ADR 0236, ADR 0539 clause 14): " +
"for a module of policy record, the assigner writes no want naming a composition judged to recreate a container or move " +
"data until the operator answers its ask; with where the policy comes from (a person, the module, the bus, its " +
"irreplaceable data, the default) and why. With module, that one; with policy, a person's choice for it — roll-out, " +
"record (with why) or default to take the choice back. The bus is never moved by the assigner (ADR 0539 clause 15).",
Input: schema(map[string]string{
"module": "one module",
"policy": "with module: roll-out, record or default",
"together": "\"true\": with roll-out, every machine at once instead of one machine first",
"why": "with policy or release-backlog: why — required for record and for a release, kept",
"backlog": "\"true\": every build waiting for a gate, per machine — what a release plan would send",
"release-backlog": "\"true\": release what waits now, one machine at a time, each judged — after a release " +
"plan failed, the mesh waits for this; with why",
"cause": "with release-backlog: the cause in a word (optional)",
}, nil, "together", "backlog", "release-backlog")},
{Name: "bus", Description: "The bus as a planned step (novox/hq to-be 45 §8, ADR 0236): what a bus upgrade " +
"would do — the bus's build on each machine against the one the mesh holds — and how the last step went. " +
"With upgrade, start one: a person's act with why, after the streams are snapshotted (snapshot-taken says " +
"where, while the mesh takes none itself), saying first whether it can be reverted; bus-maintenance is open " +
"while it runs and every stream, consumer and a round trip are checked after.",
"module": "one module",
"policy": "with module: roll-out, record or default",
"why": "with policy: why — required for record, kept",
}, nil)},
{Name: "bus", Description: "The bus as a person's step (novox/hq to-be 45 §8, ADR 0236, ADR 0539 clause 15): what a bus " +
"upgrade would do — the bus's build on each machine against the one the mesh holds — and how the last step went. " +
"With upgrade, check one: a person's act with why, after the streams are snapshotted (snapshot-taken says where, " +
"while the mesh takes none itself), saying first whether it can be reverted; then it answers the want the operator " +
"writes with the operator's credential, want.<node>.<bus module>, recorded — nothing is sent, and the assigner never " +
"moves the bus.",
Input: schema(map[string]string{
"upgrade": "\"true\": start the bus's upgrade",
"why": "with upgrade: why — required, recorded in the hand-act log",
@@ -566,6 +506,15 @@ var ControllerVerbs = []Verb{
"repository": "only this repository's references, as <module>/<artifact> (the counts stay whole)",
"collected": "\"true\": list the collected references too (they only grow; counted always)",
}, nil, "collected")},
// Whether the artifact store holds what a builder's done key names, before the builder reuses it (novox/hq issue
// 1937): the store's collector deletes a blob nothing holds, and a key the controller never took in holds nothing.
{Name: "held", Description: "Whether the artifact store holds each reference given, asked with one HEAD each: " +
"held, gone (the store answered 404) or unknown with why. The builder asks it before it reuses a done key, and " +
"builds a key whose artifact is gone again. Reads only: nothing is held or written (novox/hq issue 1937).",
Input: listed(schema(map[string]string{
"references": "the artifacts' references, artifact-store://<module>/<artifact>/blobs/sha256:<hex>: a list, " +
"or one text separated by commas",
}, []string{"references"}), "references")},
{Name: "collect", Description: "The artifact store's sweep, now: what the mesh made and keeps for no reason " +
"is let go of. Without confirm a dry run — every kept archive asked about with HEADs, what would be let go " +
"listed, nothing held or deleted. With confirm and why: every kept archive held first, then each eligible " +
+115
View File
@@ -0,0 +1,115 @@
package catalogue
import (
"strings"
"testing"
)
// 1943/AC4: the controller's seat no longer offers a verb that starts, stops, retries, follows or reads a delivery
// plan, nor a push: each is a row with no input whose description refuses it by name and names its replacement, and
// each is in the removed list the holder answers from. upgrade keeps its policy and drops the plan's backlog.
func TestTheControllerSeatHasNoPlanVerbs(t *testing.T) {
rows := map[string]Verb{}
for _, v := range ControllerVerbs {
rows[v.Name] = v
}
gone := []string{"plans", "plan", "push", "deliver", "delivery-stop", "delivery-backups-again", "delivery-walks", "delivery-order"}
for _, name := range gone {
v, ok := rows[name]
if !ok {
t.Errorf("%s is not in the table, so a caller is not answered by name", name)
continue
}
replacement, removed := RemovedControllerVerbs[name]
if !removed || replacement == "" {
t.Errorf("%s is not in the removed list with its replacement", name)
}
if !strings.HasPrefix(v.Description, "Removed in R6 (ADR 0539): refused by name") || !strings.Contains(v.Description, replacement) {
t.Errorf("%s = %q, want refused by name, naming %q", name, v.Description, replacement)
}
if props, _ := v.Input["properties"].(map[string]any); len(props) > 0 {
t.Errorf("%s still takes %v", name, props)
}
}
props, _ := rows["upgrade"].Input["properties"].(map[string]any)
for _, arg := range []string{"together", "backlog", "release-backlog", "cause"} {
if _, has := props[arg]; has {
t.Errorf("upgrade still takes %s, a plan's", arg)
}
}
for _, arg := range []string{"module", "policy", "why"} {
if _, has := props[arg]; !has {
t.Errorf("upgrade no longer takes %s", arg)
}
}
for _, word := range []string{"rolled", "gate", "backlog", "push"} {
if strings.Contains(rows["upgrade"].Description, word) {
t.Errorf("upgrade's description still speaks of %q: %q", word, rows["upgrade"].Description)
}
}
if !strings.Contains(rows["bus"].Description, "want") {
t.Errorf("bus does not say the operator writes the bus's want: %q", rows["bus"].Description)
}
for name := range RemovedControllerVerbs {
if v := rows[name]; !strings.HasPrefix(v.Description, "Removed in R6") {
t.Errorf("%s is in the removed list and its row says %q", name, v.Description)
}
}
}
// 1943/AC4: the delivery seat keeps the merge's verbs and no longer offers times, what-if, release, stop or
// retire-history: each is a row with no input, optional so the holder of before and the holder of release 6 both hold
// the seat, refusing it by name; mark-ready, served by nobody since issue 1022, is gone.
func TestTheDeliverySeatHasNoPlanVerbs(t *testing.T) {
seat, ok := SeatNamed(DeliverySeat)
if !ok {
t.Fatal("the delivery seat is not in the set")
}
rows := map[string]Verb{}
for _, v := range seat.Serves {
rows[v.Name] = v
}
for _, name := range []string{"times", "what-if", "release", "stop", "retire-history"} {
v, ok := rows[name]
replacement, removed := RemovedDeliveryVerbs[name]
if !ok || !removed || replacement == "" {
t.Errorf("%s: in the table %v, in the removed list %v", name, ok, removed)
continue
}
if !strings.HasPrefix(v.Description, "Removed in R6 (ADR 0539): refused by name") || !strings.Contains(v.Description, replacement) || !v.Optional {
t.Errorf("%s = %q (optional %v), want an optional row refusing it by name", name, v.Description, v.Optional)
}
if props, _ := v.Input["properties"].(map[string]any); len(props) > 0 {
t.Errorf("%s still takes %v", name, props)
}
}
if _, ok := rows["mark-ready"]; ok {
t.Errorf("mark-ready is still promised")
}
for _, kept := range []string{"deliveries", "show", "groups", "checks", "table", "recheck", "close", "stalled"} {
if v, ok := rows[kept]; !ok || strings.HasPrefix(v.Description, "Removed") {
t.Errorf("%s is not kept (ADR 0539 clause 17)", kept)
}
}
gate := Manifest{Module: "mesh-delivery", Claims: []Claim{{Name: DeliverySeat, Scope: ScopeMesh,
Serves: []string{"deliveries", "show", "groups", "checks", "table", "stalled", "recheck", "close"}}}}
if err := CanHold(gate, seat); err != nil {
t.Errorf("the gate of release 6 cannot hold the seat: %v", err)
}
gate.Claims[0].Serves = append(gate.Claims[0].Serves, "times", "what-if", "release", "stop", "retire-history")
if err := CanHold(gate, seat); err != nil {
t.Errorf("the gate of release 5 cannot hold the seat: %v", err)
}
}
// 1943 (for 1945/AC14): the controller's seat says a bucket it deleted at its first start as its event, bucket-deleted
// (ADR 0539 clause 13).
func TestTheControllerSaysABucketItDeleted(t *testing.T) {
seat, ok := SeatNamed(ControllerSeatName)
if !ok {
t.Fatal("no controller seat")
}
if !SeatSays(seat.Emits, "bucket-deleted") {
t.Errorf("the controller's seat does not emit bucket-deleted: %v", seat.Emits)
}
}
+163
View File
@@ -0,0 +1,163 @@
// mesh-compose is mesh-sdk's composer command (novox/hq ADR 0482 clause 4, issue 1621): it reads a composer.Request on
// standard input, validates every manifest with this mesh-sdk's reader, composes the machines named in each case and
// writes one composer.Verdict on standard output. Built at any commit of mesh-sdk (or at the mesh-sdk a module pins),
// it answers what that commit's composition answers, so a program that links one mesh-sdk can have another's judged.
//
// mesh-compose < request.json > verdict.json
// mesh-compose -request request.json > verdict.json
// mesh-compose -catalogue <dir> [-node <name>]... [-depth resolve|compose] [-declarations] [< request.json]
//
// -catalogue reads <dir>/modules/*/module.json and a module.json at <dir>'s root as the first case's manifests (a
// checkout or `git archive` of the catalogue repository at a ref: the command fetches nothing). -node adds a machine to
// compose; -declarations answers each machine's declared paths with their owners and modes.
//
// It opens no network connection, bus, store or file beyond its standard input, the -request file and the -catalogue
// directory, and reads
// no environment. Exit codes: 0 the verdict is written (a machine that does not compose is a verdict, not an error);
// 1 the request cannot be read or run, said on standard error; 2 the flags are wrong.
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/composer"
)
func main() {
os.Exit(run(os.Args[1:], os.Stdin, os.Stdout, os.Stderr))
}
type nodes []string
func (n *nodes) String() string { return strings.Join(*n, ",") }
func (n *nodes) Set(v string) error { *n = append(*n, v); return nil }
// run is the command: its arguments, standard input, output and error; it answers the exit code.
func run(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("mesh-compose", flag.ContinueOnError)
fs.SetOutput(stderr)
catalogueDir := fs.String("catalogue", "", "a catalogue checkout: <dir>/modules/*/module.json are the first case's manifests")
var machines nodes
fs.Var(&machines, "node", "a machine to compose (repeatable)")
depth := fs.String("depth", "", "resolve or compose (default: the request's, else compose)")
declarations := fs.Bool("declarations", false, "answer each composed machine's declared paths, owners and modes")
requestFile := fs.String("request", "", "read the request from this file instead of standard input (a run step that hands no standard input)")
if err := fs.Parse(args); err != nil {
return 2
}
if fs.NArg() > 0 {
fmt.Fprintf(stderr, "mesh-compose: unexpected arguments %q\n", fs.Args())
return 2
}
var raw []byte
var err error
if *requestFile != "" {
raw, err = os.ReadFile(*requestFile)
} else {
raw, err = io.ReadAll(stdin)
}
if err != nil {
fmt.Fprintln(stderr, "mesh-compose: the request cannot be read:", err)
return 1
}
var req composer.Request
if strings.TrimSpace(string(raw)) == "" {
if *catalogueDir == "" {
fmt.Fprintln(stderr, "mesh-compose: no request on standard input and no -catalogue")
return 1
}
req = composer.Request{V: composer.Version}
} else if req, err = composer.Decode(raw); err != nil {
fmt.Fprintln(stderr, "mesh-compose:", err)
return 1
}
if *catalogueDir != "" {
read, err := readCatalogue(*catalogueDir)
if err != nil {
fmt.Fprintln(stderr, "mesh-compose:", err)
return 1
}
if len(req.Cases) == 0 {
req.Cases = []composer.Case{{Name: "catalogue", Manifests: map[string]composer.Manifest{}}}
}
if req.Cases[0].Manifests == nil {
req.Cases[0].Manifests = map[string]composer.Manifest{}
}
for name, m := range read {
if _, given := req.Cases[0].Manifests[name]; !given {
req.Cases[0].Manifests[name] = m
}
}
// The catalogue's list of predefined keys is the one its manifests' fallbacks are judged against (ADR 0348 §7).
if req.Predefined == nil {
list, err := catalogue.ReadPredefined(*catalogueDir)
if err != nil {
fmt.Fprintln(stderr, "mesh-compose:", err)
return 1
}
req.Predefined = list
}
}
req.Machines = append(req.Machines, machines...)
if *depth != "" {
req.Depth = *depth
}
if *declarations {
req.Declarations = true
}
v, err := composer.Run(req)
if err != nil {
fmt.Fprintln(stderr, "mesh-compose:", err)
return 1
}
enc := json.NewEncoder(stdout)
if err := enc.Encode(v); err != nil {
fmt.Fprintln(stderr, "mesh-compose: the verdict cannot be written:", err)
return 1
}
return 0
}
// readCatalogue reads every module.json of a catalogue checkout by its module's name: the root's and each of
// modules/*/. A manifest whose name cannot be read is kept under its directory's name, so the reader's refusal is said.
func readCatalogue(dir string) (map[string]composer.Manifest, error) {
paths, err := filepath.Glob(filepath.Join(dir, "modules", "*", "module.json"))
if err != nil {
return nil, err
}
if _, err := os.Stat(filepath.Join(dir, "module.json")); err == nil {
paths = append(paths, filepath.Join(dir, "module.json"))
}
if len(paths) == 0 {
return nil, fmt.Errorf("the catalogue %s holds no module.json", dir)
}
sort.Strings(paths)
out := map[string]composer.Manifest{}
for _, p := range paths {
raw, err := os.ReadFile(p)
if err != nil {
return nil, err
}
var probe struct {
Module string `json:"module"`
}
name := filepath.Base(filepath.Dir(p))
if json.Unmarshal(raw, &probe) == nil && probe.Module != "" {
name = probe.Module
}
if _, twice := out[name]; twice {
return nil, errors.New("the catalogue " + dir + " defines " + name + " twice")
}
out[name] = composer.Manifest{Raw: raw}
}
return out, nil
}
+347
View File
@@ -0,0 +1,347 @@
package main
import (
"bytes"
"encoding/json"
"go/build"
"os"
"os/exec"
"path/filepath"
"reflect"
"runtime"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/internal/beside"
)
// captured is every manifest of the catalogue captured in testdata/beside (internal/beside), raw, by module name.
func captured(t *testing.T) map[string]composer.Manifest {
t.Helper()
read, err := readCatalogue(filepath.Dir(beside.Catalogue(t)))
if err != nil {
t.Fatal(err)
}
if len(read) < 100 {
t.Fatalf("the captured catalogue holds %d manifests, fewer than the catalogue's", len(read))
}
return read
}
// meshOf is a mesh of three machines over the captured catalogue, each assigned real modules, the input the assigner
// composes with (nodes and their assignments, holdings from the claims): anchor runs the bus and the store, laptop a
// desktop's modules, ace a module whose requirement is provided elsewhere.
func meshOf() compose.Input {
return compose.Input{Nodes: []compose.Node{
{Name: "anchor", Assigned: []string{"nats", "postgres"}},
{Name: "laptop", Assigned: []string{"sudo", "zsh"}},
{Name: "ace", Assigned: []string{"mesh-reads", "nats"}},
}}
}
func request(t *testing.T, depth string) composer.Request {
t.Helper()
return composer.Request{V: composer.Version, Input: meshOf(), Depth: depth, HoldingsFromClaims: true,
Machines: []string{"anchor", "laptop", "ace"},
Cases: []composer.Case{{Name: "trunk", Manifests: captured(t)}}}
}
func runJSON(t *testing.T, args []string, in []byte) (int, composer.Verdict, string) {
t.Helper()
var out, errOut bytes.Buffer
code := run(args, bytes.NewReader(in), &out, &errOut)
var v composer.Verdict
if code == 0 {
if err := json.Unmarshal(out.Bytes(), &v); err != nil {
t.Fatalf("the verdict is not JSON: %v\n%s", err, out.String())
}
}
return code, v, errOut.String()
}
func encode(t *testing.T, r composer.Request) []byte {
t.Helper()
raw, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
return raw
}
// 1621/AC1: the command reads the plain input and the manifests on standard input, validates every manifest, composes
// the machines named and writes one verdict: per machine composes or the reason, per manifest accepted or the field
// refused.
func TestTheCommandWritesOneVerdictPerManifestAndMachine(t *testing.T) {
r := request(t, composer.DepthResolve)
r.Cases[0].Manifests["broken"] = composer.Manifest{Raw: json.RawMessage(`{"module":"broken","version":"1","shares":["x"]}`)}
code, v, stderr := runJSON(t, nil, encode(t, r))
if code != 0 {
t.Fatalf("exit %d: %s", code, stderr)
}
if len(v.Cases) != 1 || v.Cases[0].Name != "trunk" || v.SDK == "" {
t.Fatalf("verdict = %+v", v)
}
c := v.Cases[0]
if got := c.Manifests["broken"]; got.Accepted || !strings.Contains(got.Refused, `"shares"`) {
t.Errorf("the manifest with an unknown field reads %+v, want refused naming the field", got)
}
if got := c.Manifests["nats"]; !got.Accepted {
t.Errorf("nats reads %+v, want accepted", got)
}
if len(c.Machines) != 3 {
t.Fatalf("machines = %v, want the three named", c.Machines)
}
for name, m := range c.Machines {
if !m.Composes && m.Reason == "" {
t.Errorf("%s does not compose and says no reason", name)
}
}
}
// 1621/AC3: over the captured catalogue and a mesh of three machines, the command's verdict equals, machine for
// machine and manifest for manifest, what the linked reader, resolution and composition answer, at both depths.
func TestTheCommandAnswersWhatTheLinkedCompositionAnswers(t *testing.T) {
for _, depth := range []string{composer.DepthResolve, composer.DepthCompose} {
r := request(t, depth)
code, v, stderr := runJSON(t, nil, encode(t, r))
if code != 0 {
t.Fatalf("%s: exit %d: %s", depth, code, stderr)
}
in := meshOf()
in.Catalogue = map[string]catalogue.Manifest{}
for name, m := range r.Cases[0].Manifests {
// A request without a list of predefined keys reads a fallback as the indexer does (Request.Predefined).
parsed, err := catalogue.ParseManifestForInputs(m.Raw)
got := v.Cases[0].Manifests[name]
if (err == nil) != got.Accepted || (err != nil && err.Error() != got.Refused) {
t.Errorf("%s: the manifest of %s: linked %v, command %+v", depth, name, err, got)
}
if err == nil {
in.Catalogue[name] = parsed
}
}
in.Holdings = composer.HoldingsOf(in.Catalogue, in.Nodes)
composed := 0
for _, node := range r.Machines {
got := v.Cases[0].Machines[node]
var want composer.Machine
if depth == composer.DepthResolve {
if _, err := compose.Resolve(in, node); err != nil {
want = composer.Machine{Reason: err.Error()}
} else {
want = composer.Machine{Composes: true}
}
} else {
d, problems, err := compose.Compose(in, node)
switch {
case err != nil:
want = composer.Machine{Reason: err.Error()}
default:
want = composer.Machine{Composes: len(problems) == 0, Problems: problems, Refusals: d.Refusals()}
if len(problems) > 0 {
want.Reason = "the node-engine's validator refuses the declaration"
}
}
}
if !reflect.DeepEqual(got, want) {
t.Errorf("%s: %s: command %+v, linked %+v", depth, node, got, want)
}
if got.Composes {
composed++
} else {
t.Logf("%s: %s does not compose: %s %v", depth, node, got.Reason, got.Problems)
}
}
t.Logf("%s: %d of %d machines compose", depth, composed, len(r.Machines))
}
}
// 1621 note 6: a person composes a catalogue checkout (repository@ref checked out or archived: the command fetches
// nothing) for a node and reads its declaration: each path with its kind, owner and mode.
func TestACatalogueCheckoutIsComposedForANode(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "modules", "demo"), 0o755); err != nil {
t.Fatal(err)
}
manifest := `{"module": "demo", "version": "1", "resources": [{"id": "state", "type": "directory", ` +
`"path": "/var/lib/demo", "mode": "0750", "owner": "root"}]}`
if err := os.WriteFile(filepath.Join(dir, "modules", "demo", "module.json"), []byte(manifest), 0o644); err != nil {
t.Fatal(err)
}
in := encode(t, composer.Request{V: composer.Version, Input: compose.Input{Nodes: []compose.Node{{Name: "laptop", Assigned: []string{"demo"}}}}})
code, v, stderr := runJSON(t, []string{"-catalogue", dir, "-node", "laptop", "-declarations"}, in)
if code != 0 {
t.Fatalf("exit %d: %s", code, stderr)
}
m := v.Cases[0].Machines["laptop"]
want := []composer.Path{{Path: "/var/lib/demo", Kind: "directory", Owner: "root", Mode: "0750", Module: ""}}
if !m.Composes || len(m.Declared) != 1 || m.Declared[0].Path != want[0].Path || m.Declared[0].Mode != "0750" ||
m.Declared[0].Owner != "root" || m.Declared[0].Kind != "directory" {
t.Fatalf("laptop = %+v, want it composed declaring %+v", m, want)
}
}
// A run step hands the command no standard input: the request is read from the file -request names, and a file that is
// not there is a request it cannot read.
func TestTheRequestIsReadFromAFile(t *testing.T) {
file := filepath.Join(t.TempDir(), "request.json")
r := composer.Request{V: composer.Version, Cases: []composer.Case{{Name: "only", Manifests: map[string]composer.Manifest{
"demo": {Raw: json.RawMessage(`{"module":"demo","version":"1"}`)}}}}}
if err := os.WriteFile(file, encode(t, r), 0o600); err != nil {
t.Fatal(err)
}
code, v, stderr := runJSON(t, []string{"-request", file}, nil)
if code != 0 || len(v.Cases) != 1 || !v.Cases[0].Manifests["demo"].Accepted {
t.Fatalf("exit %d (%s), verdict %+v", code, stderr, v)
}
if code, _, stderr := runJSON(t, []string{"-request", file + ".gone"}, nil); code != 1 || !strings.Contains(stderr, "cannot be read") {
t.Errorf("a request file that is not there: exit %d, %q", code, stderr)
}
}
// The exit codes: 1 for a request that cannot be read or run, 2 for wrong flags, each said on standard error.
func TestTheExitCodesSayWhatFailed(t *testing.T) {
unfilled := encode(t, composer.Request{V: composer.Version, Cases: []composer.Case{{Name: "c", Manifests: map[string]composer.Manifest{"x": {Digest: "abc"}}}}})
for _, c := range []struct {
name string
args []string
in string
code int
says string
}{
{"not JSON", nil, "{", 1, "not JSON"},
{"another version", nil, `{"v":2}`, 1, "version 2"},
{"a digest left unfilled", nil, string(unfilled), 1, "digest"},
{"an unknown depth", nil, `{"v":1,"depth":"deep"}`, 1, "depth"},
{"nothing to read", nil, "", 1, "no request"},
{"a wrong flag", []string{"-nope"}, "", 2, "nope"},
{"an argument", []string{"extra"}, "", 2, "unexpected"},
} {
var out, errOut bytes.Buffer
if got := run(c.args, strings.NewReader(c.in), &out, &errOut); got != c.code || !strings.Contains(errOut.String(), c.says) {
t.Errorf("%s: exit %d saying %q, want %d saying %q", c.name, got, errOut.String(), c.code, c.says)
}
}
}
// sdkPackages is every package of this module the command imports, directly or not, with the packages each imports.
func sdkPackages(t *testing.T) map[string][]string {
t.Helper()
_, file, _, _ := runtime.Caller(0)
root := filepath.Join(filepath.Dir(file), "..", "..")
const module = "git.novox.be/novox/mesh-sdk/go"
out := map[string][]string{}
var walk func(path string)
walk = func(path string) {
if _, done := out[path]; done {
return
}
rel := strings.TrimPrefix(strings.TrimPrefix(path, module), "/")
pkg, err := build.Default.ImportDir(filepath.Join(root, rel), 0)
if err != nil {
t.Fatalf("%s: %v", path, err)
}
out[path] = pkg.Imports
for _, imp := range pkg.Imports {
if strings.HasPrefix(imp, module+"/") {
walk(imp)
}
}
}
walk(composer.Package)
return out
}
// 1621/AC2 and AC4: the command imports, through every package of mesh-sdk it reaches, only the standard library and
// mesh-sdk's own packages (so `go build` from mesh-sdk's module needs nothing beyond its lock), and none that reaches
// a bus, a store, a network service or another process.
func TestTheCommandImportsNoBusStoreNetworkOrProcess(t *testing.T) {
forbidden := []string{"net/http", "net/rpc", "net/smtp", "os/exec", "database/sql", "plugin", "syscall"}
for path, imports := range sdkPackages(t) {
for _, imp := range imports {
std := !strings.Contains(strings.SplitN(imp, "/", 2)[0], ".")
if !std && !strings.HasPrefix(imp, "git.novox.be/novox/mesh-sdk/go/") {
t.Errorf("%s imports %s, which is neither the standard library nor mesh-sdk", path, imp)
}
for _, f := range forbidden {
if imp == f {
t.Errorf("%s imports %s", path, imp)
}
}
}
}
}
// 1621/AC2: run with an empty network namespace, the command still writes its verdict: it needs no network. The test
// runs its own binary again under `unshare --net` as an unprivileged user's namespace; where the kernel or the sandbox
// allows no new namespace (a build module's run step is already one with no network), it says so and skips.
func TestTheCommandRunsWithNoNetwork(t *testing.T) {
if os.Getenv("MESH_COMPOSE_HELPER") == "1" {
os.Exit(run(nil, os.Stdin, os.Stdout, os.Stderr))
}
unshare, err := exec.LookPath("unshare")
if err != nil {
t.Skip("no unshare on this machine, so no empty network namespace can be made here")
}
if out, err := exec.Command(unshare, "--net", "--map-root-user", "true").CombinedOutput(); err != nil {
t.Skipf("this machine makes no unprivileged network namespace (%v: %s); a run step's own has no network", err, out)
}
r := request(t, composer.DepthResolve)
cmd := exec.Command(unshare, "--net", "--map-root-user", os.Args[0], "-test.run=^TestTheCommandRunsWithNoNetwork$")
cmd.Env = []string{"MESH_COMPOSE_HELPER=1"}
cmd.Stdin = bytes.NewReader(encode(t, r))
var out, errOut bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errOut
if err := cmd.Run(); err != nil {
t.Fatalf("the command in an empty network namespace: %v: %s", err, errOut.String())
}
var v composer.Verdict
if err := json.Unmarshal(out.Bytes(), &v); err != nil || len(v.Cases) != 1 || len(v.Cases[0].Machines) != 3 {
t.Fatalf("no verdict from the command with no network: %v\n%s", err, out.String())
}
names := make([]string, 0, len(v.Cases[0].Machines))
for n := range v.Cases[0].Machines {
names = append(names, n)
}
sort.Strings(names)
t.Logf("composed with no network: %v", names)
}
// -catalogue judges a fallback to a predefined key against the predefined.json at the checkout's root (ADR 0348 §7):
// xterm (mesh-catalog at 835bf22b) falls back to "font-size", accepted where the root lists it and refused where the
// root's list does not.
func TestACatalogueCheckoutJudgesFallbacksAgainstItsPredefinedKeys(t *testing.T) {
raw, err := os.ReadFile("../../catalogue/testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
for _, c := range []struct {
list string
want bool
}{
{`{"font-size": {"type": "number", "description": "the desktop's font size"}}`, true},
{`{"font-family": {"type": "string", "description": "the desktop's font"}}`, false},
} {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "modules", "xterm"), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "modules", "xterm", "module.json"), raw, 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, catalogue.PredefinedFile), []byte(c.list), 0o600); err != nil {
t.Fatal(err)
}
code, v, stderr := runJSON(t, []string{"-catalogue", dir}, nil)
if code != 0 {
t.Fatalf("exit %d: %s", code, stderr)
}
if got := v.Cases[0].Manifests["xterm"]; got.Accepted != c.want {
t.Errorf("predefined.json %s: xterm %+v, want accepted %v", c.list, got, c.want)
}
}
}
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+209
View File
@@ -0,0 +1,209 @@
// mesh-hand-compose composes one module for one node by hand, from the inputs the controller keeps, and signs it with the
// operator's key, for the hand path of novox/hq ADR 0539 clause 10: a fingerprint the controller never composed (a
// hand build), while the controller cannot compose its own fix. It composes with mesh-sdk's composition, the one the
// controller writes with (compose.Compositions), and prints the composition, its key and its digest; only with -write
// does it write them, with the operator's credential: the composition into the node's bucket composed-<node>, each
// file over 64 KiB into the node's object bucket, and the digest into mesh-controller_composed. The want is the
// operator's next act, naming that digest; each act is recorded in the hand-act log.
//
// mesh-hand-compose -input <inputs.json> -node <node> -module <module> -fp <src2:...> -key <operator key>
// [-write -nats <url> -creds <operator credential>]
//
// The inputs are a compose.Input as JSON. The key is the operator's ed25519 private key, PKCS#8 in PEM; the
// node-engine trusts it beside the controller's (ADR 0539 clause 3).
package main
import (
"context"
"crypto/ed25519"
"crypto/x509"
"encoding/json"
"encoding/pem"
"errors"
"flag"
"fmt"
"io"
"os"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func main() {
if err := run(os.Args[1:], os.Stdout); err != nil {
fmt.Fprintln(os.Stderr, "mesh-hand-compose:", err)
os.Exit(1)
}
}
// printed is what mesh-hand-compose prints: where the composition goes, its digest, the composition, and whether it was
// written.
type printed struct {
Bucket string `json:"bucket"`
Key string `json:"key"`
Digest string `json:"digest"`
Composition json.RawMessage `json:"composition"`
Objects []string `json:"objects,omitempty"`
Written bool `json:"written"`
Next string `json:"next"`
}
func run(args []string, stdout io.Writer) error {
fs := flag.NewFlagSet("mesh-hand-compose", flag.ContinueOnError)
input := fs.String("input", "", "the controller's kept inputs, a compose.Input as JSON")
node := fs.String("node", "", "the node to compose for")
module := fs.String("module", "", "the module to compose")
fp := fs.String("fp", "", "the module's fingerprint, src2: and 64 hex digits")
keyFile := fs.String("key", "", "the operator's ed25519 private key, PKCS#8 in PEM")
write := fs.Bool("write", false, "write the composition, its files and its digest, with the operator's credential")
url := fs.String("nats", nats.DefaultURL, "with -write: the bus")
creds := fs.String("creds", "", "with -write: the operator's credential file")
if err := fs.Parse(args); err != nil {
return err
}
switch {
case *input == "" || *node == "" || *module == "" || *fp == "":
return errors.New("-input, -node, -module and -fp are each required")
case *keyFile == "":
return errors.New("-key is required: a composition made by hand is signed by the operator's key, and the node-engine applies no other")
case *write && *creds == "":
return errors.New("-write needs -creds: the operator's credential is the only hand writer of a composition")
}
key, err := readKey(*keyFile)
if err != nil {
return err
}
key2, err := delivery.ComposedKey(*node, *module, *fp)
if err != nil {
return err
}
bucket, err := delivery.ComposedBucketFor(*node)
if err != nil {
return err
}
raw, err := os.ReadFile(*input)
if err != nil {
return err
}
var in compose.Input
if err := json.Unmarshal(raw, &in); err != nil {
return fmt.Errorf("the inputs in %s are not a compose.Input: %w", *input, err)
}
all, _, err := compose.Compositions(in, *node, map[string]string{*module: *fp}, compose.Reading)
if err != nil {
return err
}
c, ok := all[key2]
if !ok {
return fmt.Errorf("%s does not run %s: its composition holds none of it", *node, *module)
}
c, objects, err := c.ByDigest()
if err != nil {
return err
}
if c, err = delivery.SignComposed(c, key); err != nil {
return err
}
encoded, err := delivery.EncodeComposed(c)
if err != nil {
return err
}
digest, err := delivery.ComposedDigestOf(c)
if err != nil {
return err
}
out := printed{Bucket: bucket, Key: key2, Digest: digest, Composition: encoded,
Next: fmt.Sprintf("write want.%s.%s naming fp %s and composed %s with the operator's credential, and record each act in the hand-act log", *node, *module, *fp, digest)}
for d := range objects {
out.Objects = append(out.Objects, d)
}
if *write {
if err := writeIt(*url, *creds, *node, bucket, key2, encoded, digest, objects); err != nil {
return err
}
out.Written = true
}
enc := json.NewEncoder(stdout)
enc.SetIndent("", " ")
return enc.Encode(out)
}
func readKey(path string) (ed25519.PrivateKey, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
block, _ := pem.Decode(raw)
if block == nil {
return nil, fmt.Errorf("%s holds no PEM block", path)
}
parsed, err := x509.ParsePKCS8PrivateKey(block.Bytes)
if err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
key, ok := parsed.(ed25519.PrivateKey)
if !ok {
return nil, fmt.Errorf("%s is not an ed25519 key", path)
}
return key, nil
}
// writeIt writes the files by digest, then the composition, then its digest: a reader that finds the digest finds the
// composition and its files already there.
func writeIt(url, creds, node, bucket, key string, encoded []byte, digest string, objects map[string][]byte) error {
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()
nc, err := nats.Connect(url, nats.UserCredentials(creds), nats.Name("mesh-hand-compose"))
if err != nil {
return err
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
return err
}
if len(objects) > 0 {
filesBucket, err := declaration.FileBucketFor(node)
if err != nil {
return err
}
store, err := js.ObjectStore(ctx, filesBucket)
if err != nil {
return fmt.Errorf("the node's object bucket %s: %w", filesBucket, err)
}
for d, body := range objects {
name, err := declaration.ObjectName(d)
if err != nil {
return err
}
if _, err := store.PutBytes(ctx, name, body); err != nil {
return fmt.Errorf("the file %s: %w", d, err)
}
}
}
composed, err := js.KeyValue(ctx, bucket)
if err != nil {
return fmt.Errorf("the bucket %s: %w", bucket, err)
}
if _, err := composed.Put(ctx, key, encoded); err != nil {
return fmt.Errorf("%s in %s: %w", key, bucket, err)
}
value, err := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: digest, Judgement: delivery.JudgedCannot,
Words: "composed by hand with mesh-hand-compose (ADR 0539 clause 10): the operator writes the want"})
if err != nil {
return err
}
digests, err := js.KeyValue(ctx, delivery.ControllerComposedBucket)
if err != nil {
return fmt.Errorf("the bucket %s: %w", delivery.ControllerComposedBucket, err)
}
if _, err := digests.Put(ctx, key, value); err != nil {
return fmt.Errorf("%s in %s: %w", key, delivery.ControllerComposedBucket, err)
}
return nil
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"bytes"
"crypto/ed25519"
"crypto/rand"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"os"
"path/filepath"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
)
// aMesh is the Input of mesh-controller's aMesh, as mesh-sdk's compose tests make it (compose/fixture_test.go): two
// machines on the private network, the module the mesh provides for itself assigned to both. The manifest is the one
// the controller provides (overlay.Manifest), read back as the catalogue reads it.
func aMesh(t *testing.T) compose.Input {
t.Helper()
var m catalogue.Manifest
b, _ := json.Marshal(overlay.Manifest())
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
in := compose.Input{Catalogue: map[string]catalogue.Manifest{overlay.Name: m}}
for i, name := range []string{"anchor", "laptop"} {
pub, _, _ := ed25519.GenerateKey(rand.Reader)
in.Nodes = append(in.Nodes, compose.Node{Name: name,
Place: compose.Place{Key: base64.StdEncoding.EncodeToString(pub), Endpoint: name + ".example:51820", Site: "here", Hub: name == "anchor", Address: []string{"10.77.0.1", "10.77.0.2"}[i]},
Assigned: []string{overlay.Name}, Capabilities: map[string]bool{"container-runtime": true, "wireguard": true, "systemd": true}})
}
return in
}
func writeKey(t *testing.T, dir string) ed25519.PublicKey {
t.Helper()
pub, priv, _ := ed25519.GenerateKey(rand.Reader)
der, err := x509.MarshalPKCS8PrivateKey(priv)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "operator.key"), pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der}), 0o600); err != nil {
t.Fatal(err)
}
return pub
}
// 1943/AC5: mesh-hand-compose, given the kept inputs, a node, a module and its fingerprint, prints that module's
// composition for that node signed by the operator's key and its digest: its resources are the controller's for that
// module on that node (compose.Compose), and it verifies against the operator's key. Without --write it writes nothing.
func TestMeshComposeGivesTheControllersResourcesSignedByTheOperator(t *testing.T) {
dir := t.TempDir()
in := aMesh(t)
raw, err := json.Marshal(in)
if err != nil {
t.Fatal(err)
}
input := filepath.Join(dir, "input.json")
if err := os.WriteFile(input, raw, 0o600); err != nil {
t.Fatal(err)
}
operator := writeKey(t, dir)
fp := delivery.SourcePrefix2 + strings.Repeat("a", 64)
var out bytes.Buffer
if err := run([]string{"-input", input, "-node", "laptop", "-module", overlay.Name, "-fp", fp,
"-key", filepath.Join(dir, "operator.key")}, &out); err != nil {
t.Fatal(err)
}
var printed struct {
Key string `json:"key"`
Bucket string `json:"bucket"`
Digest string `json:"digest"`
Composition json.RawMessage `json:"composition"`
Written bool `json:"written"`
}
if err := json.Unmarshal(out.Bytes(), &printed); err != nil {
t.Fatalf("%s: %v", out.String(), err)
}
c, err := delivery.DecodeComposed(printed.Composition)
if err != nil {
t.Fatal(err)
}
if digest, err := delivery.VerifyComposed(c, []ed25519.PublicKey{operator}); err != nil || digest != printed.Digest {
t.Errorf("the composition verifies %q, %v against the operator's key; printed %s", digest, err, printed.Digest)
}
wantKey, _ := delivery.ComposedKey("laptop", overlay.Name, fp)
if printed.Key != wantKey || printed.Bucket != "composed-laptop" || printed.Written {
t.Errorf("printed key %s in %s, written %v", printed.Key, printed.Bucket, printed.Written)
}
d, _, err := compose.Compose(in, "laptop")
if err != nil {
t.Fatal(err)
}
var want, got []string
for _, r := range d.Resources {
want = append(want, r["id"].(string))
}
for _, r := range c.Resources {
var head struct{ ID string }
_ = json.Unmarshal(r, &head)
got = append(got, head.ID)
}
sort.Strings(want)
sort.Strings(got)
if len(got) == 0 || strings.Join(got, " ") != strings.Join(want, " ") {
t.Errorf("mesh-hand-compose gives %v, the controller's composition %v", got, want)
}
}
// 1943/AC5: mesh-hand-compose refuses what it cannot compose, naming why: no key, a fingerprint not of the new form, a
// module the node does not run, a node the mesh does not have.
func TestMeshComposeRefuses(t *testing.T) {
dir := t.TempDir()
raw, _ := json.Marshal(aMesh(t))
input := filepath.Join(dir, "input.json")
_ = os.WriteFile(input, raw, 0o600)
writeKey(t, dir)
key := filepath.Join(dir, "operator.key")
fp := delivery.SourcePrefix2 + strings.Repeat("a", 64)
for _, c := range []struct {
args []string
why string
}{
{[]string{"-input", input, "-node", "laptop", "-module", overlay.Name, "-fp", fp}, "-key"},
{[]string{"-input", input, "-node", "laptop", "-module", overlay.Name, "-fp", "src2:1", "-key", key}, "fingerprint"},
{[]string{"-input", input, "-node", "laptop", "-module", "mesh-mail", "-fp", fp, "-key", key}, "does not run mesh-mail"},
{[]string{"-input", input, "-node", "nowhere", "-module", overlay.Name, "-fp", fp, "-key", key}, "nowhere"},
{[]string{"-input", input, "-node", "laptop", "-module", overlay.Name, "-fp", fp, "-key", key, "-write"}, "-creds"},
} {
var out bytes.Buffer
if err := run(c.args, &out); err == nil || !strings.Contains(err.Error(), c.why) {
t.Errorf("mesh-hand-compose %v = %v, want a refusal saying %q", c.args, err, c.why)
}
}
}
+94
View File
@@ -0,0 +1,94 @@
package compose
import (
"encoding/json"
"fmt"
"sort"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Compositions is one node's compositions (novox/hq ADR 0539 clauses 1 and 2): its declaration composed over in, as
// the controller composes it, split into each module's composition at the fingerprint fps gives it and the node's own
// (`<node>.node`), by key (delivery.ComposedKey). Each resource says its module, whatever the node reported it reads.
// A module fps gives no fingerprint for is not composed and is named in skipped, so a caller composing one module
// (mesh-hand-compose) gets that one. Each module's Data says whether its manifest declares data. The compositions are not
// signed and not yet by digest: the writer does both (delivery.SignComposed, Composed.ByDigest).
func Compositions(in Input, node string, fps map[string]string, mode Mode) (out map[string]delivery.Composed, skipped []string, err error) {
reading := in
reading.Nodes = append([]Node(nil), in.Nodes...)
found := false
for i := range reading.Nodes {
if reading.Nodes[i].Name == node {
reading.Nodes[i].ReadsModules = true
found = true
}
}
if !found {
return nil, nil, fmt.Errorf("compose: the mesh has no node %q", node)
}
d, problems, err := ComposeIn(reading, node, mode)
if err != nil {
return nil, nil, err
}
if len(problems) > 0 {
return nil, nil, fmt.Errorf("compose: %s's declaration is refused by the node-engine's validator: %v", node, problems)
}
body, err := d.Body()
if err != nil {
return nil, nil, err
}
var envelope struct {
Resources []json.RawMessage `json:"resources"`
Adoption *declaration.Adoption `json:"adoption"`
Filled []string `json:"filled_files"`
Provisioner string `json:"provisioner_account"`
}
if err := json.Unmarshal(body, &envelope); err != nil {
return nil, nil, err
}
var kept []json.RawMessage
keptPaths := map[string]bool{}
left := map[string]bool{}
for _, raw := range envelope.Resources {
var head struct {
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
}
if err := json.Unmarshal(raw, &head); err != nil {
return nil, nil, err
}
if _, ok := fps[head.Module]; head.Module != "" && !ok {
left[head.Module] = true
continue
}
if head.Type == string(declaration.TypeFile) {
keptPaths[head.Path] = true
}
kept = append(kept, raw)
}
var filled []string
for _, p := range envelope.Filled {
if keptPaths[p] {
filled = append(filled, p)
}
}
out, err = delivery.SplitComposed(node, kept, fps, filled,
delivery.NodeOwn{Adoption: envelope.Adoption, Provisioner: envelope.Provisioner})
if err != nil {
return nil, nil, err
}
for key, c := range out {
if m, ok := in.Catalogue[c.Module]; ok && c.Module != "" {
c.Data = len(m.DataItems()) > 0
out[key] = c
}
}
for m := range left {
skipped = append(skipped, m)
}
sort.Strings(skipped)
return out, skipped, nil
}
+88
View File
@@ -0,0 +1,88 @@
package compose
import (
"encoding/json"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func fpFor(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
// 1943/AC5: a node's compositions are its declaration as Compose gives it, split by the module each resource belongs
// to: every resource once, each in its module's composition at the fingerprint given, the module's ownership said on
// it; a module given no fingerprint is named as skipped and composed nowhere.
func TestCompositionsAreTheDeclarationSplitByModule(t *testing.T) {
in := aSmallMesh(t)
d, _, err := Compose(in, "laptop")
if err != nil {
t.Fatal(err)
}
var want []string
for _, r := range d.Resources {
want = append(want, r["type"].(string)+":"+r["id"].(string))
}
sort.Strings(want)
fps := map[string]string{"mesh-wireguard": fpFor('a'), "files": fpFor('b')}
out, skipped, err := Compositions(in, "laptop", fps, Foreseeing)
if err != nil || len(skipped) != 0 {
t.Fatalf("Compositions = %v, skipped %v", err, skipped)
}
var got []string
for key, c := range out {
node, module, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || node != "laptop" || module != c.Module || fp != c.FP || (isNode != (module == "")) {
t.Errorf("key %s holds %s at %s (%v)", key, c.Module, c.FP, err)
}
if !isNode && fp != fps[module] {
t.Errorf("%s is composed at %s, given %s", module, fp, fps[module])
}
if _, err := delivery.EncodeComposed(c); err != nil {
t.Errorf("%s: %v", key, err)
}
for _, raw := range c.Resources {
var r struct{ ID, Type, Module string }
_ = json.Unmarshal(raw, &r)
if r.Module != module {
t.Errorf("%s holds %s of %q", key, r.ID, r.Module)
}
got = append(got, r.Type+":"+r.ID)
}
}
sort.Strings(got)
if strings.Join(got, " ") != strings.Join(want, " ") {
t.Errorf("the compositions hold %v, the declaration %v", got, want)
}
one, skipped, err := Compositions(in, "laptop", map[string]string{"files": fpFor('b')}, Foreseeing)
if err != nil || strings.Join(skipped, ",") != "mesh-wireguard" {
t.Fatalf("one module: %v, skipped %v", err, skipped)
}
for _, c := range one {
if c.Module == "mesh-wireguard" {
t.Errorf("a module given no fingerprint was composed")
}
}
if _, _, err := Compositions(in, "nowhere", fps, Foreseeing); err == nil {
t.Errorf("a node the mesh does not have was composed")
}
}
// 1943/AC1: the upgrade policies and sources a composition's value may name are the catalogue's own words.
func TestTheComposedPoliciesAreTheCatalogues(t *testing.T) {
for _, p := range []string{catalogue.PolicyRoll, catalogue.PolicyTogether, catalogue.PolicyRecord} {
for _, from := range []string{catalogue.FromPerson, catalogue.FromModule, catalogue.FromBus, catalogue.FromData, catalogue.FromDefault} {
v := delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("a", 64), Judgement: delivery.JudgedNeither, Policy: p, PolicyFrom: from}
if _, err := delivery.EncodeComposedDigest(v); err != nil {
t.Errorf("%s from %s: %v", p, from, err)
}
}
}
if delivery.PolicyRecord != catalogue.PolicyRecord {
t.Errorf("delivery.PolicyRecord = %q, the catalogue's %q", delivery.PolicyRecord, catalogue.PolicyRecord)
}
}
+347
View File
@@ -0,0 +1,347 @@
// Package composer is mesh-sdk's composition and validator as one function over a plain request (novox/hq ADR 0482
// clause 4, issue 1621): what the command mesh-compose runs, so a program that links any one mesh-sdk can have the
// composition of another commit judged by building that commit's command and reading its verdict.
//
// **It reads its request and nothing else.** No bus, store, network, file or environment: the manifests come inside the
// request (raw, as their repository holds them), the rest of the mesh is a compose.Input, and the answer is a value.
// A manifest named only by its digest in the delivery indexer's bucket is filled before the run by Fill, by whoever
// holds the bus (the Go build module's compose job); Run refuses a digest left unfilled.
package composer
import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/sdkversion"
)
// Version is the version of a Request and of a Verdict; a reader refuses another.
const Version = 1
// Package is the import path of the command, which a compose job builds at a commit (in mesh-sdk's own module, or in
// a module that requires mesh-sdk, at the version it pins).
const Package = "git.novox.be/novox/mesh-sdk/go/cmd/mesh-compose"
// The depths of a composition.
const (
// DepthResolve resolves each machine's set over the rest of the mesh (compose.Resolve): what the assigner judges
// while the facts snapshot is not on the bus (issue 1461).
DepthResolve = "resolve"
// DepthCompose composes each machine's whole declaration and runs the node-engine's validator over it
// (compose.Compose).
DepthCompose = "compose"
)
// Request is everything one run composes: the rest of the mesh, one or more cases of manifests composed over it, and
// the machines composed in each case.
type Request struct {
V int `json:"v"`
// Input is the composition's input without its catalogue (and without each node's kept manifests): the cases give
// those. A Catalogue or Kept it holds is ignored.
Input compose.Input `json:"input"`
// Cases are composed one after the other over Input, each with its own manifests; their names are unique.
Cases []Case `json:"cases"`
// Machines are the nodes composed in every case; none means the manifests are only read.
Machines []string `json:"machines,omitempty"`
// Depth is DepthResolve or DepthCompose; empty is DepthCompose.
Depth string `json:"depth,omitempty"`
// HoldingsFromClaims adds to Input's holdings every seat an assigned module claims, held on its node (the
// assigner's input, which has no holdings of record).
HoldingsFromClaims bool `json:"holdings_from_claims,omitempty"`
// Declarations asks each composed machine's declared paths (DepthCompose only).
Declarations bool `json:"declarations,omitempty"`
// Predefined is the list of predefined keys at the catalogue's root (predefined.json, novox/hq ADR 0348 §7) that
// every config value falling back to a key is judged against, when the asker has it (mesh-compose -catalogue
// reads it). Without it a fallback is read as catalogue.ParseManifestForInputs reads it: against the list the mesh
// published when this process holds one, else each key as its value declares it, for the merge gate judges the
// fallback in the catalogue and a manifest that only carries one is not refused here.
Predefined catalogue.PredefinedList `json:"predefined,omitempty"`
}
// Case is one catalogue composed: every module's manifest by name, and per node the manifests kept at the build it runs.
type Case struct {
Name string `json:"name"`
Manifests map[string]Manifest `json:"manifests"`
Kept map[string]map[string]Manifest `json:"kept,omitempty"`
}
// Manifest is a module's manifest as its repository holds it (Raw, the bytes), or the digest the delivery indexer keeps
// it under (Digest), which Fill replaces by the bytes. On the wire it is the manifest itself, or {"digest": "..."}.
type Manifest struct {
Raw json.RawMessage
Digest string
}
// MarshalJSON writes the raw manifest, or {"digest": ...} when only the digest is known.
func (m Manifest) MarshalJSON() ([]byte, error) {
if len(m.Raw) > 0 {
return m.Raw, nil
}
return json.Marshal(map[string]string{"digest": m.Digest})
}
// UnmarshalJSON reads {"digest": "..."} (an object with that one key) as a digest, and any other value as the raw
// manifest.
func (m *Manifest) UnmarshalJSON(raw []byte) error {
var probe map[string]json.RawMessage
if json.Unmarshal(raw, &probe) == nil && len(probe) == 1 {
if d, ok := probe["digest"]; ok {
var digest string
if err := json.Unmarshal(d, &digest); err != nil {
return fmt.Errorf("a manifest's digest is not a string: %w", err)
}
*m = Manifest{Digest: digest}
return nil
}
}
*m = Manifest{Raw: slices.Clone(raw)}
return nil
}
// Verdict is what one run answers: the mesh-sdk the command was built with, and each case's verdict in the order asked.
type Verdict struct {
V int `json:"v"`
SDK string `json:"sdk"`
Cases []CaseVerdict `json:"cases"`
}
// CaseVerdict is one case's: every manifest read or refused, every machine composed or not.
type CaseVerdict struct {
Name string `json:"name"`
Manifests map[string]ManifestRead `json:"manifests"`
Machines map[string]Machine `json:"machines,omitempty"`
}
// ManifestRead is one manifest as this mesh-sdk reads it: accepted, or refused with the reader's words.
type ManifestRead struct {
Accepted bool `json:"accepted"`
Refused string `json:"refused,omitempty"`
}
// Machine is one machine composed: whether it composes (resolves, at DepthResolve), why not, the node-engine
// validator's problems and the composition's refusals (DepthCompose), and its declared paths when asked.
type Machine struct {
Composes bool `json:"composes"`
Reason string `json:"reason,omitempty"`
Problems []string `json:"problems,omitempty"`
Refusals []string `json:"refusals,omitempty"`
Declared []Path `json:"declared,omitempty"`
}
// Path is one path a declaration puts on the machine, with its kind (the resource's type), owner, group and mode as
// declared, and the module it is of.
type Path struct {
Path string `json:"path"`
Kind string `json:"kind"`
Owner string `json:"owner,omitempty"`
Group string `json:"group,omitempty"`
Mode string `json:"mode,omitempty"`
Module string `json:"module,omitempty"`
}
// ErrUnfilled is a manifest named by a digest that nobody replaced by its bytes.
var ErrUnfilled = errors.New("composer: a manifest is named by its digest and was not filled")
// Decode reads a request; it refuses one that is not JSON or of another version.
func Decode(data []byte) (Request, error) {
var r Request
if err := json.Unmarshal(data, &r); err != nil {
return Request{}, fmt.Errorf("composer: the request is not JSON of its shape: %w", err)
}
if r.V != Version {
return Request{}, fmt.Errorf("composer: a request of version %d, this composer reads %d", r.V, Version)
}
return r, nil
}
// Fill replaces every manifest named by its digest by the bytes get answers for that digest; a digest get cannot answer
// stops it, named.
func Fill(r *Request, get func(digest string) ([]byte, error)) error {
fill := func(m map[string]Manifest) error {
for name, man := range m {
if len(man.Raw) > 0 {
continue
}
raw, err := get(man.Digest)
if err != nil {
return fmt.Errorf("composer: the manifest of %s (digest %s): %w", name, man.Digest, err)
}
m[name] = Manifest{Raw: raw}
}
return nil
}
for _, c := range r.Cases {
if err := fill(c.Manifests); err != nil {
return err
}
for _, kept := range c.Kept {
if err := fill(kept); err != nil {
return err
}
}
}
return nil
}
// Run composes every case of r and answers the verdict, its SDK this binary's mesh-sdk. An error is a request it
// cannot run (an unknown depth, a case named twice, a manifest left as a digest); a machine that does not compose is a
// verdict.
func Run(r Request) (Verdict, error) {
depth := r.Depth
if depth == "" {
depth = DepthCompose
}
if depth != DepthResolve && depth != DepthCompose {
return Verdict{}, fmt.Errorf("composer: the depth %q is not resolve or compose", r.Depth)
}
v := Verdict{V: Version, SDK: sdkversion.Version()}
seen := map[string]bool{}
for _, c := range r.Cases {
if seen[c.Name] {
return Verdict{}, fmt.Errorf("composer: the case %q is named twice", c.Name)
}
seen[c.Name] = true
cv, err := runCase(r, c, depth)
if err != nil {
return Verdict{}, err
}
v.Cases = append(v.Cases, cv)
}
return v, nil
}
// readManifest reads one manifest as this run reads every one: against the request's list of predefined keys when it
// has one, else as the indexer and the builder read it (Request.Predefined).
func readManifest(raw []byte, predefined catalogue.PredefinedList) (catalogue.Manifest, error) {
if predefined != nil {
return catalogue.ParseManifestWith(raw, predefined)
}
return catalogue.ParseManifestForInputs(raw)
}
func parse(m map[string]Manifest, reads map[string]ManifestRead, predefined catalogue.PredefinedList) (map[string]catalogue.Manifest, error) {
out := map[string]catalogue.Manifest{}
for name, man := range m {
if len(man.Raw) == 0 {
return nil, fmt.Errorf("%w: %s (digest %s)", ErrUnfilled, name, man.Digest)
}
parsed, err := readManifest(man.Raw, predefined)
if reads != nil {
if err != nil {
reads[name] = ManifestRead{Refused: err.Error()}
} else {
reads[name] = ManifestRead{Accepted: true}
}
}
if err == nil {
out[name] = parsed
}
}
return out, nil
}
func runCase(r Request, c Case, depth string) (CaseVerdict, error) {
cv := CaseVerdict{Name: c.Name, Manifests: map[string]ManifestRead{}}
shelf, err := parse(c.Manifests, cv.Manifests, r.Predefined)
if err != nil {
return CaseVerdict{}, err
}
in := r.Input
in.Catalogue = shelf
in.Nodes = slices.Clone(in.Nodes)
in.Holdings = slices.Clone(in.Holdings)
for i := range in.Nodes {
in.Nodes[i].Kept = nil
if kept := c.Kept[in.Nodes[i].Name]; len(kept) > 0 {
k, err := parse(kept, nil, r.Predefined)
if err != nil {
return CaseVerdict{}, err
}
in.Nodes[i].Kept = k
}
}
if r.HoldingsFromClaims {
in.Holdings = append(in.Holdings, HoldingsOf(shelf, in.Nodes)...)
}
if len(r.Machines) == 0 {
return cv, nil
}
cv.Machines = map[string]Machine{}
for _, node := range r.Machines {
cv.Machines[node] = machine(in, node, depth, r.Declarations)
}
return cv, nil
}
// HoldingsOf is every seat an assigned module claims, held on the node it is assigned to, in node order: the holdings
// of an input that has none of record (the assigner's, mesh-delivery-assigner internal/assign input).
func HoldingsOf(shelf map[string]catalogue.Manifest, nodes []compose.Node) []catalogue.Held {
var out []catalogue.Held
sorted := slices.Clone(nodes)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Name < sorted[j].Name })
for _, n := range sorted {
for _, mod := range n.Assigned {
for _, cl := range shelf[mod].Claims {
scope := cl.Scope
if seat, ok := catalogue.SeatNamed(cl.Name); ok && scope == "" {
scope = seat.Scope
}
out = append(out, catalogue.Held{Claim: cl.Name, Scope: scope, Node: n.Name, Module: mod})
}
}
}
return out
}
func machine(in compose.Input, node, depth string, declarations bool) Machine {
if depth == DepthResolve {
if _, err := compose.Resolve(in, node); err != nil {
return Machine{Reason: err.Error()}
}
return Machine{Composes: true}
}
d, problems, err := compose.Compose(in, node)
if err != nil {
return Machine{Reason: err.Error()}
}
m := Machine{Composes: len(problems) == 0, Problems: problems, Refusals: d.Refusals()}
if len(problems) > 0 {
m.Reason = "the node-engine's validator refuses the declaration"
}
if declarations {
m.Declared = pathsOf(d)
}
return m
}
// pathsOf is every resource of a declaration that names a path, with its owner, group and mode as declared, sorted by
// path.
func pathsOf(d compose.Declared) []Path {
var out []Path
str := func(r map[string]any, key string) string {
if v, ok := r[key]; ok && v != nil {
return fmt.Sprint(v)
}
return ""
}
for _, r := range d.Resources {
p := str(r, "path")
if p == "" {
continue
}
out = append(out, Path{Path: p, Kind: str(r, "type"), Owner: str(r, "owner"), Group: str(r, "group"),
Mode: str(r, "mode"), Module: d.ModuleOf[str(r, "id")]})
}
sort.Slice(out, func(i, j int) bool {
if out[i].Path != out[j].Path {
return out[i].Path < out[j].Path
}
return out[i].Kind < out[j].Kind
})
return out
}
+137
View File
@@ -0,0 +1,137 @@
package composer
import (
"encoding/json"
"errors"
"os"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
)
const demo = `{"module": "demo", "version": "1", "claims": [{"name": "node-files"}]}`
// A manifest named by its digest goes on the wire as {"digest": ...}, and Fill puts the bytes the indexer keeps in its
// place; a raw manifest goes as itself.
func TestAManifestIsItsBytesOrItsDigest(t *testing.T) {
r := Request{V: Version, Cases: []Case{{Name: "change", Manifests: map[string]Manifest{
"demo": {Digest: "d1"}, "raw": {Raw: json.RawMessage(`{"module":"raw","version":"1"}`)}},
Kept: map[string]map[string]Manifest{"laptop": {"demo": {Digest: "d1"}}}}}}
wire, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(wire), `"demo":{"digest":"d1"}`) || !strings.Contains(string(wire), `"raw":{"module":"raw","version":"1"}`) {
t.Fatalf("on the wire: %s", wire)
}
back, err := Decode(wire)
if err != nil {
t.Fatal(err)
}
if _, err := Run(back); !errors.Is(err, ErrUnfilled) {
t.Fatalf("a digest left unfilled runs: %v", err)
}
asked := 0
if err := Fill(&back, func(d string) ([]byte, error) {
asked++
if d != "d1" {
return nil, errors.New("no such digest")
}
return []byte(demo), nil
}); err != nil {
t.Fatal(err)
}
if asked != 2 || string(back.Cases[0].Manifests["demo"].Raw) != demo || string(back.Cases[0].Kept["laptop"]["demo"].Raw) != demo {
t.Fatalf("filled %d times: %+v", asked, back.Cases[0])
}
bad := Request{V: Version, Cases: []Case{{Name: "c", Manifests: map[string]Manifest{"x": {Digest: "nope"}}}}}
if err := Fill(&bad, func(string) ([]byte, error) { return nil, errors.New("not there") }); err == nil || !strings.Contains(err.Error(), "nope") {
t.Fatalf("a digest the indexer does not hold: %v", err)
}
}
// The input travels as JSON and comes back equal, so the command composes what its caller meant.
func TestTheInputTravelsAsJSON(t *testing.T) {
in := compose.Input{
Nodes: []compose.Node{{Name: "anchor", Assigned: []string{"nats"}, Capabilities: map[string]bool{"systemd": true},
Place: compose.Place{Endpoint: "anchor.example:51820", Site: "here", Hub: true, Address: "10.77.0.1"},
Pins: map[string]catalogue.Chosen{"bus": {Node: "anchor", Module: "nats"}},
Settings: map[string][]catalogue.Layer{"nats": {{Values: map[string]any{"port": float64(4222)}}}}}},
Holdings: []catalogue.Held{{Claim: "mesh-broker", Scope: "mesh", Node: "anchor", Module: "nats"}},
Issued: []compose.Issued{{Provision: "bus", Consumer: "laptop", ConsumerModule: "x", Provider: "anchor"}},
}
raw, err := json.Marshal(Request{V: Version, Input: in})
if err != nil {
t.Fatal(err)
}
back, err := Decode(raw)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(back.Input, in) {
t.Fatalf("the input came back as\n%+v\nwant\n%+v", back.Input, in)
}
}
// The seats a node's assigned modules claim are held there when the request says the holdings come from the claims.
func TestHoldingsComeFromTheClaims(t *testing.T) {
m, err := catalogue.ParseManifest([]byte(demo))
if err != nil {
t.Fatal(err)
}
got := HoldingsOf(map[string]catalogue.Manifest{"demo": m}, []compose.Node{{Name: "b", Assigned: []string{"demo"}}, {Name: "a", Assigned: []string{"demo"}}})
if len(got) != 2 || got[0].Node != "a" || got[0].Claim != "node-files" || got[0].Scope != "node" {
t.Fatalf("holdings = %+v", got)
}
}
// A request the command cannot run is refused, never answered as a verdict: another version, a case named twice, an
// unknown depth.
func TestARequestItCannotRunIsRefused(t *testing.T) {
if _, err := Decode([]byte(`{"v": 2}`)); err == nil {
t.Error("a request of version 2 is read")
}
if _, err := Run(Request{V: Version, Cases: []Case{{Name: "a"}, {Name: "a"}}}); err == nil {
t.Error("a case named twice runs")
}
if _, err := Run(Request{V: Version, Depth: "deep"}); err == nil {
t.Error("an unknown depth runs")
}
v, err := Run(Request{V: Version, Cases: []Case{{Name: "only-read", Manifests: map[string]Manifest{"demo": {Raw: json.RawMessage(demo)}}}}})
if err != nil || len(v.Cases) != 1 || !v.Cases[0].Manifests["demo"].Accepted || v.Cases[0].Machines != nil || v.SDK == "" {
t.Fatalf("a request with no machines: %+v, %v", v, err)
}
}
// A manifest whose config value falls back to a predefined key (novox/hq ADR 0348 §7) is read, not refused for the
// list of predefined keys at the catalogue's root that a run is never handed with its manifests: the merge gate judges
// the fallback in the catalogue. xterm's module.json from mesh-catalog at 835bf22b falls back to "font-size"; a group
// touching xterm or pacman failed its composition on this (group check 18df81380db3). A list in the request is the one
// judged against, so a fallback it does not name is still refused.
func TestAManifestFallingBackToAPredefinedKeyIsRead(t *testing.T) {
raw, err := os.ReadFile("../catalogue/testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), `"predefined"`) {
t.Fatal("the fixture names no predefined key")
}
r := Request{V: Version, Cases: []Case{{Name: "change", Manifests: map[string]Manifest{"xterm": {Raw: raw}}}}}
v, err := Run(r)
if err != nil {
t.Fatal(err)
}
if got := v.Cases[0].Manifests["xterm"]; !got.Accepted {
t.Fatalf("xterm: %+v, want accepted", got)
}
r.Predefined = catalogue.PredefinedList{"something-else": {Type: "string", Description: "x"}}
if v, err = Run(r); err != nil {
t.Fatal(err)
}
if got := v.Cases[0].Manifests["xterm"]; got.Accepted || !strings.Contains(got.Refused, "not on the list of predefined keys") {
t.Fatalf("xterm judged against a list without its key: %+v, want refused naming the list", got)
}
}
+47
View File
@@ -0,0 +1,47 @@
package declaration
import (
"encoding/json"
"fmt"
)
// ParseResource reads one resource as Parse reads each of a declaration's resources: by its type, strictly, refusing
// a field its kind does not use and anything its kind refuses, and never an action (the link may not carry one, ADR
// 0005). It judges the resource alone: what Parse judges across resources (an opening on a node not adopted, the
// files the provisioner fills, what replaces what) is for the whole a node applies (novox/hq ADR 0539 clause 1: a
// composition is one module's resources, and a node applies its compositions together with its own key).
func ParseResource(raw json.RawMessage) (Resource, error) {
var head struct {
ID string `json:"id"`
Type Type `json:"type"`
}
if err := json.Unmarshal(raw, &head); err != nil {
return nil, &RefusalError{Problems: []string{"not a resource: " + err.Error()}}
}
where := "resource"
if head.ID == "" {
return nil, &RefusalError{Problems: []string{where + ": no id. Identity is what lets the host know this is " +
"the same resource it applied last time"}}
}
where = fmt.Sprintf("resource %q", head.ID)
resource := newOf(head.Type)
if resource == nil {
return nil, &RefusalError{Problems: []string{fmt.Sprintf("%s: unknown type %q. This host understands %s",
where, head.Type, vocabulary())}}
}
if unknown := unknownFields(raw, resource); len(unknown) > 0 {
var problems []string
for _, field := range unknown {
problems = append(problems, fmt.Sprintf("%s: a %s does not use %q, and it is set. Refused rather "+
"than ignored", where, head.Type, field))
}
return nil, &RefusalError{Problems: problems}
}
if err := json.Unmarshal(raw, resource); err != nil {
return nil, &RefusalError{Problems: []string{fmt.Sprintf("%s: %s", where, err)}}
}
if problems := resource.validate(where, false); len(problems) > 0 {
return nil, &RefusalError{Problems: problems}
}
return resource, nil
}
+108 -16
View File
@@ -1,6 +1,7 @@
package delivery
import (
"bytes"
"encoding/json"
"fmt"
"reflect"
@@ -16,6 +17,10 @@ import (
// only compared; the node agent applying them is release 6 (ADR 0429 §7). Only the assigner's credential writes these
// buckets; the watchman, the view and the controller read them.
//
// **Release 6** (ADR 0539 clauses 1, 6 and 16): a want also names the digest of the one composition it allows, written
// only once that composition exists (the seventh rule, RuleComposed), and carries the answer to a failed backup; the
// node's own want `want.<node>.node` names the digest of the node's own composition (NodeWant).
//
// **The bucket is the first word, the key the rest** (as the builder's `builds.<module>.<fp>` is the key
// `<module>.<fp>` in its builds bucket): the differences bucket's keys are `<node>.<module>`. The wants bucket holds
// two kinds of key, so each keeps its word: `want.<node>.<module>` and `held.<module>.<fingerprint hex>`.
@@ -50,7 +55,33 @@ func WantKey(node, module string) (string, error) {
return wantWord + "." + have, nil
}
// ParseWantKey is the inverse of WantKey; it refuses a held key, a have key and any key of another bucket, naming it.
// WantNodeKey is the key, in AssignerWantsBucket, of a node's own want: `want.<node>.node`, naming the digest of the
// node's own composition (ADR 0539 clause 6; ADR 0429 §4's node key). The assigner writes it once the node's own
// composition exists.
func WantNodeKey(node string) (string, error) {
own, err := HaveNodeKey(node)
if err != nil {
return "", err
}
return wantWord + "." + own, nil
}
// ParseWantNodeKey is the inverse of WantNodeKey; it refuses a module's want key and any key of another bucket,
// naming it.
func ParseWantNodeKey(key string) (node string, err error) {
rest, ok := strings.CutPrefix(key, wantWord+".")
if !ok {
return "", fmt.Errorf("delivery: %q is not a node's own want key, want.<node>.node", key)
}
node, _, isNode, err := ParseHaveKey(rest)
if err != nil || !isNode {
return "", fmt.Errorf("delivery: %q is not a node's own want key, want.<node>.node", key)
}
return node, nil
}
// ParseWantKey is the inverse of WantKey; it refuses a held key, a have key, the node's own want key and any key of
// another bucket, naming it.
func ParseWantKey(key string) (node, module string, err error) {
rest, ok := strings.CutPrefix(key, wantWord+".")
if !ok {
@@ -100,7 +131,8 @@ func ParseDifferenceKey(key string) (node, module string, err error) {
return node, module, nil
}
// The six rules of ADR 0429 §6, by number, as a want or a difference names the one that holds a node back.
// The rules of ADR 0429 §6 and ADR 0539 clause 6, by number, as a want or a difference names the one that holds a node
// back.
const (
RuleBuildDone = 1 // the build of F is done
RuleNotHeld = 2 // F is not held
@@ -108,15 +140,25 @@ const (
RuleStep = 4 // at most step other nodes are ahead (step 0: no limit, ADR 0528 §3)
RuleProviders = 5 // every provider the module requires runs healthy on that node, or at its canary
RuleGrant = 6 // the module's credential for F exists (grants.<module>.<fp>)
RuleComposed = 7 // the module's composition for that node at F exists (mesh-controller_composed, ADR 0539 §6)
)
func checkRule(rule int, of string) error {
if rule < RuleBuildDone || rule > RuleGrant {
return fmt.Errorf("delivery: %s names the rule %d, which is not one of the six (1 to 6)", of, rule)
if rule < RuleBuildDone || rule > RuleComposed {
return fmt.Errorf("delivery: %s names the rule %d, which is not one of the seven (1 to 7)", of, rule)
}
return nil
}
// The answers a want carries to a failed backup before an apply (ADR 0539 clause 16): the node agent reads them from
// its own want and from nothing else.
const (
// BackupAgain is the answer to back the module up again before the apply.
BackupAgain = "again"
// BackupSkip is the answer to go on without a backup.
BackupSkip = "skip"
)
func checkFingerprint(fp, of string) error {
if _, err := FingerprintKeyPart(fp); err != nil {
return fmt.Errorf("delivery: %s: %w", of, err)
@@ -124,29 +166,44 @@ func checkFingerprint(fp, of string) error {
return nil
}
// Want is a node's want for a module: the fingerprint FP the node should run, or Removed for a module that left the
// node's modules (then FP is empty). When the module's newest fingerprint is not yet allowed on this node, Newest names
// it and Rule the rule that holds the node back; both are empty when FP is the newest. Words say why in a sentence.
// The entry's own time says since when.
// Want is a node's want for a module: the fingerprint FP the node should run and the digest Composed of the one
// composition of it the node may apply (ADR 0539 clause 1), or Removed for a module that left the node's modules (then
// both are empty). When the module's newest fingerprint is not yet allowed on this node, Newest names it and Rule the
// rule that holds the node back; both are empty when FP is the newest. Backup is the answer to a failed backup before
// the apply, BackupAgain or BackupSkip (clause 16), empty when none was asked. Words say why in a sentence. The
// entry's own time says since when.
type Want struct {
V int `json:"v"`
FP string `json:"fp,omitempty"`
Removed bool `json:"removed,omitempty"`
Newest string `json:"newest,omitempty"`
Rule int `json:"rule,omitempty"`
Words string `json:"words,omitempty"`
V int `json:"v"`
FP string `json:"fp,omitempty"`
Composed string `json:"composed,omitempty"`
Removed bool `json:"removed,omitempty"`
Newest string `json:"newest,omitempty"`
Rule int `json:"rule,omitempty"`
Backup string `json:"backup,omitempty"`
Words string `json:"words,omitempty"`
}
func (w Want) check() error {
if w.Removed {
if w.FP != "" || w.Newest != "" || w.Rule != 0 {
return fmt.Errorf("delivery: a removed want names no fingerprint and no rule")
if w.FP != "" || w.Composed != "" || w.Newest != "" || w.Rule != 0 || w.Backup != "" {
return fmt.Errorf("delivery: a removed want names no fingerprint, composition, rule or backup answer")
}
return nil
}
if err := checkFingerprint(w.FP, "a want's fingerprint"); err != nil {
return err
}
if w.Composed == "" {
return fmt.Errorf("delivery: the want of %s names no composition: a want names the digest of the one composition it allows (ADR 0539 clause 1)", w.FP)
}
if err := checkDigest(w.Composed, "a want's composition digest"); err != nil {
return err
}
switch w.Backup {
case "", BackupAgain, BackupSkip:
default:
return fmt.Errorf("delivery: the backup answer %q is not %s or %s", w.Backup, BackupAgain, BackupSkip)
}
if w.Newest == "" && w.Rule == 0 {
return nil
}
@@ -236,6 +293,41 @@ func DecodeWant(data []byte) (Want, error) {
// SameWant says whether two wants are the same value, so the assigner writes only when they differ.
func SameWant(a, b Want) bool { return a == b }
// NodeWant is a node's own want (WantNodeKey): the digest Composed of the node's own composition it may apply. Words
// say why in a sentence.
type NodeWant struct {
V int `json:"v"`
Composed string `json:"composed"`
Words string `json:"words,omitempty"`
}
// EncodeNodeWant is w as JSON at Version; it refuses a digest not of the sha256 form.
func EncodeNodeWant(w NodeWant) ([]byte, error) {
if err := checkDigest(w.Composed, "the node's own want's composition digest"); err != nil {
return nil, err
}
w.V = Version
return json.Marshal(w)
}
// DecodeNodeWant reads a node's own want; it refuses an unknown version (ErrUnknownVersion), a field a node's own want
// does not have (a fingerprint among them), and what EncodeNodeWant refuses.
func DecodeNodeWant(data []byte) (NodeWant, error) {
if err := checkVersion(data); err != nil {
return NodeWant{}, err
}
var w NodeWant
d := json.NewDecoder(bytes.NewReader(data))
d.DisallowUnknownFields()
if err := d.Decode(&w); err != nil {
return NodeWant{}, fmt.Errorf("delivery: a node's own want does not fit its shape: %w", err)
}
if err := checkDigest(w.Composed, "the node's own want's composition digest"); err != nil {
return NodeWant{}, err
}
return w, nil
}
// EncodeHeld is h as JSON at Version; it refuses a node name a key refuses.
func EncodeHeld(h Held) ([]byte, error) {
if err := h.check(); err != nil {
+3 -3
View File
@@ -74,8 +74,8 @@ func TestDifferenceKey(t *testing.T) {
// 1297/B1a: wants, holds and differences round-trip through their encoders unchanged, at Version.
func TestAssignerValuesRoundTrip(t *testing.T) {
wants := []Want{
{FP: fp2of('a')},
{FP: fp2of('a'), Newest: fp2of('b'), Rule: RuleCanary, Words: "the canary ace does not run it healthy yet"},
{FP: fp2of('a'), Composed: hashOf('1')},
{FP: fp2of('a'), Composed: hashOf('1'), Newest: fp2of('b'), Rule: RuleCanary, Words: "the canary ace does not run it healthy yet"},
{Removed: true, Words: "the module left g14"},
}
for _, w := range wants {
@@ -127,7 +127,7 @@ func TestAssignerValuesRefused(t *testing.T) {
{Removed: true, FP: fp2of('a')},
{FP: fp2of('a'), Newest: fp2of('b')},
{FP: fp2of('a'), Newest: fp2of('a'), Rule: 3},
{FP: fp2of('a'), Newest: fp2of('b'), Rule: 7},
{FP: fp2of('a'), Composed: hashOf('1'), Newest: fp2of('b'), Rule: 8},
} {
if _, err := EncodeWant(w); err == nil {
t.Errorf("EncodeWant(%+v) accepted", w)
+27
View File
@@ -0,0 +1,27 @@
package delivery
import (
"strings"
"testing"
)
// 1791/1921 S9: a member of a group that merged stays in the candidate, marked merged, with its head, so the heads hash
// the Approve covers holds until every member merged; a candidate whose every member merged is refused.
func TestACandidateKeepsAMergedMemberAndRefusesOneWhoseEveryMemberMerged(t *testing.T) {
a := Head{Repository: "novox/a", Number: 1, Head: strings.Repeat("1", 40)}
b := Head{Repository: "novox/b", Number: 2, Head: strings.Repeat("2", 40)}
hash, _ := HeadsHash([]Head{a, b})
c := Candidate{Group: "feat/x", HeadsHash: hash, Pulls: []CandidatePull{{Head: a, Modules: []string{}, Merged: true}, {Head: b, Modules: []string{}}}}
raw, err := EncodeCandidate(c)
if err != nil {
t.Fatal(err)
}
back, err := DecodeCandidate(raw)
if err != nil || !back.Pulls[0].Merged || back.Pulls[1].Merged || back.HeadsHash != hash {
t.Fatalf("read back %+v, %v", back, err)
}
c.Pulls[1].Merged = true
if _, err := EncodeCandidate(c); err == nil {
t.Error("a candidate whose every member merged was written")
}
}
+21 -6
View File
@@ -139,16 +139,25 @@ type FailedFirst struct {
Words string `json:"words,omitempty"`
}
// MaxFailures is the most entries a verdict's Failures holds (issue 1921): fifty tests named one by one, and an entry
// after them saying how many more failed.
const MaxFailures = 50
// Verdict is the value of a verdict key: the group and its heads hash, the trunk head of each repository it was judged
// against (repository to full sha), the State (running, pass or fail), what failed first, the tests run and kept per
// level, each part's time, the machines composed and the mesh-sdk version the composition ran with (ADR 0446 clause 5).
type Verdict struct {
V int `json:"v"`
Group string `json:"group"`
Heads string `json:"heads"`
Trunk map[string]string `json:"trunk,omitempty"`
State string `json:"state"`
Failed FailedFirst `json:"failed,omitzero"`
V int `json:"v"`
Group string `json:"group"`
Heads string `json:"heads"`
Trunk map[string]string `json:"trunk,omitempty"`
State string `json:"state"`
Failed FailedFirst `json:"failed,omitzero"`
// Failures names every test that failed, in the order judged, each with the first line of its words (issue 1921):
// Failed keeps the first one's words whole, and a check that ran every test says all it found here. At most
// MaxFailures entries, the last of a longer list saying how many more failed. Absent in a value written before it,
// and from a verdict whose failure is no test.
Failures []FailedFirst `json:"failures,omitempty"`
Levels map[string]LevelResult `json:"levels,omitempty"`
Took Took `json:"took,omitzero"`
Machines []string `json:"machines,omitempty"`
@@ -222,6 +231,12 @@ func (v Verdict) check() error {
}
}
named := v.Failed.Test != "" || v.Failed.Machine != ""
if len(v.Failures) > 0 && v.State != VerdictFail {
return fmt.Errorf("delivery: a %s verdict names no failures", v.State)
}
if len(v.Failures) > MaxFailures+1 {
return fmt.Errorf("delivery: the verdict names %d failures, more than %d and the count of the rest", len(v.Failures), MaxFailures)
}
switch v.State {
case VerdictRunning:
if v.Failed != (FailedFirst{}) {
+36
View File
@@ -3,6 +3,7 @@ package delivery
import (
"encoding/json"
"errors"
"strconv"
"strings"
"testing"
"time"
@@ -600,3 +601,38 @@ func TestPreviewOfAHeadInConflict(t *testing.T) {
t.Errorf("the old reader: %+v %v", old, err)
}
}
// Issue 1921: a failed verdict names every failed test in Failures and reads back the same; an older reader's value
// without it reads too; Failures on a verdict that is no fail, and more than MaxFailures and the count of the rest, are
// refused.
func TestVerdictNamesEveryFailure(t *testing.T) {
v := passVerdict()
v.State = VerdictFail
v.Failed = FailedFirst{Test: "integration: x/a.TestA", Words: "a_test.go:9: want 1, got 2\nmore"}
v.Failures = []FailedFirst{{Test: "integration: x/a.TestA", Words: "a_test.go:9: want 1, got 2"}, {Test: "integration: x/b.TestB", Words: "b_test.go:3: boom"}}
data, err := EncodeVerdict(v)
if err != nil {
t.Fatal(err)
}
got, err := DecodeVerdict(data)
v.V = Version
if err != nil || !SameVerdict(got, v) || !strings.Contains(string(data), `"failures":[{"test":"integration: x/a.TestA"`) {
t.Fatalf("%s: %+v %v", data, got, err)
}
old := strings.Replace(string(data), string(data[strings.Index(string(data), `,"failures":`):strings.Index(string(data), `,"levels":`)]), "", 1)
if got, err := DecodeVerdict([]byte(old)); err != nil || len(got.Failures) != 0 || got.Failed != v.Failed {
t.Fatalf("a value without failures: %+v %v (%s)", got, err, old)
}
p := passVerdict()
p.Failures = v.Failures
if _, err := EncodeVerdict(p); err == nil {
t.Error("a pass with failures was written")
}
v.Failures = make([]FailedFirst, MaxFailures+2)
for i := range v.Failures {
v.Failures[i] = FailedFirst{Test: "unit: x.T" + strconv.Itoa(i)}
}
if _, err := EncodeVerdict(v); err == nil {
t.Error("a verdict naming more than MaxFailures and the rest was written")
}
}
+241
View File
@@ -0,0 +1,241 @@
package delivery
import (
"encoding/json"
"fmt"
"regexp"
"strings"
)
// The controller's compositions (novox/hq ADR 0539 clauses 1 to 4). A **composition** is one module's resources
// composed for one node at one fingerprint, written by the controller, which keeps the inputs. It lives in a bucket of
// each node, `composed-<node>`, which only that node's node-engine reads (a composition can carry a sealed value), under
// `<node>.<module>.<fingerprint hex>`, and the node's own resources (adoption, firewall openings, accounts, the
// resolver, identities and groups) under `<node>.node`. Beside it, under the same key in `mesh-controller_composed`,
// which the assigner reads, the controller writes the composition's digest and its judgement of the move from the
// composition the node runs now (ComposedDigest). The controller writes both by compare-and-set while it holds the
// lease; the operator's credential is a second writer on the hand path alone (clause 10, mesh-hand-compose).
//
// A composition is derived state (clause 4): its key and its digest are deleted by the controller when its fingerprint
// is no longer among the module's five newest and no want names it, and each bucket keeps five revisions of a key.
const (
// ComposedBucketPrefix starts the name of a node's bucket of compositions, ComposedBucketFor(node).
ComposedBucketPrefix = "composed-"
// ComposedHistory is the revisions a node's composition bucket keeps per key (ADR 0539 clause 4).
ComposedHistory = 5
// ControllerModule is the module of the controller, the writer of the compositions.
ControllerModule = "mesh-controller"
// ControllerComposedBucket is the controller's bucket of each composition's digest and judgement, `<owner
// module>_<name>`, read by the assigner (ADR 0539 clauses 2 and 6).
ControllerComposedBucket = ControllerModule + "_composed"
// ControllerComposedHistory is the revisions that bucket keeps per key.
ControllerComposedHistory = 5
// ComposedKeep is how many of a module's newest fingerprints keep their compositions though no want names them
// (ADR 0539 clause 4).
ComposedKeep = 5
)
// nodeWord is the last part of a node's own key, in the have bucket and in a node's composition bucket alike.
const nodeWord = "node"
var bucketNode = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
// ComposedBucketFor is the name of a node's bucket of compositions, "composed-<node>", as declaration.FileBucketFor
// names a node's object bucket (ADR 0488 §4). The node name must be one a bucket and a subject token accept: letters,
// digits, '_' and '-'.
func ComposedBucketFor(node string) (string, error) {
if node == "" {
return "", fmt.Errorf("delivery: the node name is empty; a node's composition bucket is named for its node")
}
if !bucketNode.MatchString(node) {
return "", fmt.Errorf("delivery: the node name %q is not one a bucket accepts: letters, digits, '_' and '-' only", node)
}
return ComposedBucketPrefix + node, nil
}
// ComposedKey is the key, in ComposedBucketFor(node) and in ControllerComposedBucket, of a module's composition for a
// node at a fingerprint: `<node>.<module>.<hex>`, the hex being the fingerprint's key part (FingerprintKeyPart). A
// module named `node` is refused, since `<node>.node` is the node's own key.
func ComposedKey(node, module, fingerprint string) (string, error) {
have, err := HaveKey(node, module)
if err != nil {
return "", err
}
part, err := FingerprintKeyPart(fingerprint)
if err != nil {
return "", err
}
return have + "." + part, nil
}
// ComposedNodeKey is the key of a node's own composition: `<node>.node`.
func ComposedNodeKey(node string) (string, error) {
return HaveNodeKey(node)
}
// ParseComposedKey is the inverse of ComposedKey and ComposedNodeKey: isNode is true for `<node>.node`, with module
// and fingerprint empty. It refuses any other shape, naming the key.
func ParseComposedKey(key string) (node, module, fingerprint string, isNode bool, err error) {
refused := func(why error) (string, string, string, bool, error) {
return "", "", "", false, fmt.Errorf("delivery: %q is not a composition key, <node>.<module>.<fingerprint hex> or <node>.node: %w", key, why)
}
parts := strings.Split(key, ".")
switch len(parts) {
case 2:
if parts[1] != nodeWord {
return refused(fmt.Errorf("a module's composition names its fingerprint"))
}
if err := checkName("node", parts[0]); err != nil {
return refused(err)
}
return parts[0], "", "", true, nil
case 3:
node, module, isNode, err := ParseHaveKey(parts[0] + "." + parts[1])
if err != nil {
return refused(err)
}
if isNode {
return refused(fmt.Errorf("the node's own key names no fingerprint"))
}
fp, err := ParseFingerprintKeyPart(parts[2])
if err != nil {
return refused(err)
}
return node, module, fp, false, nil
}
return refused(fmt.Errorf("it has %d parts", len(parts)))
}
// Judgement is the controller's judgement of a move from the composition a node runs, as its have names it, to a new
// one (ADR 0539 clauses 2 and 14, by ADR 0377 §3's tests).
type Judgement string
// The judgements of a move.
const (
// JudgedRecreates is a move that recreates a container.
JudgedRecreates Judgement = "recreates"
// JudgedMovesData is a move that gives the module's data another place.
JudgedMovesData Judgement = "moves-data"
// JudgedNeither is a move that does neither, and the judgement of a composition the node never ran.
JudgedNeither Judgement = "neither"
// JudgedCannot is a move that cannot be judged, because the composition the node runs is gone or cannot be read.
JudgedCannot Judgement = "cannot-judge"
// JudgedUnchanged is a composition equal to the module's slice of the declaration the node last applied, for a node
// whose have names no composition yet (the first want at the switch, novox/hq ADR 0540 clause 9): applying it
// changes nothing on the node, so it needs no person's yes.
JudgedUnchanged Judgement = "unchanged"
)
var digestForm = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
func checkDigest(digest, of string) error {
if !digestForm.MatchString(digest) {
return fmt.Errorf("delivery: %s %q is not sha256:<64 lowercase hex digits>", of, digest)
}
return nil
}
// ComposedDigest is the value, in ControllerComposedBucket, beside a composition: its Digest (ComposedDigestOf), and
// the Judgement of the move to it from the composition the node runs now, whose digest is Against (empty when the node
// runs no composition of the module). Words say why, and are required for a move that cannot be judged. Rewritten when
// the node's have changes.
//
// Policy is the module's upgrade policy as the controller holds it (ADR 0236, ADR 0539 clause 14), with PolicyFrom
// where it comes from and PolicyWhy why: the assigner holds a move of a module of policy record from it, the operator's
// own choice (`upgrade` with policy) included. Rewritten when the policy changes. Empty on a composition made by hand
// (mesh-hand-compose), whose want the operator writes.
type ComposedDigest struct {
V int `json:"v"`
Digest string `json:"digest"`
Judgement Judgement `json:"judgement"`
Against string `json:"against,omitempty"`
Words string `json:"words,omitempty"`
// UserListOnly says the composition differs from the one the node's have names only in the bus's user list (the
// bus module's file `<module>.bus-users`), whatever the judgement says of its mount: the assigner writes such a
// revision into the bus node's want at once, outside the record hold, at the fingerprint the node runs (ADR 0539
// clause 7).
UserListOnly bool `json:"user-list-only,omitempty"`
Policy string `json:"policy,omitempty"`
PolicyFrom string `json:"policy-from,omitempty"`
PolicyWhy string `json:"policy-why,omitempty"`
}
// The upgrade policies and their sources a composition's value names: catalogue.PolicyRoll, PolicyTogether and
// PolicyRecord, and catalogue.FromPerson, FromModule, FromBus, FromData and FromDefault, the same words (the compose
// package's test holds them equal; this package does not import the catalogue).
var (
composedPolicies = []string{"roll", "together", "record"}
composedPolicyFroms = []string{"person", "module", "the bus", "irreplaceable data", "default"}
)
func oneOf(v string, of []string) bool {
for _, o := range of {
if v == o {
return true
}
}
return false
}
func (c ComposedDigest) check() error {
if err := checkDigest(c.Digest, "the composition's digest"); err != nil {
return err
}
if c.Against != "" {
if err := checkDigest(c.Against, "the digest a move is judged against"); err != nil {
return err
}
}
switch c.Judgement {
case JudgedRecreates, JudgedMovesData:
if c.Against == "" {
return fmt.Errorf("delivery: a move judged %s names the composition it is judged against", c.Judgement)
}
case JudgedNeither, JudgedUnchanged:
case JudgedCannot:
if c.Words == "" {
return fmt.Errorf("delivery: a move that cannot be judged says why")
}
default:
return fmt.Errorf("delivery: the judgement %q is not %s, %s, %s, %s or %s", c.Judgement, JudgedRecreates,
JudgedMovesData, JudgedNeither, JudgedCannot, JudgedUnchanged)
}
if c.Policy != "" || c.PolicyFrom != "" {
if !oneOf(c.Policy, composedPolicies) {
return fmt.Errorf("delivery: the upgrade policy %q is not one of %v", c.Policy, composedPolicies)
}
if !oneOf(c.PolicyFrom, composedPolicyFroms) {
return fmt.Errorf("delivery: the upgrade policy's source %q is not one of %v", c.PolicyFrom, composedPolicyFroms)
}
}
return nil
}
// PolicyRecord is the policy whose moves the assigner holds for a person's yes (ADR 0539 clause 14).
const PolicyRecord = "record"
// EncodeComposedDigest is c as JSON at Version; it refuses an unknown judgement, a digest not of the sha256 form, a
// move judged to recreate or move data against nothing, and a move that cannot be judged without its reason.
func EncodeComposedDigest(c ComposedDigest) ([]byte, error) {
if err := c.check(); err != nil {
return nil, err
}
c.V = Version
return json.Marshal(c)
}
// DecodeComposedDigest reads the value beside a composition; it refuses an unknown version (ErrUnknownVersion) and
// what EncodeComposedDigest refuses.
func DecodeComposedDigest(data []byte) (ComposedDigest, error) {
var c ComposedDigest
if err := decode(data, &c); err != nil {
return ComposedDigest{}, err
}
if err := c.check(); err != nil {
return ComposedDigest{}, err
}
return c, nil
}
// SameComposedDigest says whether two values are the same, so the controller writes only when they differ.
func SameComposedDigest(a, b ComposedDigest) bool { return a == b }
+125
View File
@@ -0,0 +1,125 @@
package delivery
import (
"errors"
"strings"
"testing"
)
// 1943/AC1: a node's composition bucket is named for the node, and a node with no name, or one a bucket refuses,
// is refused, naming why.
func TestComposedBucketFor(t *testing.T) {
b, err := ComposedBucketFor("ace")
if err != nil || b != "composed-ace" {
t.Fatalf("ComposedBucketFor(ace) = %q, %v", b, err)
}
for _, bad := range []string{"", "a.b", "a*", "a b", "a>"} {
if _, err := ComposedBucketFor(bad); err == nil || !strings.Contains(err.Error(), "node") {
t.Errorf("ComposedBucketFor(%q) = %v, want a refusal naming the node", bad, err)
}
}
}
// 1943/AC1: a module's composition key and the node's own key round-trip through one parser; an empty node, a
// module named node, a bad fingerprint and a key of another shape are refused, naming why.
func TestComposedKeys(t *testing.T) {
fp := fp2of('c')
key, err := ComposedKey("ace", "mesh-mail", fp)
if err != nil || key != "ace.mesh-mail."+strings.Repeat("c", 64) {
t.Fatalf("ComposedKey = %q, %v", key, err)
}
node, module, back, isNode, err := ParseComposedKey(key)
if err != nil || node != "ace" || module != "mesh-mail" || back != fp || isNode {
t.Errorf("ParseComposedKey(%q) = %q, %q, %q, %v, %v", key, node, module, back, isNode, err)
}
own, err := ComposedNodeKey("ace")
if err != nil || own != "ace.node" {
t.Fatalf("ComposedNodeKey = %q, %v", own, err)
}
node, module, back, isNode, err = ParseComposedKey(own)
if err != nil || node != "ace" || module != "" || back != "" || !isNode {
t.Errorf("ParseComposedKey(%q) = %q, %q, %q, %v, %v", own, node, module, back, isNode, err)
}
refusals := []struct {
node, module, fp, why string
}{
{"", "mesh-mail", fp, "node name is empty"},
{"ace", "node", fp, `"node" is refused`},
{"ace", "", fp, "module name is empty"},
{"ace", "mesh-mail", "src2:abc", "not a fingerprint of the new form"},
{"ace", "mesh-mail", "src1:" + strings.Repeat("c", 64), "not a fingerprint of the new form"},
{"a.b", "mesh-mail", fp, "node name"},
}
for _, r := range refusals {
if _, err := ComposedKey(r.node, r.module, r.fp); err == nil || !strings.Contains(err.Error(), r.why) {
t.Errorf("ComposedKey(%q, %q, %q) = %v, want a refusal saying %q", r.node, r.module, r.fp, err, r.why)
}
}
if _, err := ComposedNodeKey(""); err == nil {
t.Errorf("ComposedNodeKey(\"\") accepted")
}
for _, bad := range []string{"", "ace", "ace.mesh-mail", "ace.node.x", "ace.mesh-mail.abc", ".mesh-mail." + strings.Repeat("c", 64),
"ace.node." + strings.Repeat("c", 64), "ace.mesh-mail." + strings.Repeat("c", 64) + ".x", "want.ace.mesh-mail"} {
if _, _, _, _, err := ParseComposedKey(bad); err == nil || !strings.Contains(err.Error(), "\""+bad+"\"") {
t.Errorf("ParseComposedKey(%q) = %v, want a refusal naming it", bad, err)
}
}
}
// 1943/AC1: the controller's value beside a composition, its digest and the judgement of the move, round-trips; an
// unknown judgement, a digest not of the sha256 form, a move judged against nothing, a cannot-judge without its
// reason, and an upgrade policy or source outside the catalogue's words are refused, naming why; an unknown version is
// refused as one. The module's upgrade policy and its source ride beside the judgement, for the assigner's hold.
func TestComposedDigestValue(t *testing.T) {
digest := "sha256:" + strings.Repeat("d", 64)
against := "sha256:" + strings.Repeat("e", 64)
for _, v := range []ComposedDigest{
{Digest: digest, Judgement: JudgedNeither},
{Digest: digest, Judgement: JudgedNeither, Against: against},
{Digest: digest, Judgement: JudgedRecreates, Against: against, Words: "the container's image changes"},
{Digest: digest, Judgement: JudgedMovesData, Against: against},
{Digest: digest, Judgement: JudgedCannot, Words: "the composition the node runs is gone"},
{Digest: digest, Judgement: JudgedUnchanged, Words: "equal to the declaration the node last applied"},
{Digest: digest, Judgement: JudgedNeither, Policy: "record", PolicyFrom: "person", PolicyWhy: "the operator: its data"},
{Digest: digest, Judgement: JudgedNeither, Policy: "roll", PolicyFrom: "default"},
{Digest: digest, Judgement: JudgedNeither, Against: against, UserListOnly: true, Words: "only the bus's user list changes"},
} {
raw, err := EncodeComposedDigest(v)
if err != nil {
t.Fatalf("EncodeComposedDigest(%+v): %v", v, err)
}
back, err := DecodeComposedDigest(raw)
v.V = Version
if err != nil || !SameComposedDigest(back, v) {
t.Errorf("round trip of %+v = %+v, %v", v, back, err)
}
}
refusals := []struct {
v ComposedDigest
why string
}{
{ComposedDigest{Digest: digest, Judgement: "maybe"}, `judgement "maybe"`},
{ComposedDigest{Digest: digest}, `judgement ""`},
{ComposedDigest{Digest: "sha256:abc", Judgement: JudgedNeither}, "not sha256"},
{ComposedDigest{Judgement: JudgedNeither}, "not sha256"},
{ComposedDigest{Digest: digest, Judgement: JudgedRecreates}, "against"},
{ComposedDigest{Digest: digest, Judgement: JudgedMovesData}, "against"},
{ComposedDigest{Digest: digest, Judgement: JudgedCannot}, "why"},
{ComposedDigest{Digest: digest, Judgement: JudgedNeither, Against: "sha256:zz"}, "not sha256"},
{ComposedDigest{Digest: digest, Judgement: JudgedNeither, Policy: "sometimes", PolicyFrom: "person"}, `policy "sometimes"`},
{ComposedDigest{Digest: digest, Judgement: JudgedNeither, Policy: "record"}, `source ""`},
{ComposedDigest{Digest: digest, Judgement: JudgedNeither, PolicyFrom: "person"}, `policy ""`},
}
for _, r := range refusals {
if _, err := EncodeComposedDigest(r.v); err == nil || !strings.Contains(err.Error(), r.why) {
t.Errorf("EncodeComposedDigest(%+v) = %v, want a refusal saying %q", r.v, err, r.why)
}
}
if _, err := DecodeComposedDigest([]byte(`{"v":2,"digest":"` + digest + `","judgement":"neither"}`)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("version 2 = %v, want ErrUnknownVersion", err)
}
if _, err := DecodeComposedDigest([]byte(`{"v":1,"digest":"` + digest + `","judgement":"perhaps"}`)); err == nil {
t.Errorf("an unknown judgement was read")
}
}
+100
View File
@@ -0,0 +1,100 @@
package delivery
import (
"crypto/ed25519"
"encoding/base64"
"fmt"
)
// A composition's signature (novox/hq ADR 0539 clause 3): the controller signs each composition it writes with its
// key, and the operator signs one made by hand with mesh-hand-compose with theirs. The node-engine trusts exactly these two
// signers and applies nothing of a composition whose signature fails or whose signer is neither. The signature is an
// ed25519 signature, as a declaration's is, of the composition's digest (ComposedDigestOf) after a context line, so a
// signature made for anything else never verifies here.
const composedSigningContext = "novox composition v1\n"
// Signature is who signed a composition and the signature: Signer the signer's ed25519 public key and Sig the
// signature, both in standard base64.
type Signature struct {
Signer string `json:"signer"`
Sig string `json:"sig"`
}
func (s Signature) check() error {
if key, err := base64.StdEncoding.DecodeString(s.Signer); err != nil || len(key) != ed25519.PublicKeySize {
return fmt.Errorf("delivery: a composition's signer %q is not an ed25519 public key in base64", s.Signer)
}
if sig, err := base64.StdEncoding.DecodeString(s.Sig); err != nil || len(sig) != ed25519.SignatureSize {
return fmt.Errorf("delivery: a composition's signature is not an ed25519 signature in base64")
}
return nil
}
// SignComposed is c signed with key: its digest signed, and the key's public half named as the signer. A signature c
// already carries is replaced.
func SignComposed(c Composed, key ed25519.PrivateKey) (Composed, error) {
if len(key) != ed25519.PrivateKeySize {
return Composed{}, fmt.Errorf("delivery: the key to sign a composition with is not an ed25519 private key")
}
return SignComposedBy(c, key.Public().(ed25519.PublicKey), func(message []byte) ([]byte, error) {
return ed25519.Sign(key, message), nil
})
}
// SignComposedBy is SignComposed for a signer that keeps its private half to itself (the controller's identity store
// signs a message and hands back the signature): sign signs the message it is given with the key whose public half is
// signer. A signature that does not verify against signer is refused, so a composition is never written signed by a
// key other than the one it names.
func SignComposedBy(c Composed, signer ed25519.PublicKey, sign func(message []byte) ([]byte, error)) (Composed, error) {
if len(signer) != ed25519.PublicKeySize {
return Composed{}, fmt.Errorf("delivery: the signer of a composition is not an ed25519 public key")
}
c.Signature = nil
digest, err := ComposedDigestOf(c)
if err != nil {
return Composed{}, err
}
message := []byte(composedSigningContext + digest)
sig, err := sign(message)
if err != nil {
return Composed{}, err
}
if !ed25519.Verify(signer, message, sig) {
return Composed{}, fmt.Errorf("delivery: the signature made for the composition of %q on %s does not verify against its signer", c.Module, c.Node)
}
c.Signature = &Signature{Signer: base64.StdEncoding.EncodeToString(signer), Sig: base64.StdEncoding.EncodeToString(sig)}
return c, nil
}
// VerifyComposed is c's digest when c is signed by one of the trusted keys and its signature verifies against its
// digest; otherwise it refuses, saying why, and nothing of c is to be applied: an unsigned composition, one whose
// signer is not a trusted key, and one changed after it was signed.
func VerifyComposed(c Composed, trusted []ed25519.PublicKey) (string, error) {
if c.Signature == nil {
return "", fmt.Errorf("delivery: the composition of %q on %s is unsigned, and only a signed one is applied", c.Module, c.Node)
}
if err := c.Signature.check(); err != nil {
return "", err
}
signer, _ := base64.StdEncoding.DecodeString(c.Signature.Signer)
var key ed25519.PublicKey
for _, t := range trusted {
if t.Equal(ed25519.PublicKey(signer)) {
key = t
}
}
if key == nil {
return "", fmt.Errorf("delivery: the composition of %q on %s is signed by %s, which is not a trusted signer (the controller's key or the operator's)",
c.Module, c.Node, c.Signature.Signer)
}
digest, err := ComposedDigestOf(c)
if err != nil {
return "", err
}
sig, _ := base64.StdEncoding.DecodeString(c.Signature.Sig)
if !ed25519.Verify(key, []byte(composedSigningContext+digest), sig) {
return "", fmt.Errorf("delivery: the composition of %q on %s does not match its signature: it was changed after it was signed, or signed for another",
c.Module, c.Node)
}
return digest, nil
}
+103
View File
@@ -0,0 +1,103 @@
package delivery
import (
"crypto/ed25519"
"crypto/rand"
"encoding/json"
"strings"
"testing"
)
func aSigningKey(t *testing.T) (ed25519.PublicKey, ed25519.PrivateKey) {
t.Helper()
pub, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
return pub, priv
}
func aComposition() Composed {
return Composed{Node: "ace", Module: "m", FP: fp2of('a'), Resources: []json.RawMessage{
json.RawMessage(`{"id":"m.f","type":"file","module":"m","path":"/etc/m","content":"x"}`)}}
}
// 1943/AC5: a composition signed by the controller's key or by the operator's verifies against the two trusted keys,
// keeps its digest, and survives its encoding; one signed by another key, unsigned, or changed by one byte after it was
// signed does not verify, and nothing of it is to be applied.
func TestACompositionIsSignedByATrustedKey(t *testing.T) {
controllerPub, controller := aSigningKey(t)
operatorPub, operator := aSigningKey(t)
_, stranger := aSigningKey(t)
trusted := []ed25519.PublicKey{controllerPub, operatorPub}
c := aComposition()
want, _ := ComposedDigestOf(c)
for name, key := range map[string]ed25519.PrivateKey{"controller": controller, "operator": operator} {
signed, err := SignComposed(c, key)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
raw, err := EncodeComposed(signed)
if err != nil {
t.Fatal(err)
}
back, err := DecodeComposed(raw)
if err != nil {
t.Fatal(err)
}
digest, err := VerifyComposed(back, trusted)
if err != nil || digest != want {
t.Errorf("%s's signature: %q, %v; want the digest %s", name, digest, err, want)
}
if d, _ := ComposedDigestOf(back); d != want {
t.Errorf("%s's signature changed the digest: %s", name, d)
}
changed := strings.Replace(string(raw), `"content":"x"`, `"content":"y"`, 1)
if changed == string(raw) {
t.Fatal("the test changed nothing")
}
if tampered, err := DecodeComposed([]byte(changed)); err != nil {
t.Fatal(err)
} else if _, err := VerifyComposed(tampered, trusted); err == nil || !strings.Contains(err.Error(), "signature") {
t.Errorf("%s: a composition changed by one byte verified: %v", name, err)
}
}
byStranger, _ := SignComposed(c, stranger)
if _, err := VerifyComposed(byStranger, trusted); err == nil || !strings.Contains(err.Error(), "not a trusted") {
t.Errorf("a stranger's signature = %v, want refused as no trusted signer", err)
}
if _, err := VerifyComposed(c, trusted); err == nil || !strings.Contains(err.Error(), "unsigned") {
t.Errorf("an unsigned composition = %v, want refused", err)
}
signed, _ := SignComposed(c, controller)
forged := signed
forged.Signature = &Signature{Signer: signed.Signature.Signer, Sig: byStranger.Signature.Sig}
if _, err := VerifyComposed(forged, trusted); err == nil {
t.Errorf("a stranger's signature under the controller's name verified")
}
if _, err := EncodeComposed(Composed{Node: "ace", Module: "m", FP: fp2of('a'), Resources: c.Resources,
Signature: &Signature{Signer: "not base64!", Sig: "x"}}); err == nil {
t.Errorf("a signature naming no key was encoded")
}
}
// 1943/AC5: a signer that keeps its private half signs through a function, and the composition verifies as one signed
// with the key; a function that signs with another key than the one named is refused.
func TestACompositionIsSignedByAKeptKey(t *testing.T) {
pub, priv := aSigningKey(t)
_, other := aSigningKey(t)
c := aComposition()
signed, err := SignComposedBy(c, pub, func(m []byte) ([]byte, error) { return ed25519.Sign(priv, m), nil })
if err != nil {
t.Fatal(err)
}
if _, err := VerifyComposed(signed, []ed25519.PublicKey{pub}); err != nil {
t.Errorf("a composition signed through the function does not verify: %v", err)
}
if _, err := SignComposedBy(c, pub, func(m []byte) ([]byte, error) { return ed25519.Sign(other, m), nil }); err == nil {
t.Errorf("a signature by another key than the one named was accepted")
}
}
+330
View File
@@ -0,0 +1,330 @@
package delivery
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// NodeOwn is what a node's own composition (`<node>.node`) carries beside its resources: what the declaration says of
// the node as a whole (novox/hq ADR 0539 clause 2: adoption, accounts, identities and groups). A module's composition
// carries none of it.
type NodeOwn struct {
// Adoption says the node is adopted and which of its modules are taken (ADR 0100, ADR 0103); nil for a converged node.
Adoption *declaration.Adoption `json:"adoption,omitempty"`
// Identities is the identity of every module assigned to the node (ADR 0328 §4), Groups and Libraries every library
// it is reached by (ADR 0328 §3).
Identities []declaration.Identity `json:"identities,omitempty"`
Groups []declaration.Group `json:"groups,omitempty"`
Libraries []declaration.Library `json:"libraries,omitempty"`
// Provisioner is the account the provisioner's client runs as on the node (ADR 0349 §4).
Provisioner string `json:"provisioner_account,omitempty"`
}
func (o NodeOwn) empty() bool {
return o.Adoption == nil && len(o.Identities) == 0 && len(o.Groups) == 0 && len(o.Libraries) == 0 && o.Provisioner == ""
}
// Composed is a composition (ADR 0539 clause 1): one module's resources composed for one node at one fingerprint, or,
// with Module and FP empty, the node's own resources and parts (Own). Each resource is a declaration resource as the
// node-engine reads it, saying its module; it is kept as JSON and written canonically (its keys sorted, its spacing
// dropped), so its digest (ComposedDigestOf) is a function of its content alone. Data says the module declares data,
// so the node agent backs it up before it moves (ADR 0429 §14). Filled names the files of the module the provisioner
// fills (ADR 0349 §5). A file over 64 KiB travels by digest and size (ByDigest), its content in the node's object
// bucket (ADR 0539 clause 4).
type Composed struct {
V int
Node string
Module string
FP string
Data bool
Resources []json.RawMessage
Filled []string
Own NodeOwn
// Signature is the controller's or, made by hand, the operator's (ADR 0539 clause 3, SignComposed); it signs the
// digest and is never part of it.
Signature *Signature
}
// composedWire is a composition as it is written.
type composedWire struct {
V int `json:"v"`
Node string `json:"node"`
Module string `json:"module,omitempty"`
FP string `json:"fp,omitempty"`
Data bool `json:"data,omitempty"`
Resources []json.RawMessage `json:"resources"`
Filled []string `json:"filled_files,omitempty"`
Own *NodeOwn `json:"own,omitempty"`
Signature *Signature `json:"signature,omitempty"`
}
// canonicalJSON is raw with its object keys sorted and its spacing dropped, its numbers kept as written.
func canonicalJSON(raw json.RawMessage) (json.RawMessage, error) {
var v any
d := json.NewDecoder(bytes.NewReader(raw))
d.UseNumber()
if err := d.Decode(&v); err != nil {
return nil, err
}
if _, err := d.Token(); err != io.EOF {
return nil, fmt.Errorf("more after the value")
}
return json.Marshal(v)
}
func (c Composed) check() error {
if _, err := HaveNodeKey(c.Node); err != nil {
return err
}
if c.Module == "" {
if c.FP != "" {
return fmt.Errorf("delivery: the node's own composition names no fingerprint, and this one names %s", c.FP)
}
// It carries the filled paths of its own files, those of a module with no fingerprint of the new form moved
// into it (issue 1945), and never a module's data.
if c.Data {
return fmt.Errorf("delivery: the node's own composition holds no module's data")
}
} else {
if _, err := ComposedKey(c.Node, c.Module, c.FP); err != nil {
return err
}
if !c.Own.empty() {
return fmt.Errorf("delivery: the composition of %s carries the node's own parts, which only <node>.node carries", c.Module)
}
}
seen := map[string]bool{}
files := map[string][]*declaration.File{}
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
return fmt.Errorf("delivery: the composition of %q on %s: %w", c.Module, c.Node, err)
}
if seen[r.Identity()] {
return fmt.Errorf("delivery: the composition of %q on %s holds the resource %q twice", c.Module, c.Node, r.Identity())
}
seen[r.Identity()] = true
if r.DeclaredModule() != c.Module {
return fmt.Errorf("delivery: the resource %q is of %q, and this is the composition of %q on %s",
r.Identity(), r.DeclaredModule(), c.Module, c.Node)
}
if f, ok := r.(*declaration.File); ok {
if f.TravelsByDigest() {
return fmt.Errorf("delivery: the file %q is over %d bytes and travels by digest (ByDigest), not whole",
f.ID, declaration.FileReferenceThreshold)
}
files[f.Path] = append(files[f.Path], f)
}
}
// A file the provisioner fills says no content, so a composition that does not say it is filled would have the
// node-engine write it empty (issue 1945, live on 2026-10-25): refused, naming it.
filledHere := make(map[string]bool, len(c.Filled))
for _, path := range c.Filled {
filledHere[path] = true
}
for path, fs := range files {
for _, f := range fs {
if f.OwnerAtRuntime == declaration.RuntimeOwned && !filledHere[path] && !saysContent(f) {
return fmt.Errorf("delivery: the file %q says no content and the composition of %q on %s does not say "+
"the provisioner fills %s: the node-engine would write it empty", f.ID, c.Module, c.Node, path)
}
}
}
for _, path := range c.Filled {
if len(files[path]) == 0 {
return fmt.Errorf("delivery: the composition of %q fills %q, and declares no file there", c.Module, path)
}
for _, f := range files[path] {
if f.OwnerAtRuntime == "" {
return fmt.Errorf("delivery: the file %q is filled by the provisioner and says no owner-at-runtime", f.ID)
}
}
}
if c.Signature != nil {
if err := c.Signature.check(); err != nil {
return err
}
}
return nil
}
// saysContent says a file names what to write: content, bytes, a digest or a sealed value. A file the provisioner fills
// names none of them.
func saysContent(f *declaration.File) bool {
return f.Content != "" || f.Bytes != "" || f.Digest != "" || f.Sealed != "" || f.VaultSealed != nil
}
// wire is c checked and made canonical, ready to marshal.
func (c Composed) wire() (composedWire, error) {
if err := c.check(); err != nil {
return composedWire{}, err
}
w := composedWire{V: Version, Node: c.Node, Module: c.Module, FP: c.FP, Data: c.Data, Filled: c.Filled,
Signature: c.Signature, Resources: make([]json.RawMessage, len(c.Resources))}
for i, r := range c.Resources {
canon, err := canonicalJSON(r)
if err != nil {
return composedWire{}, fmt.Errorf("delivery: a resource of %q is not JSON: %w", c.Module, err)
}
w.Resources[i] = canon
}
if !c.Own.empty() {
own := c.Own
w.Own = &own
}
return w, nil
}
// EncodeComposed is c as JSON at Version, each resource canonical. It refuses a composition whose node or fingerprint
// a key refuses, a resource its kind refuses (an action among them), a resource of another module, a module's
// composition carrying the node's own parts, a file over 64 KiB carried whole, and a filled path no file declares.
func EncodeComposed(c Composed) ([]byte, error) {
w, err := c.wire()
if err != nil {
return nil, err
}
return json.Marshal(w)
}
// DecodeComposed reads a composition; it refuses an unknown version (ErrUnknownVersion), a field it does not know, and
// what EncodeComposed refuses.
func DecodeComposed(data []byte) (Composed, error) {
if err := checkVersion(data); err != nil {
return Composed{}, err
}
var w composedWire
d := json.NewDecoder(bytes.NewReader(data))
d.DisallowUnknownFields()
if err := d.Decode(&w); err != nil {
return Composed{}, fmt.Errorf("delivery: a composition does not fit its shape: %w", err)
}
c := Composed{V: w.V, Node: w.Node, Module: w.Module, FP: w.FP, Data: w.Data, Resources: w.Resources,
Filled: w.Filled, Signature: w.Signature}
if w.Own != nil {
c.Own = *w.Own
}
if err := c.check(); err != nil {
return Composed{}, err
}
return c, nil
}
// ComposedDigestOf is a composition's digest, "sha256:<hex>", of its encoding without its signature. A want names it
// (ADR 0539 clause 1), the controller writes it beside the composition, and the signature signs it.
func ComposedDigestOf(c Composed) (string, error) {
c.Signature = nil
raw, err := EncodeComposed(c)
if err != nil {
return "", err
}
sum := sha256.Sum256(raw)
return "sha256:" + hex.EncodeToString(sum[:]), nil
}
// SameComposed says whether two compositions are the same value, their signatures included, so the controller writes
// only when they differ.
func SameComposed(a, b Composed) bool {
ra, errA := EncodeComposed(a)
rb, errB := EncodeComposed(b)
return errA == nil && errB == nil && bytes.Equal(ra, rb)
}
// ByDigest is c with every file over 64 KiB given by its digest and size, and the content of each by digest, for the
// writer to put in the node's object bucket (declaration.FileBucketFor) under declaration.ObjectName before it writes
// the composition (ADR 0539 clause 4, ADR 0488 §3). A file that is sealed, filled with secrets or written into another
// stays as it is (declaration.File.TravelsByDigest).
func (c Composed) ByDigest() (Composed, map[string][]byte, error) {
out := c
out.Resources = make([]json.RawMessage, len(c.Resources))
objects := map[string][]byte{}
for i, raw := range c.Resources {
out.Resources[i] = raw
r, err := declaration.ParseResource(raw)
if err != nil {
return Composed{}, nil, err
}
f, ok := r.(*declaration.File)
if !ok || !f.TravelsByDigest() {
continue
}
ref, body, err := f.AsReference()
if err != nil {
return Composed{}, nil, err
}
var fields map[string]json.RawMessage
if err := json.Unmarshal(raw, &fields); err != nil {
return Composed{}, nil, err
}
delete(fields, "content")
delete(fields, "bytes")
fields["digest"], _ = json.Marshal(ref.Digest)
fields["size"], _ = json.Marshal(ref.Size)
rewritten, err := json.Marshal(fields)
if err != nil {
return Composed{}, nil, err
}
out.Resources[i] = rewritten
objects[ref.Digest] = body
}
return out, objects, nil
}
// SplitComposed splits a node's composed resources into its compositions, by key (ComposedKey, ComposedNodeKey): each
// resource into the composition of the module it says, under that module's fingerprint in fps, and a resource that
// says no module into the node's own, with own. Each filled path goes with the module whose file declares it. The order
// of the resources is kept within each composition. A resource of a module fps names no fingerprint for is refused,
// naming it. Data is the caller's to set, from the module's manifest.
func SplitComposed(node string, resources []json.RawMessage, fps map[string]string, filled []string, own NodeOwn) (map[string]Composed, error) {
nodeKey, err := ComposedNodeKey(node)
if err != nil {
return nil, err
}
out := map[string]Composed{nodeKey: {Node: node, Own: own, Resources: []json.RawMessage{}}}
pathOf := map[string]string{}
for _, raw := range resources {
var head struct {
ID string `json:"id"`
Type string `json:"type"`
Module string `json:"module"`
Path string `json:"path"`
}
if err := json.Unmarshal(raw, &head); err != nil {
return nil, fmt.Errorf("delivery: a resource of %s is not JSON: %w", node, err)
}
key := nodeKey
if head.Module != "" {
fp, ok := fps[head.Module]
if !ok {
return nil, fmt.Errorf("delivery: the resource %q is of %q, whose fingerprint on %s is not given", head.ID, head.Module, node)
}
if key, err = ComposedKey(node, head.Module, fp); err != nil {
return nil, err
}
if _, ok := out[key]; !ok {
out[key] = Composed{Node: node, Module: head.Module, FP: fp}
}
}
if head.Type == string(declaration.TypeFile) && head.Module != "" {
pathOf[head.Path] = key
}
c := out[key]
c.Resources = append(c.Resources, raw)
out[key] = c
}
for _, path := range filled {
key, ok := pathOf[path]
if !ok {
return nil, fmt.Errorf("delivery: the provisioner fills %q on %s, and no module's file declares it", path, node)
}
c := out[key]
c.Filled = append(c.Filled, path)
out[key] = c
}
return out, nil
}
+364
View File
@@ -0,0 +1,364 @@
package delivery
import (
"bytes"
"compress/gzip"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// capturedDeclaration is a node's declaration as the controller's `plan` verb answered it on 2026-10-24 at 16:19
// (testdata/declaration-<node>.json.gz; the resources carrying a sealed value or a secret taken out, and the filled
// files that named them), never typed by hand.
type capturedDeclaration struct {
Builds map[string]string `json:"builds"`
Resources []json.RawMessage `json:"resources"`
Filled []string `json:"filled_files"`
Provisioner string `json:"provisioner_account"`
}
func readCaptured(t *testing.T, node string) capturedDeclaration {
t.Helper()
f, err := os.Open("testdata/declaration-" + node + ".json.gz")
if err != nil {
t.Fatal(err)
}
defer f.Close()
z, err := gzip.NewReader(f)
if err != nil {
t.Fatal(err)
}
raw, err := io.ReadAll(z)
if err != nil {
t.Fatal(err)
}
var c capturedDeclaration
if err := json.Unmarshal(raw, &c); err != nil {
t.Fatal(err)
}
return c
}
// fpOfBuild stands for each module's fingerprint in a captured declaration, which names builds, not fingerprints.
func fpOfBuild(build string) string {
var b strings.Builder
for b.Len() < 64 {
b.WriteString(fmt.Sprintf("%x", []byte(build)))
}
return SourcePrefix2 + b.String()[:64]
}
func splitCaptured(t *testing.T, node string, c capturedDeclaration) map[string]Composed {
t.Helper()
fps := map[string]string{}
for m, b := range c.Builds {
fps[m] = fpOfBuild(b)
}
// mesh-wireguard's resources are composed with no build of it named; its fingerprint is the indexer's, as every
// module's is, and stood in for here by its name.
for _, r := range c.Resources {
var head struct {
Module string `json:"module"`
}
_ = json.Unmarshal(r, &head)
if _, ok := fps[head.Module]; head.Module != "" && !ok {
fps[head.Module] = fpOfBuild(head.Module)
}
}
split, err := SplitComposed(node, c.Resources, fps, c.Filled, NodeOwn{Provisioner: c.Provisioner})
if err != nil {
t.Fatalf("SplitComposed(%s): %v", node, err)
}
return split
}
func canonical(t *testing.T, raw json.RawMessage) string {
t.Helper()
var v any
d := json.NewDecoder(bytes.NewReader(raw))
d.UseNumber()
if err := d.Decode(&v); err != nil {
t.Fatal(err)
}
out, _ := json.Marshal(v)
return string(out)
}
// 1943/AC2: each module's slice of a captured declaration of each node, and the node's own resources, round-trip
// through the composition's encoding with the same digest; together they are the declaration's resources, each once,
// each in the composition of the module it says.
func TestCompositionsOfACapturedDeclaration(t *testing.T) {
t.Parallel()
for _, node := range []string{"ace", "g14"} {
t.Run(node, func(t *testing.T) {
t.Parallel()
oneCapturedDeclaration(t, node)
})
}
}
func oneCapturedDeclaration(t *testing.T, node string) {
{
c := readCaptured(t, node)
split := splitCaptured(t, node, c)
modules := map[string]bool{}
for m := range c.Builds {
modules[m] = true
}
var union []string
for key, comp := range split {
n, module, fp, isNode, err := ParseComposedKey(key)
if err != nil || n != node || comp.Node != node || comp.Module != module || comp.FP != fp {
t.Fatalf("%s: key %q holds %s/%s/%s (%v)", node, key, comp.Node, comp.Module, comp.FP, err)
}
if !isNode && !modules[module] && module != "mesh-wireguard" {
t.Errorf("%s: a composition of %s, which the declaration names no build of", node, module)
}
raw, err := EncodeComposed(comp)
if err != nil {
t.Fatalf("%s: EncodeComposed(%s): %v", node, key, err)
}
back, err := DecodeComposed(raw)
if err != nil {
t.Fatalf("%s: DecodeComposed(%s): %v", node, key, err)
}
if !SameComposed(back, comp) {
t.Errorf("%s: %s did not round-trip", node, key)
}
// SameComposed encodes both, so the digest of what was read back, a hash of its encoding, is the one written.
if d, err := ComposedDigestOf(back); err != nil || !digestForm.MatchString(d) {
t.Errorf("%s: %s digests %q (%v)", node, key, d, err)
}
for _, r := range back.Resources {
var head struct {
Module string `json:"module"`
}
_ = json.Unmarshal(r, &head)
if head.Module != module {
t.Errorf("%s: %s holds a resource of %q", node, key, head.Module)
}
union = append(union, canonical(t, r))
}
}
var want []string
for _, r := range c.Resources {
want = append(want, canonical(t, r))
}
sort.Strings(union)
sort.Strings(want)
if strings.Join(union, "\n") != strings.Join(want, "\n") {
t.Errorf("%s: the compositions hold %d resources, the declaration %d, or they differ", node, len(union), len(want))
}
}
}
// 1943/AC2: one changed byte of a resource is another digest; the same composition written with its keys in another
// order is the same digest.
func TestComposedDigestFollowsTheContent(t *testing.T) {
c := readCaptured(t, "g14")
split := splitCaptured(t, "g14", c)
var comp Composed
for _, v := range split {
if v.Module != "" && len(v.Resources) > 1 {
comp = v
break
}
}
d1, _ := ComposedDigestOf(comp)
var m map[string]json.RawMessage
_ = json.Unmarshal(comp.Resources[0], &m)
var keys []string
for k := range m {
keys = append(keys, k)
}
sort.Sort(sort.Reverse(sort.StringSlice(keys)))
var parts []string
for _, k := range keys {
parts = append(parts, fmt.Sprintf("%q : %s", k, m[k]))
}
reordered := json.RawMessage("{ " + strings.Join(parts, ",\n ") + " }")
shuffled := comp
shuffled.Resources = append([]json.RawMessage{reordered}, comp.Resources[1:]...)
if d2, _ := ComposedDigestOf(shuffled); d2 != d1 {
t.Errorf("the same resources in another order and spacing digest %s and %s", d1, d2)
}
changed := comp
changed.Resources = append([]json.RawMessage{json.RawMessage(strings.Replace(canonical(t, comp.Resources[0]), `"id":"`, `"id":"x`, 1))}, comp.Resources[1:]...)
if d3, _ := ComposedDigestOf(changed); d3 == d1 {
t.Errorf("a changed resource kept its digest")
}
revised := comp
revised.Data = !comp.Data
if d4, _ := ComposedDigestOf(revised); d4 == d1 {
t.Errorf("a changed data flag kept its digest")
}
}
// 1943/AC2: a file over 64 KiB is carried by digest and size, its content handed back to be put in the node's object
// bucket; a composition that carries such a file whole is refused.
func TestAComposedFileOver64KiBIsCarriedByDigest(t *testing.T) {
c := readCaptured(t, "g14")
split := splitCaptured(t, "g14", c)
var comp Composed
var at int
for _, v := range split {
for i, r := range v.Resources {
var f map[string]any
_ = json.Unmarshal(r, &f)
if f["type"] == "file" && v.Module != "" && f["bytes"] == nil && f["into"] == nil {
comp, at = v, i
break
}
}
if comp.Module != "" {
break
}
}
var f map[string]any
_ = json.Unmarshal(comp.Resources[at], &f)
// Grown past the threshold from a captured file: no captured file of this node is over 64 KiB.
body := strings.Repeat("x", declaration.FileReferenceThreshold+1)
f["content"] = body
big, _ := json.Marshal(f)
comp.Resources = append([]json.RawMessage{}, comp.Resources...)
comp.Resources[at] = big
if _, err := EncodeComposed(comp); err == nil || !strings.Contains(err.Error(), "by digest") {
t.Fatalf("a file over 64 KiB carried whole = %v, want refused", err)
}
byDigest, objects, err := comp.ByDigest()
if err != nil {
t.Fatal(err)
}
var ref map[string]any
_ = json.Unmarshal(byDigest.Resources[at], &ref)
digest := declaration.DigestOf([]byte(body))
if ref["digest"] != digest || ref["size"] != float64(len(body)) || ref["content"] != nil {
t.Errorf("the file became %v, want digest %s and size %d", ref, digest, len(body))
}
if string(objects[digest]) != body || len(objects) != 1 {
t.Errorf("the objects handed back are %d, want the file's content under its digest", len(objects))
}
if _, err := EncodeComposed(byDigest); err != nil {
t.Errorf("the composition by digest is refused: %v", err)
}
}
// 1943/AC2: an unknown version is refused as one; a module's composition carrying the node's own parts, a resource of
// another module, a resource its kind refuses, or no fingerprint, is refused, naming why.
func TestComposedRefusals(t *testing.T) {
fp := fp2of('a')
file := json.RawMessage(`{"id":"m.f","type":"file","module":"m","path":"/etc/m","content":"x"}`)
ok := Composed{Node: "ace", Module: "m", FP: fp, Resources: []json.RawMessage{file}}
raw, err := EncodeComposed(ok)
if err != nil {
t.Fatal(err)
}
if _, err := DecodeComposed(bytes.Replace(raw, []byte(`"v":1`), []byte(`"v":2`), 1)); !errors.Is(err, ErrUnknownVersion) {
t.Errorf("version 2 = %v, want ErrUnknownVersion", err)
}
refusals := []struct {
c Composed
why string
}{
{Composed{Node: "ace", Module: "m", Resources: []json.RawMessage{file}}, "fingerprint"},
{Composed{Node: "", Module: "m", FP: fp, Resources: []json.RawMessage{file}}, "node name is empty"},
{Composed{Node: "ace", Module: "other", FP: fp, Resources: []json.RawMessage{file}}, `of "m"`},
{Composed{Node: "ace", Module: "m", FP: fp, Resources: []json.RawMessage{file}, Own: NodeOwn{Provisioner: "p"}}, "node's own"},
{Composed{Node: "ace", FP: fp, Resources: []json.RawMessage{}}, "names no fingerprint"},
{Composed{Node: "ace", Resources: []json.RawMessage{file}}, `of "m"`},
{Composed{Node: "ace", Module: "m", FP: fp, Resources: []json.RawMessage{json.RawMessage(`{"id":"m.f","type":"file","module":"m","path":"/etc/m","content":"x","image":"y"}`)}}, `"image"`},
{Composed{Node: "ace", Module: "m", FP: fp, Resources: []json.RawMessage{json.RawMessage(`{"id":"m.a","type":"action","module":"m"}`)}}, "action"},
}
for _, r := range refusals {
if _, err := EncodeComposed(r.c); err == nil || !strings.Contains(err.Error(), r.why) {
t.Errorf("EncodeComposed(%s/%s) = %v, want a refusal saying %q", r.c.Node, r.c.Module, err, r.why)
}
}
}
// 1943/AC2 (issue 1150): a module whose name holds a dot keys its have, its want and its composition escaped, and the
// parsers give its name back; a name with a wildcard, whitespace or an escape of its own is still refused.
func TestADottedModuleKeysEscaped(t *testing.T) {
fp := fp2of('f')
have, err := HaveKey("ace", "novox.be")
if err != nil || have != "ace.novox=2ebe" {
t.Fatalf("HaveKey = %q, %v", have, err)
}
if n, m, isNode, err := ParseHaveKey(have); err != nil || n != "ace" || m != "novox.be" || isNode {
t.Errorf("ParseHaveKey(%q) = %q, %q, %v, %v", have, n, m, isNode, err)
}
want, err := WantKey("ace", "novox.be")
if err != nil || want != "want.ace.novox=2ebe" {
t.Fatalf("WantKey = %q, %v", want, err)
}
if n, m, err := ParseWantKey(want); err != nil || n != "ace" || m != "novox.be" {
t.Errorf("ParseWantKey(%q) = %q, %q, %v", want, n, m, err)
}
comp, err := ComposedKey("ace", "novox.be", fp)
if err != nil || comp != "ace.novox=2ebe."+strings.Repeat("f", 64) {
t.Fatalf("ComposedKey = %q, %v", comp, err)
}
if n, m, back, _, err := ParseComposedKey(comp); err != nil || n != "ace" || m != "novox.be" || back != fp {
t.Errorf("ParseComposedKey(%q) = %q, %q, %q, %v", comp, n, m, back, err)
}
for _, bad := range []string{"m*", "m x", "m=2e", "m>", ".m", "m.", "a..b"} {
if _, err := HaveKey("ace", bad); err == nil {
t.Errorf("HaveKey(ace, %q) accepted", bad)
}
}
for _, bad := range []string{"ace.m=2", "ace.m=2E", "ace.m=41", "ace.=6eode"} {
if _, _, _, err := ParseHaveKey(bad); err == nil {
t.Errorf("ParseHaveKey(%q) accepted", bad)
}
}
}
// The node's own composition carries the filled paths of the files moved into it, the files of a module with no
// fingerprint of the new form: the provisioner fills them all the same (issue 1945, live on 2026-10-25: the paths were
// dropped and the node-engine wrote those files empty). It still holds no module's data.
func TestTheNodesOwnCompositionCarriesTheFilledPathsOfItsFiles(t *testing.T) {
filled := json.RawMessage(`{"id":"power.unit","type":"file","path":"/etc/systemd/system/p.service","mode":"0644","owner-at-runtime":"owned"}`)
own := Composed{Node: "ace", Resources: []json.RawMessage{filled}, Filled: []string{"/etc/systemd/system/p.service"},
Own: NodeOwn{Provisioner: "mesh-provisioner-client"}}
raw, err := EncodeComposed(own)
if err != nil {
t.Fatalf("the node's own composition with a filled file: %v", err)
}
back, err := DecodeComposed(raw)
if err != nil || len(back.Filled) != 1 {
t.Fatalf("read back: %v %v", back.Filled, err)
}
if _, err := EncodeComposed(Composed{Node: "ace", Data: true, Resources: []json.RawMessage{filled},
Filled: own.Filled}); err == nil {
t.Error("the node's own composition was let hold a module's data")
}
}
// A file the provisioner fills says no content: in a composition that does not say it is filled, the node-engine would
// write it empty. Such a composition is refused, in the node's own as in a module's (issue 1945).
func TestAFileWithNoContentThatIsNotFilledIsRefused(t *testing.T) {
fp := fp2of('a')
for _, c := range []Composed{
{Node: "ace", Resources: []json.RawMessage{json.RawMessage(
`{"id":"power.unit","type":"file","path":"/etc/p","mode":"0644","owner-at-runtime":"owned"}`)}},
{Node: "ace", Module: "m", FP: fp, Resources: []json.RawMessage{json.RawMessage(
`{"id":"m.f","type":"file","module":"m","path":"/etc/m","mode":"0644","owner-at-runtime":"owned"}`)}},
} {
if _, err := EncodeComposed(c); err == nil || !strings.Contains(err.Error(), "empty") {
t.Errorf("%s/%s: %v, want a refusal saying it would be written empty", c.Node, c.Module, err)
}
}
// A file that says its content, even an empty one, is written as it says.
if _, err := EncodeComposed(Composed{Node: "ace", Resources: []json.RawMessage{json.RawMessage(
`{"id":"x.f","type":"file","path":"/etc/x","mode":"0644","content":""}`)}}); err != nil {
t.Errorf("a file that says it is empty: %v", err)
}
}
+72 -22
View File
@@ -11,7 +11,8 @@
// `node`. A presence key is `<node>` for a node and `part.<part>.<instance>` for a part's instance, so a node may
// not be named `part`. A pulse key is `<bucket>.<key>`: the full name of the bucket watched, then that bucket's key,
// which may hold dots, so it splits at the first dot. A name is refused when it is empty or holds a dot, a wildcard,
// whitespace or a character a bus key refuses, and the error names it.
// whitespace or a character a bus key refuses, and the error names it; but a module's name may hold dots between its
// parts (`novox.be`), and keys with each dot escaped as EscapeName writes it (`novox=2ebe`, issue 1150).
//
// **Values** are JSON of under about 500 bytes, each carrying its version in `v`. A reader refuses a value whose
// version is missing or one it does not know, with an error that is ErrUnknownVersion (research 063 section 05
@@ -35,7 +36,7 @@
// builder's build keys (builder.go) and the checker's verdicts and kept verdicts (checker.go). A manifests value is the
// one exception to the size above: a manifest is up to about 21 KB, whole, byte for byte.
//
// Standard library only, and this module's own asks package.
// Standard library only, and this module's own asks and declaration packages (a composition holds declaration resources).
package delivery
import (
@@ -89,6 +90,10 @@ const (
// for which it had written a have key; the key is never deleted (issue 1369, ADR 0408 §4), so a reader can tell
// that the module stopped.
PhaseRemoved Phase = "removed"
// PhaseHeld is a module whose backup before an apply failed: it keeps running what it has, nothing of the new
// composition is applied, and the node waits for the answer in its want (novox/hq ADR 0539 clause 16). Detail
// carries the backup's own words.
PhaseHeld Phase = "held"
)
// Health is what the module's own check says.
@@ -164,13 +169,45 @@ func HaveKey(node, module string) (string, error) {
if err := checkName("node", node); err != nil {
return "", err
}
if err := checkName("module", module); err != nil {
part, err := moduleKeyPart(module)
if err != nil {
return "", err
}
if module == "node" {
return "", fmt.Errorf(`delivery: the module name "node" is refused: %s.node is the node's own key`, node)
}
return node + "." + module, nil
return node + "." + part, nil
}
var moduleNameOK = regexp.MustCompile(`^[-/_a-zA-Z0-9]+(\.[-/_a-zA-Z0-9]+)*$`)
// moduleKeyPart is a module's name as a key part (issue 1150): a name may hold a dot (`novox.be`), which a key part
// may not, so a dot is escaped as EscapeName writes it (`novox=2ebe`); a name of letters, digits, `-`, `/` and `_` keys
// as it is. A name that is empty, or holds a wildcard, whitespace, `=`, or an empty part between dots, is refused.
func moduleKeyPart(module string) (string, error) {
if module == "" {
return "", fmt.Errorf("delivery: the module name is empty")
}
if !moduleNameOK.MatchString(module) {
return "", fmt.Errorf("delivery: the module name %q holds a wildcard, whitespace, '=', an empty part between dots or a character a key refuses", module)
}
return EscapeName(module), nil
}
// parseModuleKeyPart is the inverse of moduleKeyPart; it refuses a part that is not the one moduleKeyPart writes for
// its name (an escape of a character that keys as it is, an upper-case escape, an escape of a refused character).
func parseModuleKeyPart(part string) (string, error) {
if err := checkName("module", part); err != nil {
return "", err
}
module, err := UnescapeName(part)
if err != nil {
return "", err
}
if back, err := moduleKeyPart(module); err != nil || back != part {
return "", fmt.Errorf("delivery: the module part %q is not a module name as a key writes it", part)
}
return module, nil
}
// HaveNodeKey is the key, in HaveBucket, of a node's own facts: `<node>.node`.
@@ -190,12 +227,13 @@ func ParseHaveKey(key string) (node, module string, isNode bool, err error) {
if err := checkName("node", node); err != nil {
return "", "", false, fmt.Errorf("delivery: %q is not a have key: %w", key, err)
}
if err := checkName("module", module); err != nil {
return "", "", false, fmt.Errorf("delivery: %q is not a have key: %w", key, err)
}
if module == "node" {
return node, "", true, nil
}
module, err = parseModuleKeyPart(module)
if err != nil {
return "", "", false, fmt.Errorf("delivery: %q is not a have key: %w", key, err)
}
return node, module, false, nil
}
@@ -282,24 +320,31 @@ func PulseTTL(moveAge time.Duration) (ttl time.Duration, stalled bool) {
// Have is a module's state on a node, written by that node's node-engine under HaveKey. FP is the module's
// fingerprint once the indexer exists (design 47 release 3); until then Build is the build the node was sent, and
// the view compares by build.
//
// Composed is the digest of the composition the node runs for the module (novox/hq ADR 0539 clause 9): the
// controller judges a move against it (clause 14), and the assigner reads it to know a want was applied.
type Have struct {
V int `json:"v"`
FP string `json:"fp,omitempty"`
Build string `json:"build,omitempty"`
Phase Phase `json:"phase"`
Health Health `json:"health"`
Detail string `json:"detail,omitempty"`
V int `json:"v"`
FP string `json:"fp,omitempty"`
Composed string `json:"composed,omitempty"`
Build string `json:"build,omitempty"`
Phase Phase `json:"phase"`
Health Health `json:"health"`
Detail string `json:"detail,omitempty"`
}
// HaveNode is a node's own facts, written under HaveNodeKey. Each field is the node-engine's own JSON, carried whole:
// this package names the facts and does not type them.
//
// Composed is the digest of the node's own composition it runs (ADR 0539 clause 9).
type HaveNode struct {
V int `json:"v"`
Windows json.RawMessage `json:"windows,omitempty"`
Busy json.RawMessage `json:"busy,omitempty"`
Strays json.RawMessage `json:"strays,omitempty"`
Filters json.RawMessage `json:"filters,omitempty"`
Profile json.RawMessage `json:"profile,omitempty"`
V int `json:"v"`
Composed string `json:"composed,omitempty"`
Windows json.RawMessage `json:"windows,omitempty"`
Busy json.RawMessage `json:"busy,omitempty"`
Strays json.RawMessage `json:"strays,omitempty"`
Filters json.RawMessage `json:"filters,omitempty"`
Profile json.RawMessage `json:"profile,omitempty"`
}
// Presence is whether a node or a part's instance is online; the entry's own time says since when. The watchman
@@ -347,9 +392,14 @@ func decode(data []byte, into any) error {
func (h Have) check() error {
switch h.Phase {
case PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed, PhaseRemoved:
case PhaseBackingUp, PhaseApplying, PhaseChecking, PhaseRunning, PhaseFailed, PhaseRemoved, PhaseHeld:
default:
return fmt.Errorf("delivery: phase %q is not backing-up, applying, checking, running, failed or removed", h.Phase)
return fmt.Errorf("delivery: phase %q is not backing-up, applying, checking, running, failed, removed or held", h.Phase)
}
if h.Composed != "" {
if err := checkDigest(h.Composed, "the have's composition digest"); err != nil {
return err
}
}
switch h.Health {
case HealthOK, HealthFailed, HealthWaits, HealthUnknown:
@@ -424,7 +474,7 @@ func SameHave(a, b Have) bool { return a == b }
// SameHaveNode says whether two node facts are the same value, comparing each carried JSON without its spacing.
func SameHaveNode(a, b HaveNode) bool {
return a.V == b.V && sameJSON(a.Windows, b.Windows) && sameJSON(a.Busy, b.Busy) &&
return a.V == b.V && a.Composed == b.Composed && sameJSON(a.Windows, b.Windows) && sameJSON(a.Busy, b.Busy) &&
sameJSON(a.Strays, b.Strays) && sameJSON(a.Filters, b.Filters) && sameJSON(a.Profile, b.Profile)
}
+104
View File
@@ -0,0 +1,104 @@
package delivery
import (
"encoding/json"
"fmt"
"sort"
"time"
)
// Whether the artifact store still holds what a done key names (novox/hq issue 1937). A build module pushes a bundle or
// an archive as a bare blob (ADR 0498 §1), and the store's nightly collector deletes every blob no manifest names; only
// the controller's take-in and sweep hold one by its manifest. A done key the controller has not taken in can therefore
// name a blob that is gone, and reusing it sends a definition every machine fails on. So the builder asks before it
// reuses a done key, and the controller says when a take-in found a gone blob.
const (
// HeldVerb is the controller seat's verb the builder asks, as a module's invokes names it: references in, each
// answered held or gone.
HeldVerb = "seat:mesh-controller.held"
// ArtifactsGoneEvent is the controller seat's event said when a take-in is refused because the store does not hold
// an artifact the key names: as a consumer names it, `mesh-controller.artifacts-gone`.
ArtifactsGoneEvent = "artifacts-gone"
// ArtifactsGoneSubject is where it is published (the seat's event subject).
ArtifactsGoneSubject = "mesh.seat.mesh-controller.event." + ArtifactsGoneEvent
)
// HeldAnswer is the answer of HeldVerb: the references the store holds, and those it answered 404 for. A reference the
// store could not be asked about is in neither, and named in Unknown with why.
type HeldAnswer struct {
Store string `json:"store,omitempty"`
Held []string `json:"held"`
Gone []string `json:"gone"`
Unknown map[string]string `json:"unknown,omitempty"`
}
// DecodeHeldAnswer reads HeldVerb's answer: the verb answers {output, ok, answer}, the answer being HeldAnswer. A
// reference asked that the answer names in none of held, gone and unknown is an error: an answer that leaves one out
// would let a key with a gone blob be reused.
func DecodeHeldAnswer(raw []byte, asked []string) (HeldAnswer, error) {
var wrapped struct {
Output string `json:"output"`
OK bool `json:"ok"`
Answer json.RawMessage `json:"answer"`
}
if err := json.Unmarshal(raw, &wrapped); err != nil {
return HeldAnswer{}, fmt.Errorf("the answer of %s is not {output, ok, answer}: %w", HeldVerb, err)
}
if !wrapped.OK || len(wrapped.Answer) == 0 {
return HeldAnswer{}, fmt.Errorf("%s did not answer: %s", HeldVerb, wrapped.Output)
}
var a HeldAnswer
if err := json.Unmarshal(wrapped.Answer, &a); err != nil {
return HeldAnswer{}, fmt.Errorf("the answer of %s is not {held, gone}: %w", HeldVerb, err)
}
named := map[string]bool{}
for _, r := range a.Held {
named[r] = true
}
for _, r := range a.Gone {
named[r] = true
}
for r := range a.Unknown {
named[r] = true
}
for _, r := range asked {
if !named[r] {
return HeldAnswer{}, fmt.Errorf("the answer of %s names nothing of %s", HeldVerb, r)
}
}
return a, nil
}
// ArtifactsGone is the body of ArtifactsGoneEvent: the module whose builder's key the controller would not take in,
// the key and its fingerprint, and every reference of it the store does not hold.
type ArtifactsGone struct {
Event string `json:"event"`
Module string `json:"module"`
Key string `json:"key"`
Fingerprint string `json:"fingerprint"`
Gone []string `json:"gone"`
At time.Time `json:"at"`
}
// EncodeArtifactsGone is the event's body, the references sorted.
func EncodeArtifactsGone(e ArtifactsGone) ([]byte, error) {
e.Event = ArtifactsGoneEvent
e.Gone = append([]string(nil), e.Gone...)
sort.Strings(e.Gone)
if e.Module == "" || e.Fingerprint == "" || len(e.Gone) == 0 {
return nil, fmt.Errorf("delivery: %s names a module, a fingerprint and at least one gone reference", ArtifactsGoneEvent)
}
return json.Marshal(e)
}
// DecodeArtifactsGone reads the event's body, refusing one that names no module, fingerprint or reference.
func DecodeArtifactsGone(raw []byte) (ArtifactsGone, error) {
var e ArtifactsGone
if err := json.Unmarshal(raw, &e); err != nil {
return ArtifactsGone{}, fmt.Errorf("delivery: %s is not one: %w", ArtifactsGoneEvent, err)
}
if e.Event != ArtifactsGoneEvent || e.Module == "" || e.Fingerprint == "" || len(e.Gone) == 0 {
return ArtifactsGone{}, fmt.Errorf("delivery: %s names no module, fingerprint or gone reference: %s", ArtifactsGoneEvent, raw)
}
return e, nil
}
+84
View File
@@ -0,0 +1,84 @@
package delivery
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
const (
refA = "artifact-store://node-tools/runtime/blobs/sha256:a46b5d51a30e8fe71107d75ceb46e3187022fe51f551d0498b8be817f22435cb"
refB = "artifact-store://mesh-host/host-arch/blobs/sha256:8a752f914f3523904a5442cae37ce3cb1ad297d6789b0f871585bce4d0c2ea00"
)
// 1937/AC1 (the contract): the controller's answer reads into held and gone, and an answer leaving a reference out is refused.
func TestDecodeHeldAnswer(t *testing.T) {
raw := []byte(`{"output":"…","ok":true,"answer":{"store":"s:5100","held":["` + refA + `"],"gone":["` + refB + `"]}}`)
a, err := DecodeHeldAnswer(raw, []string{refA, refB})
if err != nil {
t.Fatal(err)
}
if len(a.Held) != 1 || a.Held[0] != refA || len(a.Gone) != 1 || a.Gone[0] != refB {
t.Fatalf("read %+v", a)
}
if _, err := DecodeHeldAnswer(raw, []string{refA, refB, "artifact-store://x/y/blobs/sha256:00"}); err == nil ||
!strings.Contains(err.Error(), "names nothing of") {
t.Fatalf("an answer leaving a reference out was taken: %v", err)
}
if _, err := DecodeHeldAnswer([]byte(`{"output":"the store is down","ok":false}`), nil); err == nil ||
!strings.Contains(err.Error(), "the store is down") {
t.Fatalf("a failed verb was taken: %v", err)
}
}
// 1937/AC2 (the contract): the event's body names the module, the fingerprint and the gone references, sorted.
func TestArtifactsGoneRoundTrip(t *testing.T) {
raw, err := EncodeArtifactsGone(ArtifactsGone{Module: "node-tools", Key: "node-tools.75", Fingerprint: "src2:75",
Gone: []string{refB, refA}, At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
e, err := DecodeArtifactsGone(raw)
if err != nil {
t.Fatal(err)
}
if e.Event != ArtifactsGoneEvent || e.Module != "node-tools" || len(e.Gone) != 2 || e.Gone[0] != refB {
t.Fatalf("read %+v", e)
}
if _, err := EncodeArtifactsGone(ArtifactsGone{Module: "m", Fingerprint: "src2:1"}); err == nil {
t.Fatal("an event naming no gone reference was made")
}
if _, err := DecodeArtifactsGone([]byte(`{"event":"artifacts-gone","module":"m"}`)); err == nil {
t.Fatal("an event naming no fingerprint was read")
}
}
// The verb and the event are on the controller seat's contract under the names the builder uses.
func TestHeldOnTheControllerSeat(t *testing.T) {
if HeldVerb != "seat:"+catalogue.ControllerSeatName+".held" {
t.Fatalf("HeldVerb %q", HeldVerb)
}
found := false
for _, v := range catalogue.ControllerVerbs {
if v.Name == "held" {
found = true
}
}
if !found {
t.Fatal("the controller seat declares no verb held")
}
for _, s := range catalogue.DefaultSeats() {
if s.Name != catalogue.ControllerSeatName {
continue
}
for _, e := range s.Emits {
if e == ArtifactsGoneEvent {
return
}
}
t.Fatalf("the controller seat emits %v, not %s", s.Emits, ArtifactsGoneEvent)
}
t.Fatal("no controller seat")
}
+11
View File
@@ -269,6 +269,10 @@ type CandidatePull struct {
Head
Modules []string `json:"modules"`
Fps map[string]CandidateModule `json:"fps,omitempty"`
// Merged says the pull request merged while another member of its group is still open: it stays in the candidate
// with the head it merged at, so the heads hash an Approve covers holds until every member merged, and the checker
// judges the members still open by the verdict the group had (issue 1791). A reader of the older shape ignores it.
Merged bool `json:"merged,omitempty"`
}
// CandidateNoneNoModule is the None2 of a candidate's module whose directory the merge preview names no manifest for:
@@ -346,6 +350,13 @@ func (c Candidate) check() error {
if len(c.Pulls) == 0 {
return fmt.Errorf("delivery: a candidate names no pull request")
}
open := false
for _, p := range c.Pulls {
open = open || !p.Merged
}
if !open {
return fmt.Errorf("delivery: every pull request of the candidate merged: the group is no candidate any more")
}
heads := make([]Head, len(c.Pulls))
for i, p := range c.Pulls {
heads[i] = p.Head
+76 -8
View File
@@ -1,8 +1,11 @@
package delivery
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"path"
"regexp"
"sort"
"strings"
@@ -154,8 +157,8 @@ func checkAskID(id string) error {
}
// StepAsk is the message on a build seat's stream: who asked what, of which kind and language, and the one whole job.
// Exactly one of Build and Check is set: a delivery ask is a build; a check ask a build (a candidate's fingerprint) or a
// check; a recheck ask a check; an image ask is always a build.
// Exactly one of Build, Check and Compose is set: a delivery ask is a build; a check ask a build (a candidate's
// fingerprint), a check or, on the Go seat, a compose; a recheck ask a check; an image ask is always a build.
type StepAsk struct {
V int `json:"v"`
ID string `json:"id"`
@@ -163,10 +166,48 @@ type StepAsk struct {
Language string `json:"language"`
// Asker is the module that put the ask (mesh-delivery-builder or mesh-delivery-checker); the stream's grant is what
// makes it true, this field only names it to a reader.
Asker string `json:"asker"`
At time.Time `json:"at"`
Build *BuildJob `json:"build,omitempty"`
Check *CheckJob `json:"check,omitempty"`
Asker string `json:"asker"`
At time.Time `json:"at"`
Build *BuildJob `json:"build,omitempty"`
Check *CheckJob `json:"check,omitempty"`
Compose *ComposeJob `json:"compose,omitempty"`
}
// ComposeJob is a compose ask's job (novox/hq ADR 0482 clause 4, issue 1622): fetch the commit, build the composer
// command (Package) in the Go module at Dir of the fetched tree, once per ComposerKey, run it in a run step shut off
// with Request on its standard input, and answer its standard output whole (ComposeAnswer). The assigner asks it, to
// compose a group with its mesh-sdk member's head, and to read the group's manifests with the mesh-sdk the running
// controller and node-engine pin (ADR 0482 clause 5, issue 1623).
type ComposeJob struct {
// Fetch is the fetch tool's arguments: the repository and the full commit whose mesh-sdk composes (mesh-sdk itself
// at a head, or a module that requires mesh-sdk, at the version it pins).
Fetch map[string]any `json:"fetch"`
// Dir is the directory of the Go module the composer is built in, relative to the repository's root ("go" for
// mesh-sdk, RootPath for a module at the root); Package is the composer's import path (mesh-sdk composer.Package).
Dir string `json:"dir"`
Package string `json:"package"`
// Request is the composer's request (mesh-sdk composer.Request), its manifests named by their digests in the
// indexer's ManifestsBucket: the build module fills them before the run (composer.Fill).
Request json.RawMessage `json:"request"`
// Reader names, for a reader of the ask, whose composition this is: the mesh-sdk head of a group, or a module whose
// mesh-sdk reads the group's manifests (mesh-controller, mesh-host).
Reader string `json:"reader,omitempty"`
}
// ComposeAnswer is a compose ask's step result: whether this run built the composer or took the one kept under its
// key, the key, and the composer's verdict (mesh-sdk composer.Verdict) whole.
type ComposeAnswer struct {
Built bool `json:"built"`
Key string `json:"key"`
Verdict json.RawMessage `json:"verdict"`
}
// ComposerKey is the key a build module keeps a built composer under: the repository, commit, directory and package it
// was built from and the toolchain it was built with, hashed (`composer-` and 16 hex digits), so each head is built once
// on a node (ADR 0482 clause 4).
func ComposerKey(repository, commit, dir, pkg string, toolchain ToolchainUsed) string {
sum := sha256.Sum256([]byte(strings.Join([]string{repository, commit, dir, pkg, toolchain.Version, toolchain.Recipe}, "\n")))
return "composer-" + hex.EncodeToString(sum[:8])
}
// BuildJob is a build ask's job: fetch the commit (unless every artifact is a copy), then build or copy each artifact,
@@ -232,9 +273,17 @@ func (a StepAsk) check() error {
if a.At.IsZero() {
return fmt.Errorf("delivery: the ask %s names no time", a.ID)
}
jobs := 0
for _, set := range []bool{a.Build != nil, a.Check != nil, a.Compose != nil} {
if set {
jobs++
}
}
switch {
case (a.Build == nil) == (a.Check == nil):
return fmt.Errorf("delivery: the ask %s holds a build or a check, exactly one", a.ID)
case jobs != 1:
return fmt.Errorf("delivery: the ask %s holds a build, a check or a compose, exactly one", a.ID)
case a.Compose != nil && (a.Kind != AskCheck || a.Language != AskGo):
return fmt.Errorf("delivery: the ask %s is a compose, which the Go seat's check stream alone carries", a.ID)
case a.Build != nil && a.Kind == AskRecheck:
return fmt.Errorf("delivery: the ask %s is a build on the recheck stream, which carries checks alone", a.ID)
case a.Check != nil && a.Kind == AskDelivery:
@@ -245,9 +294,28 @@ func (a StepAsk) check() error {
if a.Build != nil {
return a.Build.check(a.ID)
}
if a.Compose != nil {
return a.Compose.check(a.ID)
}
return a.Check.check(a.ID)
}
func (c ComposeJob) check(id string) error {
if err := checkFetchArgs(id, c.Fetch); err != nil {
return err
}
if c.Dir == "" || c.Dir != path.Clean(c.Dir) || strings.HasPrefix(c.Dir, "/") || c.Dir == ".." || strings.HasPrefix(c.Dir, "../") {
return fmt.Errorf("delivery: the compose ask %s names no clean directory inside the repository (%q)", id, c.Dir)
}
if c.Package == "" {
return fmt.Errorf("delivery: the compose ask %s names no package to build", id)
}
if !json.Valid(c.Request) || len(c.Request) == 0 || c.Request[0] != '{' {
return fmt.Errorf("delivery: the compose ask %s holds no request object", id)
}
return nil
}
func (b BuildJob) check(id string) error {
if err := checkName("module", b.Module); err != nil {
return err
+76
View File
@@ -0,0 +1,76 @@
package delivery
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
)
func composeAsk() StepAsk {
return StepAsk{ID: "mh3k2x9qa1b2c5", Kind: AskCheck, Language: AskGo, Asker: "mesh-delivery-assigner",
At: time.Date(2026, 10, 24, 3, 0, 0, 0, time.UTC),
Compose: &ComposeJob{Fetch: map[string]any{"repository": "novox/mesh-sdk", "commit": askCommit},
Dir: "go", Package: "git.novox.be/novox/mesh-sdk/go/cmd/mesh-compose",
Request: json.RawMessage(`{"v":1,"cases":[{"name":"change","manifests":{"demo":{"digest":"d1"}}}]}`), Reader: "mesh-sdk head"}}
}
// 1622: a compose ask encodes and decodes with every field.
func TestAComposeAskRoundTrips(t *testing.T) {
a := composeAsk()
raw, err := EncodeStepAsk(a)
if err != nil {
t.Fatal(err)
}
got, err := DecodeStepAsk(raw)
if err != nil {
t.Fatal(err)
}
a.V = StepAskVersion
if !reflect.DeepEqual(got, a) {
t.Fatalf("round trip:\n got %+v\nwant %+v", got, a)
}
}
// 1622: a compose ask goes only on the Go seat's check stream, holds no other job beside it, fetches a full commit,
// builds a package in a clean directory of the repository and carries a request object.
func TestAComposeAskIsRefusedWhereItDoesNotBelong(t *testing.T) {
for name, change := range map[string]func(*StepAsk){
"on the delivery stream": func(a *StepAsk) { a.Kind = AskDelivery },
"on the recheck stream": func(a *StepAsk) { a.Kind = AskRecheck },
"on the Node seat": func(a *StepAsk) { a.Language = AskNode },
"beside a check": func(a *StepAsk) { a.Check = checkAsk().Check },
"with a short commit": func(a *StepAsk) { a.Compose.Fetch["commit"] = "0123456" },
"in no directory": func(a *StepAsk) { a.Compose.Dir = "" },
"outside the repository": func(a *StepAsk) { a.Compose.Dir = "../x" },
"with no package": func(a *StepAsk) { a.Compose.Package = "" },
"with no request": func(a *StepAsk) { a.Compose.Request = nil },
"with a request not JSON": func(a *StepAsk) { a.Compose.Request = json.RawMessage(`[1]`) },
} {
a := composeAsk()
change(&a)
if _, err := EncodeStepAsk(a); err == nil {
t.Errorf("a compose ask %s is encoded", name)
}
}
}
// 1622/AC3: a composer is kept under one key per repository, commit, directory, package and toolchain, so two asks of
// one head build it once on a node, and another head or toolchain builds its own.
func TestAComposerIsKeptByWhatItWasBuiltFrom(t *testing.T) {
tc := ToolchainUsed{Version: "go1.26.0", Recipe: "3"}
k := ComposerKey("novox/mesh-sdk", askCommit, "go", "p", tc)
if k != ComposerKey("novox/mesh-sdk", askCommit, "go", "p", tc) || !strings.HasPrefix(k, "composer-") || len(k) != len("composer-")+16 {
t.Fatalf("key = %q", k)
}
for _, other := range []string{
ComposerKey("novox/mesh-sdk", strings.Repeat("1", 40), "go", "p", tc),
ComposerKey("novox/mesh-controller", askCommit, ".", "p", tc),
ComposerKey("novox/mesh-sdk", askCommit, "go", "p", ToolchainUsed{Version: "go1.26.1", Recipe: "3"}),
} {
if other == k {
t.Errorf("two composers built from different inputs share the key %s", k)
}
}
}
Binary file not shown.
Binary file not shown.
+64 -10
View File
@@ -32,19 +32,27 @@ const (
AskKindHold = "hold"
// AskKindRestore is the kind of the assigner's ask about a data module set back.
AskKindRestore = "restore"
// AskKindBackup is the kind of the assigner's ask about a backup before an apply that failed: back up again, or go
// on without one (novox/hq ADR 0539 clause 16).
AskKindBackup = "backup"
// AskKindRecord is the kind of the assigner's ask about a move a person must allow, for a module of policy record:
// a composition judged to recreate a container or move data, or one that cannot be judged (ADR 0539 clause 14).
AskKindRecord = "record"
viewWord = "view"
)
// AskKey is an ask's key read whole: its Kind, and the parts its kind has: a merge's Group and heads Hash, a hold's
// Module and fingerprint FP, a restore's Node, Module and FP.
// Module and fingerprint FP, a restore's and a backup's Node, Module and FP, a record's Node, Module and the digest
// Composed of the composition it would allow.
type AskKey struct {
Kind string
Group string
Hash string
Node string
Module string
FP string
Kind string
Group string
Hash string
Node string
Module string
FP string
Composed string
}
// AskNeedsProof says whether an ask of this kind must prove who answered it, so the view may not answer it: the
@@ -53,10 +61,11 @@ func AskNeedsProof(kind string) (bool, error) {
switch kind {
case AskKindMerge:
return true, nil
case AskKindHold, AskKindRestore:
case AskKindHold, AskKindRestore, AskKindBackup, AskKindRecord:
return false, nil
}
return false, fmt.Errorf("delivery: %q is not a kind of ask, %s, %s or %s", kind, AskKindMerge, AskKindHold, AskKindRestore)
return false, fmt.Errorf("delivery: %q is not a kind of ask, %s, %s, %s, %s or %s", kind, AskKindMerge, AskKindHold,
AskKindRestore, AskKindBackup, AskKindRecord)
}
// HoldAskKey is the key, in AssignerAsksBucket, of the ask about a module's held fingerprint: `hold.<module>.<hex>`.
@@ -81,7 +90,32 @@ func RestoreAskKey(node, module, fingerprint string) (string, error) {
return AskKindRestore + "." + node + "." + key, nil
}
// ParseAnyAskKey reads an ask key of any kind (MergeAskKey, HoldAskKey, RestoreAskKey); it refuses an unknown kind
// BackupAskKey is the key, in AssignerAsksBucket, of the ask about a module's failed backup on a node before it applies
// a fingerprint: `backup.<node>.<module>.<hex>`.
func BackupAskKey(node, module, fingerprint string) (string, error) {
key, err := ComposedKey(node, module, fingerprint)
if err != nil {
return "", err
}
return AskKindBackup + "." + key, nil
}
// RecordAskKey is the key, in AssignerAsksBucket, of the ask about a move of a module on a node to the composition
// whose digest it names: `record.<node>.<module>.<digest hex>`, so a new revision at the same fingerprint is asked
// again (ADR 0539 clause 14).
func RecordAskKey(node, module, composed string) (string, error) {
have, err := HaveKey(node, module)
if err != nil {
return "", err
}
if err := checkDigest(composed, "a record ask's composition digest"); err != nil {
return "", err
}
return AskKindRecord + "." + have + "." + strings.TrimPrefix(composed, "sha256:"), nil
}
// ParseAnyAskKey reads an ask key of any kind (MergeAskKey, HoldAskKey, RestoreAskKey, BackupAskKey, RecordAskKey); it
// refuses an unknown kind
// and a key whose parts do not fit its kind, naming the key.
func ParseAnyAskKey(key string) (AskKey, error) {
kind, rest, _ := strings.Cut(key, ".")
@@ -114,6 +148,26 @@ func ParseAnyAskKey(key string) (AskKey, error) {
return refused(err)
}
return AskKey{Kind: kind, Node: node, Module: module, FP: fp}, nil
case AskKindBackup:
node, module, fp, isNode, err := ParseComposedKey(rest)
if err != nil || isNode {
return refused(fmt.Errorf("a backup ask is backup.<node>.<module>.<fingerprint hex>"))
}
return AskKey{Kind: kind, Node: node, Module: module, FP: fp}, nil
case AskKindRecord:
parts := strings.Split(rest, ".")
if len(parts) != 3 {
return refused(fmt.Errorf("a record ask is record.<node>.<module>.<digest hex>"))
}
node, module, isNode, err := ParseHaveKey(parts[0] + "." + parts[1])
if err != nil || isNode {
return refused(fmt.Errorf("a record ask is record.<node>.<module>.<digest hex>"))
}
composed := "sha256:" + parts[2]
if err := checkDigest(composed, "a record ask's composition digest"); err != nil {
return refused(err)
}
return AskKey{Kind: kind, Node: node, Module: module, Composed: composed}, nil
}
_, err := AskNeedsProof(kind)
return refused(err)
+176
View File
@@ -0,0 +1,176 @@
package delivery
import (
"strings"
"testing"
)
func hashOf(c byte) string { return "sha256:" + digestOf(c) }
// 1943/AC3: a want naming its fingerprint and the digest of the composition it allows round-trips, and so does one
// carrying the answer to a failed backup; a want naming a fingerprint without a digest, a digest not of the sha256
// form, a removed want naming a digest, an unknown backup answer and a rule past the seventh are refused.
func TestAWantNamesItsComposition(t *testing.T) {
for _, w := range []Want{
{FP: fp2of('a'), Composed: hashOf('1')},
{FP: fp2of('a'), Composed: hashOf('1'), Backup: BackupAgain},
{FP: fp2of('a'), Composed: hashOf('1'), Backup: BackupSkip, Words: "the operator: go on without a backup"},
{FP: fp2of('a'), Composed: hashOf('1'), Newest: fp2of('b'), Rule: RuleComposed, Words: "no composition of b for ace yet"},
{Removed: true},
} {
raw, err := EncodeWant(w)
if err != nil {
t.Fatalf("EncodeWant(%+v): %v", w, err)
}
back, err := DecodeWant(raw)
w.V = Version
if err != nil || !SameWant(back, w) {
t.Errorf("want %+v came back %+v, %v", w, back, err)
}
}
refusals := []struct {
w Want
why string
}{
{Want{FP: fp2of('a')}, "names no composition"},
{Want{FP: fp2of('a'), Composed: "sha256:12"}, "not sha256"},
{Want{Removed: true, Composed: hashOf('1')}, "removed want names no"},
{Want{FP: fp2of('a'), Composed: hashOf('1'), Backup: "later"}, `backup answer "later"`},
{Want{Removed: true, Backup: BackupAgain}, "removed want names no"},
{Want{FP: fp2of('a'), Composed: hashOf('1'), Newest: fp2of('b'), Rule: 8}, "rule 8"},
}
for _, r := range refusals {
if _, err := EncodeWant(r.w); err == nil || !strings.Contains(err.Error(), r.why) {
t.Errorf("EncodeWant(%+v) = %v, want a refusal saying %q", r.w, err, r.why)
}
}
if _, err := DecodeWant([]byte(`{"v":1,"fp":"` + fp2of('a') + `"}`)); err == nil || !strings.Contains(err.Error(), "names no composition") {
t.Errorf("a want of release 5, naming no composition, was read: %v", err)
}
if RuleComposed != 7 {
t.Errorf("RuleComposed = %d, want the seventh rule", RuleComposed)
}
}
// 1943/AC3: the node's own want is `want.<node>.node` and round-trips through its parser, while a module named node is
// still refused a want key; its value names the digest of the node's own composition and no fingerprint.
func TestTheNodesOwnWant(t *testing.T) {
key, err := WantNodeKey("ace")
if err != nil || key != "want.ace.node" {
t.Fatalf("WantNodeKey = %q, %v", key, err)
}
if node, err := ParseWantNodeKey(key); err != nil || node != "ace" {
t.Errorf("ParseWantNodeKey(%q) = %q, %v", key, node, err)
}
if _, err := WantKey("ace", "node"); err == nil {
t.Errorf(`WantKey(ace, "node") accepted`)
}
if _, _, err := ParseWantKey(key); err == nil {
t.Errorf("ParseWantKey read the node's own want as a module's")
}
for _, bad := range []string{"want.ace.mesh-mail", "want.node", "ace.node", "want.a.b.node", ""} {
if _, err := ParseWantNodeKey(bad); err == nil || !strings.Contains(err.Error(), "\""+bad+"\"") {
t.Errorf("ParseWantNodeKey(%q) = %v, want a refusal naming it", bad, err)
}
}
if _, err := WantNodeKey(""); err == nil {
t.Errorf("WantNodeKey(\"\") accepted")
}
w := NodeWant{Composed: hashOf('2'), Words: "the node's own composition"}
raw, err := EncodeNodeWant(w)
if err != nil {
t.Fatal(err)
}
back, err := DecodeNodeWant(raw)
w.V = Version
if err != nil || back != w {
t.Errorf("node want came back %+v, %v", back, err)
}
for _, bad := range []NodeWant{{}, {Composed: "sha256:1"}} {
if _, err := EncodeNodeWant(bad); err == nil {
t.Errorf("EncodeNodeWant(%+v) accepted", bad)
}
}
if _, err := DecodeNodeWant([]byte(`{"v":1,"composed":"` + hashOf('2') + `","fp":"` + fp2of('a') + `"}`)); err == nil {
t.Errorf("a node want naming a fingerprint was read")
}
}
// 1943/AC3: a have held after a failed backup, naming the composition it runs, round-trips; so does the node's own have
// naming its composition; a have's digest not of the sha256 form is refused.
func TestAHaveNamesItsComposition(t *testing.T) {
for _, h := range []Have{
{FP: fp2of('a'), Composed: hashOf('3'), Phase: PhaseHeld, Health: HealthOK, Detail: "the backup failed: no space left"},
{FP: fp2of('a'), Composed: hashOf('3'), Phase: PhaseRunning, Health: HealthOK},
} {
raw, err := EncodeHave(h)
if err != nil {
t.Fatalf("EncodeHave(%+v): %v", h, err)
}
back, err := DecodeHave(raw)
h.V = Version
if err != nil || !SameHave(back, h) {
t.Errorf("have %+v came back %+v, %v", h, back, err)
}
}
if _, err := EncodeHave(Have{Composed: "sha256:x", Phase: PhaseRunning, Health: HealthOK}); err == nil || !strings.Contains(err.Error(), "not sha256") {
t.Errorf("a have naming a bad digest = %v", err)
}
n := HaveNode{Composed: hashOf('4')}
raw, err := EncodeHaveNode(n)
if err != nil {
t.Fatal(err)
}
back, err := DecodeHaveNode(raw)
n.V = Version
if err != nil || !SameHaveNode(back, n) {
t.Errorf("node have came back %+v, %v", back, err)
}
if SameHaveNode(n, HaveNode{V: Version, Composed: hashOf('5')}) {
t.Errorf("node haves naming two compositions are the same")
}
}
// 1943/AC3: the assigner's backup ask (back up again, or go on without one, ADR 0539 clause 16) and its record ask (a
// move a person must allow, clause 14) key by node, module and fingerprint or composition, round-trip through
// ParseAnyAskKey, and the view may answer both.
func TestTheBackupAndRecordAsks(t *testing.T) {
fp, digest := fp2of('a'), hashOf('6')
backup, err := BackupAskKey("ace", "mesh-mail", fp)
if err != nil || backup != "backup.ace.mesh-mail."+strings.Repeat("a", 64) {
t.Fatalf("BackupAskKey = %q, %v", backup, err)
}
record, err := RecordAskKey("ace", "mesh-mail", digest)
if err != nil || record != "record.ace.mesh-mail."+strings.Repeat("6", 64) {
t.Fatalf("RecordAskKey = %q, %v", record, err)
}
if got, err := ParseAnyAskKey(backup); err != nil || got != (AskKey{Kind: AskKindBackup, Node: "ace", Module: "mesh-mail", FP: fp}) {
t.Errorf("ParseAnyAskKey(%q) = %+v, %v", backup, got, err)
}
if got, err := ParseAnyAskKey(record); err != nil || got != (AskKey{Kind: AskKindRecord, Node: "ace", Module: "mesh-mail", Composed: digest}) {
t.Errorf("ParseAnyAskKey(%q) = %+v, %v", record, got, err)
}
for _, kind := range []string{AskKindBackup, AskKindRecord} {
if proof, err := AskNeedsProof(kind); err != nil || proof {
t.Errorf("AskNeedsProof(%s) = %v, %v; the view may answer it", kind, proof, err)
}
}
for _, k := range []string{backup, record} {
if _, err := ViewAnswerKey(k); err != nil {
t.Errorf("the view may not answer %s: %v", k, err)
}
}
for _, bad := range [][3]string{{"", "mesh-mail", fp}, {"ace", "node", fp}, {"ace", "mesh-mail", "src2:1"}} {
if _, err := BackupAskKey(bad[0], bad[1], bad[2]); err == nil {
t.Errorf("BackupAskKey(%v) accepted", bad)
}
}
if _, err := RecordAskKey("ace", "mesh-mail", "sha256:1"); err == nil {
t.Errorf("a record ask naming a bad digest was keyed")
}
for _, bad := range []string{"backup.ace.mesh-mail", "record.ace.mesh-mail.12", "record.ace." + strings.Repeat("6", 64)} {
if _, err := ParseAnyAskKey(bad); err == nil {
t.Errorf("ParseAnyAskKey(%q) accepted", bad)
}
}
}
+6 -1
View File
@@ -15,7 +15,12 @@
// `mesh/repo-check` runs; a repository missing there fails the test, never skips it. Which clones a
// check gets is chosen from the inventory: mesh-catalog by the source of the `nats` module, mesh-host
// by the source of `mesh-host`, and mesh-sdk as the controller's sibling at the commit the controller's
// go.mod pins. In a mesh where either module has no source repository nothing is
// go.mod pins. Each repository's tests read these siblings, and the checker places exactly these
// (mesh-delivery's checker, besideRules, follows this list): mesh-controller's read mesh-catalog,
// mesh-host and mesh-sdk; mesh-host's read mesh-catalog, mesh-sdk and mesh-controller (its step-nine
// test reads the controller's own manifest, novox/hq issue 1295); mesh-sdk's read mesh-catalog and
// mesh-host; mesh-catalog's read mesh-sdk and mesh-controller; mesh-delivery's read mesh-sdk and
// mesh-catalog. In a mesh where either module has no source repository nothing is
// cloned, and these tests fail loudly with "not beside this check": a cause in the setup, not in the
// change. And in a delivery group that holds a mesh-catalog pull request, these tests read the
// catalogue's main, not the group's head.
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+8 -1
View File
@@ -24,7 +24,11 @@ func Version() string {
// FromBuildInfo reads the SDK's version out of a build info. A dependency replaced by another module version is that
// version; replaced by a local path it is the path, never empty. When the SDK is the main module it is "(devel)",
// followed by a space and the vcs.revision when the build stamped one. Without the SDK in the build it is "unknown".
// followed by a space and the vcs.revision when the build stamped one. An SDK command built from another module that
// requires the SDK (`go build git.novox.be/novox/mesh-sdk/go/cmd/mesh-compose` in a reader's tree, as mesh-build-go's
// compose job builds it: novox/hq ADR 0482 clause 5) has the SDK as its main module at the version that module pins,
// read from the module cache with no vcs stamp: it is that version, never a bare "(devel)" that names no SDK. Without
// the SDK in the build it is "unknown".
func FromBuildInfo(bi *debug.BuildInfo) string {
if bi == nil {
return "unknown"
@@ -35,6 +39,9 @@ func FromBuildInfo(bi *debug.BuildInfo) string {
return "(devel) " + s.Value
}
}
if v := bi.Main.Version; v != "" && v != "(devel)" {
return v
}
return "(devel)"
}
for _, d := range bi.Deps {
+12
View File
@@ -42,6 +42,18 @@ func TestTheMainModuleIsDevelWithItsRevision(t *testing.T) {
}
}
// The shape go gives `go build <the SDK's command>` run in a module that requires the SDK (captured with go1.27 from
// mesh-compose built in a reader's tree at bc0c25e51052: "mod git.novox.be/novox/mesh-sdk/go v0.1.14-…-bc0c25e51052",
// no vcs settings): the SDK is the main module at the pinned version, which names the SDK the binary carries.
func TestAnSDKCommandBuiltFromAReaderIsThePinnedVersion(t *testing.T) {
bi := &debug.BuildInfo{Path: sdkPath + "/cmd/mesh-compose",
Main: debug.Module{Path: sdkPath, Version: "v0.1.14-0.20261024112544-bc0c25e51052"},
Settings: []debug.BuildSetting{{Key: "-trimpath", Value: "true"}, {Key: "CGO_ENABLED", Value: "0"}}}
if got := FromBuildInfo(bi); got != "v0.1.14-0.20261024112544-bc0c25e51052" {
t.Fatalf("got %q", got)
}
}
func TestNoBuildInfoIsUnknown(t *testing.T) {
if got := FromBuildInfo(nil); got != "unknown" {
t.Fatalf("got %q", got)