Compare commits

...
10 Commits
Author SHA1 Message Date
mesh-tools b9cc97470b Merge pull request 'Design 47 H1: the entry pins every requirement of the module graph; same-repository directories move the fingerprint (H-A-201, H-A-202)' (#296) from fix/47-H1-package-tree-paths into main 2026-11-20 21:37:24 +00:00
mesh-tools ae91f91e88 Merge pull request 'Design 47 H1: the catalog seat's hand-written digest helpers live in its client (H-A-200)' (#295) from fix/47-H1-contracts-steps into main 2026-11-20 21:37:22 +00:00
jschoubben 4664eb1b00 Design 47 H1: the entry pins every requirement of the module graph
The Go build module borrowed a sibling's version for a mesh package the
entry did not pin (a transitive requirement), and given no pin it now
refuses. The indexer now pins, in the entry's packages, every mesh package
of another repository that the go.mod of the module's directory, of each
Go package artifact's directory and of every same-repository module they
require transitively names. A module of the module's own repository is
never pinned, whatever a build reported: it is built from the tree.

Builds: design 47 H H-A-201, H-A-183, H-A-157, H-A-202
2026-11-20 22:30:45 +01:00
jschoubben d9a3458927 Design 47 H1: a Go build's same-repository modules move its fingerprint
A Go build resolves the same-repository modules its go.mod files require at
the zero version, transitively, from the fetched tree; a change in one of
them must move the fingerprint, whatever the module's form. The indexer
walks the go.mod graph of the module's directory and its Go package
artifacts at the take-in and appends a line per derived directory and its
git object to the tree input; the entry's paths keep the manifest's form,
and the Go build module derives the same directories itself.

Builds: design 47 H H-A-202, H-A-172, H-A-157, H-A-198
2026-11-20 22:29:15 +01:00
jschoubben b058389d46 Contracts: the catalog seat's manifest-digest helpers move to its client, out of the generated root (H-A-200)
The seat generator's check failed every unit step of the contracts mover (pipeline main.43): manifestdigest.go was
written by hand under go/seats/meshcatalog (598125a1), a generated root every file of which is the generator's. The two
helpers move to the seat's hand-written client meshcatalogclient and their callers in the builder and the indexer
follow; the generator then writes the contract as the module holds it. steprun's generator test now also holds this
repository's own contracts module in step under both roots.

Builds: design 47 G §370, G §578, H §774, H-A-200 (ADR 0608 §5)
2026-11-20 22:18:00 +01:00
jschoubben 6c65e82926 Design 47 H1: a step's named path may leave the module's directory, never the repository
A package artifact's tree names same-repository siblings its go.mod requires
as "../<module>/<dir>"; the fetch request's check and the narrowing refused
them as leaving the module. They are now accepted when their join with the
module's directory stays inside the repository, narrowed from the root, and
refused when they leave the repository.

Builds: design 47 H H-A-198, H-A-199, H-A-157, H-A-172
2026-11-20 22:14:53 +01:00
mesh-tools d40f314e52 Merge pull request 'Design 47 H1: the grants holder keeps the catalog revisions it read and learns the newest from the event (H-A-193, H-A-195)' (#294) from fix/47-H1-grants-for-cache into main 2026-11-20 20:56:03 +00:00
jschoubben 120a0554d6 mesh-grants: grants-for keeps a bounded cache of the followed branches
The kept catalog revisions had no design line and their bound stood in
the code. They are now bounded per branch by the module's setting
grants-for-revisions-per-branch (default 4, oldest dropped, a run's
candidate counted within its branch's), read at the start from the file
the manifest writes, and kept only for the watched branches the part's
branch follower follows: a branch that leaves them is freed at the next
grants-for or catalog-revision event. A kept revision is a cache of the
catalog seat's own truth, never a state of the mesh.

Builds: design 47 G §441, §401; H H-A-195, H-A-193
2026-11-20 21:51:43 +01:00
jschoubben a62f7b40c1 mesh-grants: grants-for learns the newest revision from the event
A grants-for call still read the first page of each revision (up to
768 KB) only to learn its number. The grants holder already hears the
seat mesh-catalog's catalog-revision event, which names each branch's
newest trunk revision and the source revision it was taken in from; the
part now records both from the event and asks the revision verb only for
a revision it does not hold, by its number. Before the first event after
a start the first ask's answer names the newest once. An event below the
newest held is the numbering started again: what is kept of the branch
is dropped.

Builds: design 47 G §441, §401; H H-A-193
2026-11-20 21:44:06 +01:00
jschoubben 20d59ff4f0 mesh-grants: grants-for keeps the catalog revisions it read
grants-for read two whole catalog revisions page by page on every call
(about 1.4 MB each) and answered in 3.4 to 5.0 s, so the pipeline
runner's 5 s read bound cut it and every widening touching the vault
said "could not run". A revision never changes, so the part now keeps
each revision it read whole, by branch, kind, group, heads and number,
the newest four per branch for the process's life (as the runner keeps
its revisionCache); once a first page names a kept revision the pages
after it are not read. The branch's newest is read on by the number its
first page names, and a revision whose pages name another is not kept.

Quick fix (not chosen): raise the runner's widens read bound, a catalog
merge that hides the cost. Thorough fix (chosen): the cache here. The
compose of every account twice is kept whole: narrowing it to the
candidate's modules would stop a refused principal elsewhere from
failing grants-for, which is not obviously safe.

Builds: design 47 G §441, §401; H §774
2026-11-20 21:40:15 +01:00
23 changed files with 887 additions and 26 deletions
@@ -45,7 +45,7 @@ func (m *entryManifests) keep(e meshcatalog.CatalogEntry) error {
if m.byFp == nil {
m.byFp = map[string]catalogue.Manifest{}
}
m.byFp[meshcatalog.BareManifestDigest(e.ManifestDigest)] = mf
m.byFp[meshcatalogclient.BareManifestDigest(e.ManifestDigest)] = mf
return nil
}
@@ -291,7 +291,7 @@ func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.
// built against, its packages and needs, and the named paths of its artifacts when every artifact names them.
func ModuleOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
m := meshcatalog.ModulesByModuleValue{Repository: e.Repository, Path: e.Path, Commit: e.Commit, Fp2: e.Fp, None2: e.None,
Manifest: meshcatalog.BareManifestDigest(e.ManifestDigest), Packages: e.Packages, Needs: e.Needs}
Manifest: meshcatalogclient.BareManifestDigest(e.ManifestDigest), Packages: e.Packages, Needs: e.Needs}
if len(e.Bases) > 0 {
m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{Bases: e.Bases}
}
@@ -6,6 +6,7 @@ import (
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
)
@@ -64,7 +65,7 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outc
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry of %s at %s is of %s at %s", e.Module, e.Fingerprint, entry.Module, entry.Commit)
}
// A package the entry names with no pin is built against its publish, waited for (H-A-192, H-A-165).
pinned, outcome, said := b.awaitPublished(ctx, entry.Module, entry.Fp, meshcatalog.BareManifestDigest(entry.ManifestDigest), entry.Packages)
pinned, outcome, said := b.awaitPublished(ctx, entry.Module, entry.Fp, meshcatalogclient.BareManifestDigest(entry.ManifestDigest), entry.Packages)
if outcome != "" {
return outcome, said
}
@@ -81,8 +82,8 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outc
}
return meshbuilds.StepEndedOutcomeFailed, failedWords
}
m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalog.BareManifestDigest(entry.ManifestDigest), Path: entry.Path,
Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalog.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages}
m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalogclient.BareManifestDigest(entry.ManifestDigest), Path: entry.Path,
Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages}
b.consider(ctx, entry.Module, m)
select {
case log := <-ended:
@@ -113,7 +114,7 @@ func (b *Builder) passedWithItsVersion(ctx context.Context, entry meshcatalog.Ca
if !reserving {
return meshbuilds.StepEndedOutcomePassed, ""
}
plan, planErr, _ := b.planOf(ctx, meshcatalog.BareManifestDigest(entry.ManifestDigest))
plan, planErr, _ := b.planOf(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))
switch {
case planErr != nil:
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the plan cannot be read to judge its version: %v", entry.Module, entry.Fp, planErr)
@@ -1,4 +1,7 @@
package meshcatalog
package meshcatalogclient
// The two helpers below are hand-written, so they live in the seat's client, never under its contract's generated
// root (design 47 H, H-A-200; ADR 0608 §5).
import "strings"
@@ -0,0 +1,22 @@
package meshcatalogclient
import "testing"
// An entry's manifest-digest carries sha256: and the module's value and the key of `manifests` the 64 digits bare, one
// digest in two forms (design 47 G §578); the two helpers live in the seat's client, outside its contract's generated
// root (H-A-200).
func TestAManifestDigestIsPrefixedForTheEntryAndBareForTheKey(t *testing.T) {
const bare = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
for _, tc := range []struct{ in, entry, key string }{
{bare, "sha256:" + bare, bare},
{"sha256:" + bare, "sha256:" + bare, bare},
{"", "", ""},
} {
if got := EntryManifestDigest(tc.in); got != tc.entry {
t.Errorf("EntryManifestDigest(%q) = %q, want %q", tc.in, got, tc.entry)
}
if got := BareManifestDigest(tc.in); got != tc.key {
t.Errorf("BareManifestDigest(%q) = %q, want %q", tc.in, got, tc.key)
}
}
}
@@ -9,6 +9,7 @@ import (
"reflect"
"slices"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/graph"
"git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -30,7 +31,7 @@ const CatalogEntryLayout = 1
// ModuleValueOf gives the module's value back from the entry alone. The acting rules are the revision's to add.
func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catalogue.Manifest, raw string, n graph.Node) meshcatalog.CatalogEntry {
e := meshcatalog.CatalogEntry{V: CatalogEntryLayout, Module: module, Repository: cur.Repository, Path: cur.Path,
Commit: cur.Commit, Manifest: manifestWhole(raw), ManifestDigest: meshcatalog.EntryManifestDigest(cur.Manifest), Verdict: cur.Verdict, Fp: n.FP, None: n.None, Bases: n.Bases,
Commit: cur.Commit, Manifest: manifestWhole(raw), ManifestDigest: meshcatalogclient.EntryManifestDigest(cur.Manifest), Verdict: cur.Verdict, Fp: n.FP, None: n.None, Bases: n.Bases,
Packages: maps.Clone(cur.Packages), Provides: n.Provides, Requires: n.Requires, Awaits: n.Awaits, Invokes: n.Invokes,
Serves: n.Serves, LeftOut: n.LeftOut, Main: MainOf(m), Contributes: contributesOf(m)}
// requires names every requirement as catalogue.Manifest.Wants lists them, optional ones and those only contributes names
@@ -67,7 +68,7 @@ func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catal
// ruling on milestone 47/G finding 1; ADR 0656 §6: one form of the fact). The old line's fields (the old-form
// fingerprint, the reader needs) are not kept.
func ModuleValueOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
v := meshcatalog.ModulesByModuleValue{V: keys.Version, Repository: e.Repository, Path: e.Path, Commit: e.Commit, Manifest: meshcatalog.BareManifestDigest(e.ManifestDigest),
v := meshcatalog.ModulesByModuleValue{V: keys.Version, Repository: e.Repository, Path: e.Path, Commit: e.Commit, Manifest: meshcatalogclient.BareManifestDigest(e.ManifestDigest),
Verdict: e.Verdict, Fp2: e.Fp, None2: e.None, Packages: maps.Clone(e.Packages)}
if in := e.Inputs; in != nil {
v.Toolchains, v.Paths, v.Problems, v.Warnings, v.Unknown = maps.Clone(in.Toolchains), maps.Clone(in.Paths), in.Problems, in.Warnings, in.Unknown
@@ -544,7 +544,7 @@ func (k *KeptTrunk) reread(ctx context.Context, o Objects) error {
}
manifests := map[string]string{}
for _, e := range entries {
if d := meshcatalog.BareManifestDigest(e.ManifestDigest); d != "" && e.Manifest != nil {
if d := meshcatalogclient.BareManifestDigest(e.ManifestDigest); d != "" && e.Manifest != nil {
if raw, err := json.Marshal(e.Manifest); err == nil {
manifests[d] = string(raw)
}
@@ -10,6 +10,7 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/views"
)
@@ -94,7 +95,7 @@ func TestTheKeptTrunkIsReadAtTheStartAndOncePerHeardRevision(t *testing.T) {
}
// A-487c: the manifests by digest and the verb revision asked of the trunk's newest answer from the view held.
listed = objects.n.Load()
if raw, err := kept.Manifest(meshcatalog.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` {
if raw, err := kept.Manifest(meshcatalogclient.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` {
t.Errorf("the held manifest = %q (%v), want the entry's manifest", raw, err)
}
if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision.Revision != 2 || len(a.Entries) != 2 {
@@ -63,6 +63,7 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
return w, err
}
objects, namedErr := ix.namedObjects(ctx, f)
trees := ix.goModTree(ctx, f)
overlay := map[string]meshcatalog.ModulesByModuleValue{}
dirOf := map[string]string{}
named, every := map[string]bool{}, true // every module the message's manifests name, and whether each names one
@@ -109,7 +110,23 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
next.None2 = "the manifest cannot be read: it cannot be kept as a value"
} else {
packages := ix.goPackages(ctx, f, dir, cur, has)
if m, err := judge.ForInputs([]byte(raw)); err == nil && usesGo(languagesOf(m)) {
// The entry pins every mesh package of another repository the module's build requires, transitively across
// the same-repository modules of its graph (design 47 H-A-201).
packages = unionSorted(packages, trees.otherRequired(goModuleDirs(m, dir)))
}
in := inputsOfNamed(name, raw, f.Trees[dir], f.Locks[dir], objects[dir], packages, ix.baseFP(ctx))
if in.Kept != nil {
// A Go build resolves the same-repository modules its go.mod files require, transitively, from the tree: their
// directories' objects join the tree input of every form, never the entry's paths (design 47 H-A-202, H-A-172).
if m, err := judge.ForInputs([]byte(raw)); err == nil && usesGo(languagesOf(m)) {
if tree, none := ix.derivedTree(ctx, f, trees, m, dir, in.Kept.Tree); none != "" {
in.Kept, in.None2 = nil, none
} else {
in.Kept.Tree = tree
}
}
}
next.Inputs2, next.None2, next.Paths = in.Kept, in.None2, in.ArtifactPaths
if in.Kept != nil && has && cur.Inputs2 != nil {
// The imports its last trunk build reported stand until its next build reports again (ADR 0593 §2, §4).
@@ -0,0 +1,196 @@
package index
import (
"context"
"fmt"
"path"
"path/filepath"
"slices"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
)
// goModTree reads the go.mod files of one merge's tree, each directory once (Indexer.GoMod), for the walk of the module
// graph a Go build resolves from the tree (design 47 H-A-202, H-A-157).
type goModTree struct {
ix *Indexer
ctx context.Context
f whole
read map[string][]byte // by directory from the repository's root; nil for none
}
func (ix *Indexer) goModTree(ctx context.Context, f whole) *goModTree {
return &goModTree{ix: ix, ctx: ctx, f: f, read: map[string][]byte{}}
}
// goMod is the go.mod of the directory dir (from the root, "" for the root) at the merge commit; nil when there is none
// or it cannot be read, said.
func (t *goModTree) goMod(dir string) []byte {
if raw, ok := t.read[dir]; ok {
return raw
}
var raw []byte
if t.ix.GoMod != nil {
var err error
if raw, err = t.ix.GoMod(t.ctx, t.f.Repository, t.f.Merge, dir); err != nil {
t.ix.logf("%s at %s: the go.mod of %q cannot be read, so no module of the tree is found there: %v", t.f.Repository, t.f.Merge, dir, err)
raw = nil
}
}
t.read[dir] = raw
return raw
}
// sameRepositoryDir is the directory of the merge's tree that holds the Go module required: the required path starts with
// the repository's module prefix, and the directory it names holds a go.mod declaring that path (H-A-198, H-A-202). false
// for a module of another repository, or one the tree holds no go.mod of.
func (t *goModTree) sameRepositoryDir(required string) (string, bool) {
repo, dir, ok := trunk.PackageRepository(required)
if !ok || dir == "" || !trunk.SameRepository(repo, t.f.Repository) {
return "", false
}
if goModModule(t.goMod(dir)) != required {
return "", false
}
return dir, true
}
// sameRepositoryRequired are the directories (from the root, sorted, each once) of every same-repository Go module the
// go.mod of each start directory requires, transitively through those modules' own go.mod files (H-A-202, H-A-157). The
// start directories themselves are not among them.
func (t *goModTree) sameRepositoryRequired(start []string) []string {
seen := map[string]bool{}
for _, s := range start {
seen[s] = true
}
queue := slices.Clone(start)
var out []string
for len(queue) > 0 {
dir := queue[0]
queue = queue[1:]
for _, req := range fingerprint.MeshRequiresOf(t.goMod(dir)) {
d, ok := t.sameRepositoryDir(req)
if !ok || seen[d] {
continue
}
seen[d] = true
out = append(out, d)
queue = append(queue, d)
}
}
slices.Sort(out)
return out
}
// goModuleDirs are the directories (from the root) a module's Go build starts from: its own directory and the directory
// of each path of each Go package artifact of its manifest m.
func goModuleDirs(m catalogue.Manifest, dir string) []string {
out := []string{dir}
if m.Build == nil {
return out
}
for _, a := range m.Build.Artifacts {
if a.Kind != catalogue.ArtifactPackage || a.Language != "go" {
continue
}
for _, p := range a.Paths {
if d := path.Join(dir, strings.TrimSuffix(filepath.ToSlash(p), "/")); !slices.Contains(out, d) {
out = append(out, d)
}
}
}
return out
}
// otherRequired are the mesh packages of other repositories (sorted, each once) that the go.mod of each start directory,
// and of every same-repository module those require transitively (sameRepositoryRequired), requires at any version, the
// zero version included: every one the module's build resolves, which the entry pins (design 47 H-A-201, H-A-183,
// H-A-157). A same-repository module is none: it is built from the tree (H-A-198, H-A-202). nil for none.
func (t *goModTree) otherRequired(start []string) []string {
var out []string
for _, d := range append(slices.Clone(start), t.sameRepositoryRequired(start)...) {
out = append(out, fingerprint.OtherPackages(fingerprint.MeshRequiresOf(t.goMod(d)), t.f.Repository)...)
}
if len(out) == 0 {
return nil
}
return slices.Compact(slices.Sorted(slices.Values(out)))
}
// unionSorted is a and b together, sorted and each once; nil for none.
func unionSorted(a, b []string) []string {
if len(a)+len(b) == 0 {
return nil
}
return slices.Compact(slices.Sorted(slices.Values(append(slices.Clone(a), b...))))
}
// derivedDirs are the same-repository module directories (from the root, sorted) the module's Go build resolves from the
// tree beyond its own tree input (H-A-202): every one its go.mod files require, transitively, that its tree input does
// not hold already, which is the module's directory for the directory's tree, and its named paths for the named tree.
func (t *goModTree) derivedDirs(m catalogue.Manifest, dir string) []string {
var held []string
if paths, every := m.NamedPaths(); every {
for _, p := range paths {
held = append(held, path.Join(dir, filepath.ToSlash(p)))
}
} else {
held = []string{dir}
}
var out []string
for _, d := range t.sameRepositoryRequired(goModuleDirs(m, dir)) {
if !slices.ContainsFunc(held, func(h string) bool { return h == "" || h == "." || d == h || strings.HasPrefix(d, h+"/") }) {
out = append(out, d)
}
}
return out
}
// DerivedLine opens each line of the tree input naming a derived directory and its git object (H-A-202, H-A-172).
const DerivedLine = "derived "
// derivedTree is the module's tree input with a line per derived directory and its object appended, sorted by directory
// (H-A-202, H-A-172: what a build reads moves its fingerprint), so a change of such a directory moves the fingerprint of
// every form. The objects are asked of the forge at the merge commit by their paths from the root (Named); an answer that
// lacks one gives the reason the module has no inputs, naming them. A module with no derived directory keeps its tree.
func (ix *Indexer) derivedTree(ctx context.Context, f whole, t *goModTree, m catalogue.Manifest, dir, tree string) (string, string) {
dirs := t.derivedDirs(m, dir)
if len(dirs) == 0 {
return tree, ""
}
if ix.Named == nil {
return "", "the forge cannot be asked the git objects of the derived directories " + strings.Join(dirs, ", ")
}
got, err := ix.Named(ctx, f.Repository, f.Merge, map[string][]string{"": dirs})
if err != nil {
return "", fmt.Sprintf("the git objects of the derived directories %s: %v", strings.Join(dirs, ", "), err)
}
lines := []string{tree}
var missing []string
for _, d := range dirs {
o := got[""][d]
if o == "" {
missing = append(missing, d)
continue
}
lines = append(lines, DerivedLine+d+" "+o)
}
if len(missing) > 0 {
return "", "the forge names no git object of the derived directories " + strings.Join(missing, ", ")
}
return strings.Join(lines, "\n"), ""
}
// goModModule is the module path a go.mod declares; "" for none.
func goModModule(raw []byte) string {
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) == 2 && f[0] == "module" {
return strings.Trim(f[1], `"`)
}
}
return ""
}
@@ -0,0 +1,144 @@
package index
import (
"context"
"strings"
"testing"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
)
// A fixture repository novox/mesh-delivery: the Go package state of modules/g requires the same-repository module
// modules/b/builds at the zero version, which requires modules/c/client in turn; mesh-sdk is of another repository, and
// modules/b/gone names a directory the tree holds no go.mod of.
const ptpZero = " v0.0.0-00010101000000-000000000000"
var ptpGoMods = map[string]string{
"modules/g/state": "module git.novox.be/novox/mesh-delivery/modules/g/state\n\nrequire (\n\tgit.novox.be/novox/mesh-delivery/modules/b/builds" + ptpZero +
"\n\tgit.novox.be/novox/mesh-sdk/go" + ptpZero + "\n\tgit.novox.be/novox/mesh-delivery/modules/b/gone" + ptpZero + "\n)\n",
"modules/b/builds": "module git.novox.be/novox/mesh-delivery/modules/b/builds\n\nrequire git.novox.be/novox/mesh-delivery/modules/c/client" + ptpZero + " // indirect\n",
"modules/c/client": "module git.novox.be/novox/mesh-delivery/modules/c/client\n",
}
// ptpProgram is modules/g's manifest: a program naming no paths beside the package state.
const ptpProgram = `{"module":"g","version":"1.0","build":{"artifacts":[{"name":"program","kind":"bundle","language":"go","system":"arch","from":"cmd/g","binary":"g","loads":["g"]},` +
`{"name":"state","kind":"package","language":"go","paths":["state"],"series":"0.1"}]}}`
// ptpIndexer answers the fixture's go.mod files and, for a path asked from the root, the object objects names for it.
func ptpIndexer(t *testing.T, objects map[string]string) (*Indexer, *memStore, *[]map[string][]string) {
ix, mods, _ := manifestsIndexer(t)
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := ptpGoMods[dir]; ok {
return []byte(c), nil
}
return nil, nil
}
var asked []map[string][]string
ix.Named = func(_ context.Context, _, _ string, paths map[string][]string) (map[string]map[string]string, error) {
asked = append(asked, paths)
out := map[string]map[string]string{}
for dir, ps := range paths {
out[dir] = map[string]string{}
for _, p := range ps {
out[dir][p] = objects[p]
}
}
return out, nil
}
return ix, mods, &asked
}
// Design 47 H-A-202, H-A-172: the same-repository modules a Go build resolves from the tree, transitively, join the tree
// input of every form with their objects, so a change of one moves the fingerprint of a module whose program names no
// paths; the entry's paths stay the manifest's.
func TestADerivedDirectoryMovesTheFingerprintOfEveryForm(t *testing.T) {
objects := map[string]string{"modules/b/builds": strings.Repeat("b", 40), "modules/c/client": strings.Repeat("c", 40)}
tree := func(objects map[string]string) string {
ix, mods, asked := ptpIndexer(t, objects)
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": ptpProgram}, nil)
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "g")
if got := strings.Join(m.Paths["state"], ","); got != "state" {
t.Errorf("the entry's paths of state are %q, want the manifest's", got)
}
if len(*asked) != 1 || strings.Join((*asked)[0][""], ",") != "modules/b/builds,modules/c/client" {
t.Fatalf("asked the forge %v, want the derived directories from the root once", *asked)
}
if m.Inputs2 == nil || !strings.HasPrefix(m.Inputs2.Tree, "gtree\n") {
t.Fatalf("inputs = %+v (%s), want the directory's tree with the derived lines", m.Inputs2, m.None2)
}
return m.Inputs2.Tree
}
before := tree(objects)
if want := "gtree\nderived modules/b/builds " + objects["modules/b/builds"] + "\nderived modules/c/client " + objects["modules/c/client"]; before != want {
t.Errorf("the tree input is %q, want %q", before, want)
}
objects["modules/c/client"] = strings.Repeat("d", 40)
after := tree(objects)
fp := func(tree string) string {
v, _ := fingerprint.Of(fingerprint.Inputs{Module: "g", Tree: tree})
return v
}
if fp(before) == fp(after) {
t.Errorf("a change of modules/c/client left the fingerprint %s", fp(after))
}
// The named tree takes them too: a module whose every artifact names its paths.
named := `{"module":"g","version":"1.0","build":{"artifacts":[{"name":"state","kind":"package","language":"go","paths":["state"],"series":"0.1"}]}}`
ix, mods, _ := ptpIndexer(t, objects)
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": named}, nil)
f.Paths = map[string]map[string]string{"modules/g": {"state": strings.Repeat("1", 40)}}
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Inputs2 == nil || !strings.HasPrefix(m.Inputs2.Tree, fingerprint.NamedTreePrefix) ||
!strings.HasSuffix(m.Inputs2.Tree, "\nderived modules/c/client "+objects["modules/c/client"]) || strings.Join(m.Paths["state"], ",") != "state" {
t.Errorf("module = %+v, want the named tree with the derived lines and the manifest's paths", m)
}
// A forge that names no object of a derived directory gives the module none, naming it.
ix, mods, _ = ptpIndexer(t, map[string]string{"modules/b/builds": strings.Repeat("b", 40)})
if err := ix.Apply(context.Background(), withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}),
map[string]string{"modules/g": ptpProgram}, nil)); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Fp2 != "" || !strings.Contains(m.None2, "modules/c/client") {
t.Errorf("module = %+v, want none naming the directory the forge gave no object of", m)
}
}
// Design 47 H-A-201: the entry pins every mesh package of another repository the module's build requires, across the
// same-repository modules of its graph: here the one modules/c/client requires, which modules/g's go.mod never names.
// A module of the module's own repository is never pinned, whatever a build reported (H-A-198, H-A-202).
func TestTheEntryPinsEveryRequirementOfTheModuleGraph(t *testing.T) {
const conditions = "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
objects := map[string]string{"modules/b/builds": strings.Repeat("b", 40), "modules/c/client": strings.Repeat("c", 40)}
ix, mods, _ := ptpIndexer(t, objects)
mod := map[string]string{
"modules/g": "module g\n\nrequire git.novox.be/novox/mesh-delivery/modules/b/builds" + ptpZero + "\n",
"modules/c/client": "module git.novox.be/novox/mesh-delivery/modules/c/client\n\nrequire " + conditions + ptpZero + "\n",
}
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := mod[dir]; ok {
return []byte(c), nil
}
return []byte(ptpGoMods[dir]), nil
}
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": ptpProgram},
map[string]map[string]string{"modules/g": {"go.sum": avahiGoSum}})
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "g")
if m.Inputs2 == nil || strings.Join(m.Inputs2.Packages, ",") != conditions+",git.novox.be/novox/mesh-sdk/go" {
t.Errorf("packages = %+v, want the client's requirement and state's mesh-sdk, and no module of the repository", m.Inputs2)
}
builds := "git.novox.be/novox/mesh-delivery/modules/b/builds"
pins := releasedPins(sdkstep.PackageVersions{builds: "v0.1.106", conditions: "v0.1.4"}, nil, catalogue.Manifest{}, "novox/mesh-delivery")
if _, pinned := pins[builds]; pinned || len(pins) != 1 {
t.Errorf("pins = %v, want %s only, never a module of the repository", pins, conditions)
}
}
@@ -298,7 +298,7 @@ func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshc
r.fp, r.none = f.fp, f.none
if r.fp != "" {
// a pin is a released version, never a commit (H-A-161), and the version line's fingerprint takes it (H-A-167)
r.fp, r.pkgs = pinnedOf(f, series, k.manifest)
r.fp, r.pkgs = pinnedOf(f, series, k.manifest, k.cur.Repository)
r.tcs = toolchainsUsed(toolchains)
}
}
@@ -517,7 +517,7 @@ func stillComputed(module string, kept meshcatalog.ModulesByModuleValueInputs2,
if f.none != "" || f.fp == "" {
return "", nil, false
}
fp, pkgs := pinnedOf(f, series, k.manifest)
fp, pkgs := pinnedOf(f, series, k.manifest, k.held.Repository)
return fp, pkgs, fp == k.held.Fp2
}
@@ -15,6 +15,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/keys"
@@ -373,13 +374,17 @@ func isFloor(raw []byte) bool {
// within the series the module accepts, the one its manifest m names or the default (Manifest.AcceptedSeries), whether
// or not that series holds contents yet; a publish in another series pins nothing and moves nothing, and a module's
// release number is never a pin. A package with none published in that series is kept with no version, so its kind-1
// edge stands (H §746) and its build waits for the publish (H-A-165). nil for none.
func releasedPins(versions sdkstep.PackageVersions, series map[string]SeriesPackage, m catalogue.Manifest) sdkstep.PackageVersions {
// edge stands (H §746) and its build waits for the publish (H-A-165). A module of the module's own repository is never
// pinned, whatever a build reported: it is built from the tree (design 47 H-A-198, H-A-201, H-A-202). nil for none.
func releasedPins(versions sdkstep.PackageVersions, series map[string]SeriesPackage, m catalogue.Manifest, repository string) sdkstep.PackageVersions {
if len(versions) == 0 {
return nil
}
out := sdkstep.PackageVersions{}
for p := range versions {
if repo, _, ok := trunk.PackageRepository(p); ok && repository != "" && trunk.SameRepository(repo, repository) {
continue
}
out[p] = ""
for _, s := range series {
if s.GoModule != p || s.Newest == nil || s.Series != m.AcceptedSeries(s.Name) {
@@ -398,8 +403,8 @@ func releasedPins(versions sdkstep.PackageVersions, series map[string]SeriesPack
// form's fingerprint takes them (design 47 H-A-167, H-A-172; ADR 0573 §4) and a re-pin moves it, the module a mover
// built again against the release; a fingerprint with no pin is the form's as it is. The held fingerprint of a build that
// stands is kept while its pins are the ones it was built with, else its import form takes the new pins.
func pinnedOf(f form, series map[string]SeriesPackage, m catalogue.Manifest) (string, sdkstep.PackageVersions) {
pins := releasedPins(f.versions, series, m)
func pinnedOf(f form, series map[string]SeriesPackage, m catalogue.Manifest, repository string) (string, sdkstep.PackageVersions) {
pins := releasedPins(f.versions, series, m, repository)
if f.stands {
if maps.Equal(f.built, map[string]string(pins)) {
return f.fp, pins
@@ -146,9 +146,15 @@ func (r FetchRequest) Check() error {
if r.Dir != "" && (outside(r.Dir) || strings.HasPrefix(r.Dir, "/")) {
return fmt.Errorf("the module directory %q leaves the repository", r.Dir)
}
// A path is relative to the module's directory; one that leaves it but stays inside the repository (a sibling the
// artifact's go.mod requires, "../<module>/<dir>") is accepted and narrowed from the repository's root, and one that
// leaves the repository is refused (design 47 H-A-198, H-A-199).
for _, p := range r.Paths {
if p == "" || strings.HasPrefix(p, "/") || outside(p) || path.Clean(p) == "." {
return fmt.Errorf("the path %q leaves the module's directory or is not a path in it", p)
if p == "" || strings.HasPrefix(p, "/") || path.Clean(p) == "." {
return fmt.Errorf("the path %q is not a path relative to the module's directory", p)
}
if outside(path.Join(r.Dir, p)) {
return fmt.Errorf("the path %q leaves the repository", p)
}
}
return nil
@@ -37,6 +37,11 @@ func TestAFetchRequestCarriesTheNamedPathsThroughTheToolCall(t *testing.T) {
if dirOnly, err := ParseFetchRequest(map[string]any{"repository": "novox/x", "commit": strings.Repeat("b", 40), "dir": "modules/a"}); err != nil || dirOnly.Dir != "modules/a" || dirOnly.Paths != nil {
t.Fatalf("a directory with no paths: %+v, %v", dirOnly, err)
}
// A sibling of the same repository is a path that leaves the module's directory, not the repository (H-A-199).
if sib, err := ParseFetchRequest(map[string]any{"repository": "novox/x", "commit": strings.Repeat("b", 40), "dir": "modules/a",
"paths": []any{"pkg", "../b/client"}}); err != nil || len(sib.Paths) != 2 {
t.Fatalf("a sibling path inside the repository: %+v, %v", sib, err)
}
for _, c := range []struct {
set map[string]any
says string
@@ -46,7 +51,8 @@ func TestAFetchRequestCarriesTheNamedPathsThroughTheToolCall(t *testing.T) {
{map[string]any{"commit": "abc"}, "is not a full commit"},
{map[string]any{"base": "abc"}, "is not a trunk commit"},
{map[string]any{"paths": []any{"src"}}, "names no module directory"},
{map[string]any{"dir": "modules/a", "paths": []any{"../b"}}, "leaves the module's directory"},
{map[string]any{"dir": "modules/a", "paths": []any{"../../../b"}}, "leaves the repository"},
{map[string]any{"dir": "modules/a", "paths": []any{"/b"}}, "not a path relative to the module's directory"},
{map[string]any{"dir": "../a", "paths": []any{"src"}}, "leaves the repository"},
{map[string]any{"dir": "modules/a", "paths": []any{3}}, "paths is a list of strings"},
{map[string]any{"packages": map[string]any{"git.novox.be/novox/mesh-sdk/go": "main"}}, "neither a full commit nor a Go semantic version"},
@@ -100,6 +100,14 @@ func TestAContractsModulesGeneratedFilesAreComparedWithTheGenerator(t *testing.T
if got, err := OutOfStep(plain, GeneratedRootGo); err != nil || len(got) != 0 {
t.Errorf("a module naming no generated-from compared: %v %v", got, err)
}
// This repository's own contracts module is in step with the generator under both roots: a hand-written file
// under a generated root (go/seats/meshcatalog/manifestdigest.go, pipeline main.43) failed every unit step of its
// mover; a seat's hand-written helper lives in its client (design 47 H, H-A-200; ADR 0608 §5).
for _, root := range []string{GeneratedRootGo, GeneratedRootTS} {
if got, err := OutOfStep(filepath.Join("..", "..", "mesh-delivery-contracts"), root); err != nil || len(got) != 0 {
t.Errorf("mesh-delivery-contracts, root %s: %v %v, want nothing out of step", root, got, err)
}
}
// The build ask answers them beside what it built, and with a failed build too.
built := 0
+3 -1
View File
@@ -25,9 +25,11 @@ func Narrow(tree string, r sdkstep.FetchRequest) error {
}
keep := map[string]bool{".git": true}
for _, p := range r.Paths {
// Kept by its path from the repository's root: a path leaving the module's directory but not the repository is
// narrowed from the root like any other (design 47 H-A-199).
rel := path.Join(r.Dir, p)
if strings.HasPrefix(rel, "../") || rel == ".." || path.IsAbs(p) {
return fmt.Errorf("the named path %q leaves the module's directory", p)
return fmt.Errorf("the named path %q leaves the repository", p)
}
if _, err := os.Lstat(filepath.Join(tree, filepath.FromSlash(rel))); err != nil {
return fmt.Errorf("the named path %q is not in the fetched tree: %w", p, err)
@@ -68,6 +68,25 @@ func TestNarrowKeepsAModuleDirectorysPathsAndFailsOnAnAbsentOne(t *testing.T) {
}
}
// A path leaving the module's directory but not the repository is kept from the root; one leaving the repository is
// refused, naming the repository (design 47 H-A-198, H-A-199).
func TestNarrowKeepsASiblingOfTheRepositoryAndRefusesOneOutsideIt(t *testing.T) {
tree := t.TempDir()
for _, f := range []string{"modules/a/module.json", "modules/a/pkg/go.mod", "modules/a/notes.md", "modules/b/client/go.mod",
"modules/b/other.go", "root.txt"} {
writeTreeFile(t, tree, f, "x")
}
if err := Narrow(tree, sdkstep.FetchRequest{Dir: "modules/a", Paths: []string{"../../../x"}}); err == nil || !strings.Contains(err.Error(), "leaves the repository") {
t.Fatalf("a path leaving the repository gave %v", err)
}
if err := Narrow(tree, sdkstep.FetchRequest{Dir: "modules/a", Paths: []string{"pkg", "../b/client"}}); err != nil {
t.Fatal(err)
}
if got := strings.Join(filesUnder(t, tree), ","); got != "modules/a/module.json,modules/a/pkg/go.mod,modules/b/client/go.mod" {
t.Fatalf("the narrowed tree holds %s", got)
}
}
// A build that needs package.json or go.mod names it among its paths, and so keys on it (ADR 0499 clause 4).
func TestNarrowKeepsAManifestOfTheLanguageOnlyWhenNamed(t *testing.T) {
tree := t.TempDir()
@@ -39,6 +39,7 @@ import (
mcseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshwantsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
@@ -180,8 +181,20 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Create: creating{ctx: ctx, js: js}, Consumers: creating{ctx: ctx, js: js}, NamedPurge: creating{ctx: ctx, js: js},
Objects: sealedObjects{ctx: ctx, js: js}, Purging: ownPurging{ctx: ctx, js: js, module: account.Module}, Logf: logger.Printf,
Bus: grants.BusAddress{URL: account.URL, Fingerprint: account.Fingerprint}}
// The setting grants-for-revisions-per-branch, read at the start, and the watched branches the part follows: grants-for
// keeps those branches' revisions alone, so many to a branch (design 47 H H-A-195).
perBranch, err := grants.RevisionsPerBranchSetting(strings.TrimSpace(getenv(grants.EnvRevisionsPerBranch)))
if err != nil {
return err
}
f.Part.RevisionsPerBranch, f.Part.Followed = perBranch, branches.Followed
f.Watch = fv
f.Part.ReadingRunning(nodeengine.StreamRunning)
// The catalog-revision event names each branch's newest revision, so grants-for asks the seat mesh-catalog's revision only
// for a revision it does not hold (design 47 H H-A-193), heard on the same reader beside the branch follower.
if err := reader.AdoptEvent(meshcatalog.Seat, meshcatalog.EventCatalogRevision, f.Part.HearCatalogRevision); err != nil {
return err
}
f.HearKept = func() error { return hearRunning(ctx, account, js, f, logger.Printf) }
// The first view of the bindings (design 25 bindings-first-view): the seat mesh-broker's list, asked at each start and by
// hand, with the seat's events held until its answer is applied or its bound passes.
+135
View File
@@ -83,6 +83,8 @@ type fakeBus struct {
refuseCreate bool
// refuseUsers refuses the bus module's nats_users: the loaded list cannot be read.
refuseUsers bool
// revisionNumber, when set, numbers each revision mesh-catalog's revision answers (newest, candidate), one entry a page.
revisionNumber map[string]int64
// rootFree answers a machine's engine's root-free verb: nil is no answerer at all (no responders).
rootFree func(machines []string) (any, error)
// consumers are the durable consumers the bus holds, by stream, as nats_streams answers them for one stream; and
@@ -379,6 +381,19 @@ func (b *fakeBus) Call(key string, body any) (json.RawMessage, error) {
_ = json.Unmarshal([]byte(manifest), &m)
out.Entries = append(out.Entries, meshcatalog.CatalogEntry{Module: module, Manifest: m})
}
if n, numbered := b.revisionNumber[which]; numbered {
slices.SortFunc(out.Entries, func(x, y meshcatalog.CatalogEntry) int { return strings.Compare(x.Module, y.Module) })
out.Revision = &meshcatalog.CatalogRevision{Branch: in.Branch, Kind: "trunk", Revision: n}
if in.Group != "" {
out.Revision.Kind, out.Revision.Group, out.Revision.Heads = "candidate", in.Group, "h1"
}
for len(out.Entries) > 0 && out.Entries[0].Module <= in.After {
out.Entries = out.Entries[1:]
}
if len(out.Entries) > 1 {
out.Entries, out.Next = out.Entries[:1], out.Entries[0].Module
}
}
return json.Marshal(out)
case grants.CallStreams:
if stream, _ := args["stream"].(string); stream != "" {
@@ -5157,6 +5172,126 @@ func TestGrantsFor(t *testing.T) {
if _, err := p.GrantsFor(meshgrants.GrantsForInput{Branch: "main"}); err == nil {
t.Error("grants-for with neither a source nor a run answered")
}
// A revision never changes, so one read whole is kept: a second grants-for of the same revisions reads only the first page
// of the candidate, which names a revision kept, and none of the newest, whose number the first ask named (H-A-193: it asks
// once); once the catalog-revision events name both, a grants-for reads no page.
b = fixture()
b.revision = map[string]map[string]string{"newest": {"mod-x": xManifest, "mod-y": xManifest}, "candidate": {"mod-x": candidateX, "mod-y": xManifest}}
b.revisionNumber = map[string]int64{"newest": 9, "candidate": 8}
p = partOn(b)
pages := func() int {
b.mu.Lock()
defer b.mu.Unlock()
n := 0
for _, c := range b.calls {
if c == grants.CallRevision {
n++
}
}
return n
}
first, _ := p.GrantsFor(meshgrants.GrantsForInput{Branch: "main", Source: "7"})
read := pages()
again, _ := p.GrantsFor(meshgrants.GrantsForInput{Branch: "main", Source: "7"})
if read != 4 || pages()-read != 1 {
t.Errorf("the revisions were read in %d pages, then %d again, want 4 then 1 (the candidate's first page)", read, pages()-read)
}
if len(first.Modules) != 1 || !reflect.DeepEqual(first, again) {
t.Errorf("grants-for from kept revisions answers %+v, first %+v", again, first)
}
p = partOn(b)
for _, ev := range []string{`{"v":1,"kind":"trunk","branch":"main","revision":8,"source":7}`, `{"v":1,"kind":"trunk","branch":"main","revision":9,"source":8}`} {
if err := p.HearCatalogRevision(context.Background(), []byte(ev)); err != nil {
t.Fatal(err)
}
}
_, _ = p.GrantsFor(meshgrants.GrantsForInput{Branch: "main", Source: "7"})
read = pages()
heard, _ := p.GrantsFor(meshgrants.GrantsForInput{Branch: "main", Source: "7"})
if pages() != read || !reflect.DeepEqual(first, heard) {
t.Errorf("after the events named both revisions held, grants-for read %d pages and answered %+v, want none and %+v", pages()-read, heard, first)
}
}
// grants-for keeps what H-A-195 says of the catalog: so many revisions to a branch as the setting
// grants-for-revisions-per-branch says, a run's candidate counted within its branch's; only the followed branches', freed when
// one leaves them; and an event below the newest held is the numbering started again, which drops what is kept.
func TestGrantsForKeepsWhatItMay(t *testing.T) {
setting := func(t *testing.T, text string) int {
path := filepath.Join(t.TempDir(), "grants-for-revisions-per-branch")
if err := os.WriteFile(path, []byte(text), 0o600); err != nil {
t.Fatal(err)
}
n, err := grants.RevisionsPerBranchSetting(path)
if err != nil {
t.Fatal(err)
}
return n
}
hear := func(t *testing.T, p *grants.Part, revision, source int) {
ev := fmt.Sprintf(`{"v":1,"kind":"trunk","branch":"main","revision":%d,"source":%d}`, revision, source)
if err := p.HearCatalogRevision(context.Background(), []byte(ev)); err != nil {
t.Fatal(err)
}
}
for _, row := range []struct {
name string
// run asks the candidate by a run, else by the source revision 7; then is done after one grants-for; pages the
// revision pages the grants-for after it reads.
run bool
// perBranch is the part's bound from the start; 0 its default.
perBranch int
then func(t *testing.T, b *fakeBus, p *grants.Part, followed *[]string)
pages int
}{
{name: "the setting bounds a branch's revisions: with 2, the candidate is dropped once two newer are kept", pages: 2,
then: func(t *testing.T, b *fakeBus, p *grants.Part, _ *[]string) {
p.RevisionsPerBranch = setting(t, "2\n")
hear(t, p, 10, 0)
b.revisionNumber["newest"] = 10
_, _ = p.GrantsFor(meshgrants.GrantsForInput{Branch: "main", Source: "7"})
}},
{name: "a run's candidate counts within its branch's bound: with 1, the candidate and the newest each drop the other", run: true,
perBranch: 1, pages: 4, then: func(*testing.T, *fakeBus, *grants.Part, *[]string) {}},
{name: "a branch that leaves the watched branches frees its revisions", pages: 4,
then: func(t *testing.T, _ *fakeBus, p *grants.Part, followed *[]string) {
*followed = nil
_, _ = p.GrantsFor(meshgrants.GrantsForInput{Branch: "main", Source: "7"})
*followed = []string{"main"}
}},
{name: "an event below the newest held drops what is kept of the branch", pages: 4,
then: func(t *testing.T, _ *fakeBus, p *grants.Part, _ *[]string) { hear(t, p, 3, 0) }},
} {
t.Run(row.name, func(t *testing.T) {
b := fixture()
b.revision = map[string]map[string]string{"newest": {"mod-x": xManifest, "mod-y": xManifest}, "candidate": {"mod-x": xManifest, "mod-y": xManifest}}
b.revisionNumber = map[string]int64{"newest": 9, "candidate": 8}
followed := []string{"main"}
p := partOn(b)
p.Followed = func() []string { return followed }
p.RevisionsPerBranch = row.perBranch
in := meshgrants.GrantsForInput{Branch: "main", Source: "7"}
if row.run {
in = meshgrants.GrantsForInput{Branch: "main", Run: "r1"}
} else {
hear(t, p, 8, 7)
hear(t, p, 9, 8)
}
if got, err := p.GrantsFor(in); err != nil || got.Error != "" {
t.Fatalf("grants-for: %+v, %v", got, err)
}
row.then(t, b, p, &followed)
before := len(slices.DeleteFunc(slices.Clone(b.calls), func(c string) bool { return c != grants.CallRevision }))
_, _ = p.GrantsFor(in)
if got := len(slices.DeleteFunc(slices.Clone(b.calls), func(c string) bool { return c != grants.CallRevision })) - before; got != row.pages {
t.Errorf("grants-for read %d revision pages, want %d", got, row.pages)
}
})
}
if n, err := grants.RevisionsPerBranchSetting(filepath.Join(t.TempDir(), "none")); n != grants.DefaultRevisionsPerBranch || err != nil {
t.Errorf("with no file the setting is %d, %v; want its default", n, err)
}
}
// heldHalf is an assignment half holding rev, or none.
@@ -1,18 +1,23 @@
package grants
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"maps"
"os"
"slices"
"sort"
"strconv"
"strings"
"sync"
"git.novox.be/novox/mesh-contracts/identity-and-access/go/seats/meshgrants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// **The seat mesh-grants' read `grants-for`** (design 47 G §441, §401, A-205): the grants a candidate would compose, per
@@ -27,10 +32,240 @@ const RunGroup = "run-"
// CallRevision is the seat mesh-catalog's `revision`, as the part calls it.
const CallRevision = "seat:" + meshcatalog.Seat + "." + meshcatalog.VerbRevision
// DefaultRevisionsPerBranch is the setting grants-for-revisions-per-branch's default (design 47 H H-A-195): how many catalog
// revisions of one branch grants-for keeps, a run's candidate counted within its branch's, the oldest read dropped first;
// twice as many source revisions are kept named by the revision taken in from each.
const DefaultRevisionsPerBranch = 4
// EnvRevisionsPerBranch names the file the manifest writes the setting grants-for-revisions-per-branch into.
const EnvRevisionsPerBranch = "MESH_GRANTS_REVISIONS_PER_BRANCH_FILE"
// RevisionsPerBranchSetting reads the setting grants-for-revisions-per-branch from its file, read at the start: a missing or
// empty file is its default, one that holds no positive whole number an error naming the file.
func RevisionsPerBranchSetting(path string) (int, error) {
if path == "" {
return DefaultRevisionsPerBranch, nil
}
raw, err := os.ReadFile(path)
if errors.Is(err, fs.ErrNotExist) {
return DefaultRevisionsPerBranch, nil
}
if err != nil {
return 0, err
}
v := strings.TrimSpace(string(raw))
if v == "" {
return DefaultRevisionsPerBranch, nil
}
n, err := strconv.Atoi(v)
if err != nil || n < 1 {
return 0, fmt.Errorf("%s holds %q, which is not a positive whole number", path, v)
}
return n, nil
}
// revisionsPerBranch is the part's bound of kept revisions per branch: its setting, else the default.
func (p *Part) revisionsPerBranch() int {
if p.RevisionsPerBranch > 0 {
return p.RevisionsPerBranch
}
return DefaultRevisionsPerBranch
}
// following frees what is kept of every branch the part follows no more, and says whether branch is followed: only a followed
// branch's revisions are kept (H-A-195). With no Followed (the tools) every branch is.
func (p *Part) following(branch string) bool {
if p.Followed == nil {
return true
}
followed := map[string]bool{}
for _, b := range p.Followed() {
followed[keys.EscapeName(b)] = true
}
p.revisions.free(followed)
return followed[branch]
}
// revisionsKept keeps the catalog revisions grants-for read, whole, as manifests by module (design 47 H H-A-195): a revision
// never changes, so one read is read for good (as the runner keeps its revisionCache), a cache of the catalog seat's own truth
// and never a state of the mesh. Each is kept by the branch grants-for was asked of (escaped as a key name is), its kind,
// group, heads and number; the newest few of each followed branch are kept in memory for the process's life. It also
// holds what the catalog-revision event names (H-A-193): each branch's newest trunk revision, and the revision each source
// revision was taken in as, so grants-for asks the seat mesh-catalog's revision only for a revision it does not hold, never
// to learn the newest.
type revisionsKept struct {
mu sync.Mutex
order map[string][]string
kept map[string]map[string]catalogue.Manifest
newest map[string]int64
sources map[string]map[int64]int64
sourced map[string][]int64
}
// revisionKey names a revision of branch; every kind but a candidate's is the branch's own series.
func revisionKey(branch string, r meshcatalog.CatalogRevision) string {
if r.Kind != "candidate" {
r.Kind, r.Group, r.Heads = "trunk", "", ""
}
return strings.Join([]string{branch, r.Kind, r.Group, r.Heads, strconv.FormatInt(r.Revision, 10)}, "\x00")
}
func (k *revisionsKept) init() {
if k.kept == nil {
k.kept, k.order, k.newest = map[string]map[string]catalogue.Manifest{}, map[string][]string{}, map[string]int64{}
k.sources, k.sourced = map[string]map[int64]int64{}, map[string][]int64{}
}
}
func (k *revisionsKept) get(branch string, r meshcatalog.CatalogRevision) (map[string]catalogue.Manifest, bool) {
k.mu.Lock()
defer k.mu.Unlock()
m, ok := k.kept[revisionKey(branch, r)]
return m, ok
}
func (k *revisionsKept) put(branch string, r meshcatalog.CatalogRevision, manifests map[string]catalogue.Manifest, max int) {
k.mu.Lock()
defer k.mu.Unlock()
k.init()
key := revisionKey(branch, r)
if _, ok := k.kept[key]; !ok {
k.order[branch] = append(k.order[branch], key)
}
k.kept[key] = manifests
for len(k.order[branch]) > max {
delete(k.kept, k.order[branch][0])
k.order[branch] = k.order[branch][1:]
}
if r.Kind != "candidate" && r.Source != 0 {
k.sourceLocked(branch, r.Source, r.Revision, max)
}
}
// free drops what is kept of each branch not in followed.
func (k *revisionsKept) free(followed map[string]bool) {
k.mu.Lock()
defer k.mu.Unlock()
for _, m := range []map[string]bool{boolsOf(k.order), boolsOf(k.newest), boolsOf(k.sources)} {
for branch := range m {
if !followed[branch] {
k.dropLocked(branch)
}
}
}
}
func boolsOf[V any](m map[string]V) map[string]bool {
out := map[string]bool{}
for k := range m {
out[k] = true
}
return out
}
// dropLocked drops everything kept of branch.
func (k *revisionsKept) dropLocked(branch string) {
for _, key := range k.order[branch] {
delete(k.kept, key)
}
delete(k.order, branch)
delete(k.newest, branch)
delete(k.sources, branch)
delete(k.sourced, branch)
}
func (k *revisionsKept) sourceLocked(branch string, source, revision int64, max int) {
if k.sources[branch] == nil {
k.sources[branch] = map[int64]int64{}
}
if _, ok := k.sources[branch][source]; !ok {
k.sourced[branch] = append(k.sourced[branch], source)
}
k.sources[branch][source] = revision
for len(k.sourced[branch]) > 2*max {
delete(k.sources[branch], k.sourced[branch][0])
k.sourced[branch] = k.sourced[branch][1:]
}
}
// heard takes a trunk revision a catalog-revision event names as its branch's newest. One below the newest held is the
// numbering started again (design 47 H H-A-141): what is kept of the branch was of the earlier numbering, and is dropped.
func (k *revisionsKept) heard(branch string, revision, source int64, max int) {
k.mu.Lock()
defer k.mu.Unlock()
k.init()
if held, ok := k.newest[branch]; ok && revision < held {
k.dropLocked(branch)
}
k.newest[branch] = revision
if source != 0 {
k.sourceLocked(branch, source, revision, max)
}
}
// asked takes the newest revision an ask answered, when no event named one since the start (H-A-193: it asks once).
func (k *revisionsKept) asked(branch string, revision int64) {
k.mu.Lock()
defer k.mu.Unlock()
k.init()
if _, ok := k.newest[branch]; !ok {
k.newest[branch] = revision
}
}
// known is the revision in names that the part knows the number of without asking: the branch's newest an event named, or
// the revision a source revision was taken in as; false when it knows none.
func (k *revisionsKept) known(in meshcatalog.RevisionInput) (int64, bool) {
k.mu.Lock()
defer k.mu.Unlock()
switch {
case in.Group != "" || in.Heads != "" || in.Revision != 0:
return 0, false
case in.Source != 0:
n, ok := k.sources[in.Branch][in.Source]
return n, ok
default:
n, ok := k.newest[in.Branch]
return n, ok
}
}
// HearCatalogRevision takes one catalog-revision event of the seat mesh-catalog, as the part's process hears it (design 47 H
// H-A-193): a trunk revision is its branch's newest, and the revision its source revision was taken in as; a candidate's moves
// nothing here. Data that is no such event is an error.
func (p *Part) HearCatalogRevision(_ context.Context, data []byte) error {
var ev meshcatalog.CatalogRevisionEvent
if err := json.Unmarshal(data, &ev); err != nil || ev.Revision <= 0 {
return fmt.Errorf("not a %s event: %.200s", meshcatalog.EventCatalogRevision, data)
}
if ev.Kind == "candidate" || !p.following(ev.Branch) {
return nil
}
p.revisions.heard(ev.Branch, ev.Revision, ev.Source, p.revisionsPerBranch())
return nil
}
// revisionEntries reads a catalog graph revision's entries, page after page, as manifests by module. none says the branch has
// no revision yet.
// no revision yet. A revision whose number the part knows (the branch's newest an event named, a source revision's) and holds
// is answered from what it holds with no read; one it knows but does not hold is read by that number. Otherwise a revision
// read whole is kept: once the first page of an answer names a revision that is kept, it is answered from what is kept,
// without the pages after it. The branch's newest is read on by the number its first page names, so its pages are of one
// revision though a newer one is written between them; a revision whose pages name another is not kept.
func (p *Part) revisionEntries(in meshcatalog.RevisionInput) (manifests map[string]catalogue.Manifest, none bool, err error) {
branch := in.Branch
followed := p.following(branch)
newestAsked := in.Group == "" && in.Heads == "" && in.Source == 0 && in.Revision == 0
if n, ok := p.revisions.known(in); ok {
if m, held := p.revisions.get(branch, meshcatalog.CatalogRevision{Revision: n}); held {
return m, false, nil
}
if newestAsked {
in.Revision = n
}
}
manifests = map[string]catalogue.Manifest{}
var first *meshcatalog.CatalogRevision
keep := true
for {
raw, err := p.Call.Call(CallRevision, in)
if err != nil {
@@ -43,6 +278,23 @@ func (p *Part) revisionEntries(in meshcatalog.RevisionInput) (manifests map[stri
if a.NoneYet {
return nil, true, nil
}
switch {
case a.Revision == nil:
keep = false
case first == nil:
if newestAsked && followed {
p.revisions.asked(branch, a.Revision.Revision)
}
if m, ok := p.revisions.get(branch, *a.Revision); ok {
return m, false, nil
}
first = a.Revision
if newestAsked {
in.Revision = a.Revision.Revision
}
case revisionKey(branch, *a.Revision) != revisionKey(branch, *first):
keep = false
}
for _, e := range a.Entries {
body, err := json.Marshal(e.Manifest)
if err != nil || len(e.Manifest) == 0 {
@@ -55,6 +307,9 @@ func (p *Part) revisionEntries(in meshcatalog.RevisionInput) (manifests map[stri
manifests[e.Module] = m
}
if a.Next == "" || a.Next == in.After {
if keep && first != nil && followed {
p.revisions.put(branch, *first, manifests, p.revisionsPerBranch())
}
return manifests, false, nil
}
in.After = a.Next
+11 -3
View File
@@ -194,11 +194,19 @@ type Part struct {
// SeatList is the seat list reader, which gates a module whose seats the list has not named (inputs.go, SeatGate); nil
// gates none.
SeatList SeatGate
// RevisionsPerBranch is the setting grants-for-revisions-per-branch, read at the start (grantsfor.go, design 47 H H-A-195);
// 0 is its default.
RevisionsPerBranch int
// Followed is the watched branches the part follows, as named (the branch follower's): grants-for keeps the revisions
// of these alone, and frees a branch's once it leaves them; nil in the tools, where every branch is kept.
Followed func() []string
cache Manifests
mu sync.Mutex
last Result
made creations
// revisions are the catalog revisions grants-for read, kept whole by their number (grantsfor.go).
revisions revisionsKept
mu sync.Mutex
last Result
made creations
// askSeen is the revision of the last ask to compose (KeyAsked) this part acted on.
askSeen uint64
// namedPurged says the last pass found the build seats' named asks unused and purged them, so the next purges only
+20 -1
View File
@@ -64,6 +64,16 @@
"seat:mesh-broker.list",
"seat:mesh-recovery.kept-points"
],
"config": {
"grants-for-revisions-per-branch": {
"type": "number",
"description": "how many catalog revisions of one watched branch the grants part keeps once read for grants-for, a run's candidate counted within its branch's, the oldest dropped first; each some 1.4 MB (design 47 H H-A-195).",
"minimum": 1,
"maximum": 32,
"trusted": false,
"default": 4
}
},
"emits": [
"asserted",
"grant-written",
@@ -156,6 +166,14 @@
"place": ".",
"unknown": "program"
},
{
"id": "grants-for-revisions-per-branch",
"type": "file",
"path": "${dir:state}/grants-for-revisions-per-branch",
"owner": "mesh-grants",
"mode": "0600",
"content": "${config:grants-for-revisions-per-branch}"
},
{
"id": "grants",
"type": "process",
@@ -165,7 +183,8 @@
"./mesh-grants"
],
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker"
"MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_GRANTS_REVISIONS_PER_BRANCH_FILE": "${dir:state}/grants-for-revisions-per-branch"
},
"user": "mesh-grants",
"health": {