diff --git a/modules/artifact-store-tools/README.md b/modules/artifact-store-tools/README.md new file mode 100644 index 00000000..37ae51ec --- /dev/null +++ b/modules/artifact-store-tools/README.md @@ -0,0 +1,76 @@ +# artifact-store-tools + +Tools that say what the mesh's artifact store holds, and what the controller's records say of it +(novox/hq ADR 0251 §1–3, to-be 51). It declares no resources and changes nothing on its machine. + +## Why a module beside the store + +The store is the `distribution` module. A module that provides the artifact store cannot also build an +artifact: building publishes to the store, so the store would be needed to create itself, and the +controller's module check refuses that manifest. So the store's tools are a second module, as the +controller's refusal says to do. **Assign it to the machine that holds the store**: its tools read the +store's files through the store's own container, and anywhere else they answer that the container is not +on that machine. + +## Tools + +| tool | | what | +|---|---|---| +| `artifact_store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size | +| `artifact_store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) | +| `artifact_store_references` | r | what the controller's records say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold | +| `artifact_store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` | + +### How the store is read + +The store's door lists repositories and tags, but not a manifest no tag names, and the mesh pins every +machine by digest, so that is most of them. So the bundle reads the store's own files through its own +container (`docker exec mesh-registry`, busybox `find`, `stat` and `cat`), and only reads them: every +blob with its size, every manifest each repository holds, what each tag names, and each manifest's +content, four hundred to an exec. What it read it keeps (a manifest is named by its content, so it never +changes), and the next call reads only what is new. A manifest that could not be read is counted and +said; what it marks beyond itself is then unknown, so sizes may read low and freed bytes high, and the +answer says so. + +docker runs as the tool runner's account; a socket that refuses it is asked again through `sudo -n`, +never with a prompt, as the container runtime's own tools do. + +A manifest *marks* its own content, its configuration and its layers, and an index marks the manifests +it lists. The store's collector removes every blob no manifest marks, so these numbers are its own. + +### The states of a manifest + +| state | what | removable | +|---|---|---| +| `kept` | a definition names it, or one of the five most recent builds of a module the mesh holds; `why` says which | no | +| `holder-of-kept-archive` | the manifest that keeps a kept archive's blob (hq issue 253) | no | +| `eligible` | the mesh made it and keeps it for no reason | through the controller's `collect` | +| `holder-of-eligible-archive` | the manifest that keeps an eligible archive's blob | with its archive | +| `let-go-yet-present` | the controller recorded letting go of it, and the store still holds it | no — a finding | +| `named-document` | a one-layer manifest the controller keeps under a tag (hq to-be 45 §9) | no | +| `unrecorded` | no record names it | **never**, by any tool (ADR 0189 §3) | + +The records come from the controller's `artifacts` verb, asked with the references already let go of; +when that answer is too large to carry, it is asked without them and the answer says so. + +### Collecting + +`artifact_store_collect` never deletes anything itself. The controller decides and records what it +lets go of (ADR 0189 §2), so a real run asks its `collect` with `why` and `confirm`; the controller holds +every kept archive first and records the run as a hand-act. Bytes come back at the store's nightly +collection, which runs with the store held still. + +## Tests + +``` +go test ./... +``` + +Against a fake container that prints what the two scripts would: the listing parsed (a repository name +with slashes, a cut listing refused), manifests read with one absent or unreadable said as unread, a +digest never becoming a path outside the blobs, the mark (an index, a shared blob, a blob nothing marks), +every state, what the records keep that the store lacks, bytes freed counting a blob shared with a kept +manifest as kept, a dry run never confirming, a real run without `why` refused before anything is asked, +the fall-back when the references let go of cannot be carried, escalation through `sudo -n`, and that the +tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs. +Any command other than the two read-only scripts fails the test. diff --git a/modules/distribution/cmd/store-tools/classify.go b/modules/artifact-store-tools/cmd/artifact-store-tools/classify.go similarity index 100% rename from modules/distribution/cmd/store-tools/classify.go rename to modules/artifact-store-tools/cmd/artifact-store-tools/classify.go diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/main.go b/modules/artifact-store-tools/cmd/artifact-store-tools/main.go new file mode 100644 index 00000000..fd0ae317 --- /dev/null +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/main.go @@ -0,0 +1,31 @@ +// The artifact-store-tools module's Go bundle (novox/hq ADR 0251 §1–3, to-be 51): a process the node's +// runtime launches and speaks MCP over stdio to. It says what the artifact store holds — its +// repositories, its size, and what the controller's records say of each manifest — read from the +// store's own files through the store's own container, and writing nothing. A collection is asked of +// the controller, which decides and records what it lets go of (ADR 0189). +// +// **A module beside the store, not the store's own.** A module that provides the artifact store cannot +// also build an artifact: building publishes to the store, so the store would be needed to create +// itself, and the controller refuses that manifest. So these tools are a second module, assigned to the +// machine that holds the store. stdout is the protocol; what this bundle says, it says on stderr. +package main + +import ( + "fmt" + "os" + + stdio "git.novox.be/novox/mesh-sdk/go" +) + +func main() { + s := &Store{ + Container: os.Getenv("MESH_ARTIFACT_STORE_CONTAINER"), + Run: ExecRunner, + UID: os.Getuid(), + } + // An empty name serves as the module the runtime names (MESH_SERVED_MODULE): artifact-store-tools. + if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask})); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/read.go b/modules/artifact-store-tools/cmd/artifact-store-tools/read.go new file mode 100644 index 00000000..d3b8f2ad --- /dev/null +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/read.go @@ -0,0 +1,175 @@ +package main + +import ( + "context" + "fmt" + "sort" + "strings" + "sync" +) + +// readScript prints each manifest named on its command line, each after a line naming its path. A +// manifest is JSON, which holds no raw line break inside a string, so no line of one starts with the +// marker. One the store does not have is said, not skipped. +const readScript = `cd ` + storageRoot + ` +for p in "$@"; do + echo "#manifest $p" + if [ -f "$p" ]; then cat "$p"; echo; else echo "#absent"; fi +done +echo '#end'` + +// readBatch is how many manifests one exec reads: well under any command line's limit. +const readBatch = 400 + +// Store reaches the artifact store's own files, through its own container, and only reads them. +type Store struct { + Container string + Run Runner + UID int + + mu sync.Mutex + // cache holds every manifest read: a manifest is named by its content's digest, so it never changes. + cache map[string]*Manifest +} + +// List reads the store's files. +func (s *Store) List(ctx context.Context) (*Listing, error) { + if s.Container == "" { + return nil, fmt.Errorf("this bundle was not told the artifact store's container (MESH_ARTIFACT_STORE_CONTAINER)") + } + out, err := docker(ctx, s.Run, s.UID, "exec", s.Container, "sh", "-c", listScript) + if err != nil { + return nil, err + } + return ParseListing(out) +} + +// Key names one manifest in one repository. +func Key(repo, digest string) string { return repo + "@" + digest } + +// blobPath is where the store keeps a blob, under storageRoot. +func blobPath(digest string) string { + algorithm, hex, _ := strings.Cut(digest, ":") + if algorithm == "" || len(hex) < 2 || strings.ContainsAny(digest, "/ \n") { + return "" + } + return "blobs/" + algorithm + "/" + hex[:2] + "/" + hex + "/data" +} + +// Manifests reads every manifest the listing names, from the cache or the store's files. Answers each +// one read, and each one not read with why. +func (s *Store) Manifests(ctx context.Context, l *Listing) (map[string]*Manifest, map[string]string) { + need := map[string]bool{} + s.mu.Lock() + if s.cache == nil { + s.cache = map[string]*Manifest{} + } + for _, name := range l.RepoNames() { + for d := range l.Repos[name].Revisions { + if _, ok := s.cache[d]; !ok { + need[d] = true + } + } + } + s.mu.Unlock() + + digests := make([]string, 0, len(need)) + for d := range need { + digests = append(digests, d) + } + sort.Strings(digests) + failed := map[string]string{} + for start := 0; start < len(digests); start += readBatch { + batch := digests[start:min(start+readBatch, len(digests))] + read, err := s.read(ctx, batch) + s.mu.Lock() + for _, d := range batch { + switch m := read[d]; { + case err != nil: + failed[d] = err.Error() + case m == nil: + failed[d] = "the store holds no manifest content for it" + default: + s.cache[d] = m + } + } + s.mu.Unlock() + } + + got := map[string]*Manifest{} + unread := map[string]string{} + s.mu.Lock() + defer s.mu.Unlock() + for _, name := range l.RepoNames() { + for d := range l.Repos[name].Revisions { + if m, ok := s.cache[d]; ok { + got[Key(name, d)] = m + } else if why, ok := failed[d]; ok { + unread[Key(name, d)] = why + } else { + unread[Key(name, d)] = "not read" + } + } + } + return got, unread +} + +// read reads one batch of manifests. A blob that is not a manifest, or not there, is answered as nil; +// an exec that fails fails the batch. +func (s *Store) read(ctx context.Context, digests []string) (map[string]*Manifest, error) { + args := []string{"exec", s.Container, "sh", "-c", readScript, "sh"} + byPath := map[string]string{} + for _, d := range digests { + if p := blobPath(d); p != "" { + byPath[p] = d + args = append(args, p) + } + } + out, err := docker(ctx, s.Run, s.UID, args...) + if err != nil { + return nil, err + } + return parseManifests(out, byPath) +} + +// parseManifests reads readScript's output. +func parseManifests(out string, byPath map[string]string) (map[string]*Manifest, error) { + got := map[string]*Manifest{} + current := "" + var body strings.Builder + flush := func() { + if d := byPath[current]; d != "" { + m, err := ParseManifest([]byte(body.String())) + if err != nil { + m = nil + } + got[d] = m + } + current = "" + body.Reset() + } + ended := false + for _, line := range strings.Split(out, "\n") { + switch { + case strings.HasPrefix(line, "#manifest "): + flush() + current = strings.TrimPrefix(line, "#manifest ") + case line == "#absent": + if d := byPath[current]; d != "" { + got[d] = nil + } + current = "" + body.Reset() + case line == "#end": + flush() + ended = true + default: + body.WriteString(line) + body.WriteString("\n") + } + } + if !ended { + return nil, fmt.Errorf("reading the store's manifests was cut short (it never reached its end)") + } + return got, nil +} diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/read_test.go b/modules/artifact-store-tools/cmd/artifact-store-tools/read_test.go new file mode 100644 index 00000000..0784fa75 --- /dev/null +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/read_test.go @@ -0,0 +1,38 @@ +package main + +import "testing" + +func TestManifestsAreReadFromTheStoresFilesAndAMissingOneIsSaid(t *testing.T) { + got, err := parseManifests("#manifest blobs/sha256/aa/aa1/data\n"+image(d("cfg"), d("l"))+"\n\n#manifest blobs/sha256/bb/bb1/data\n#absent\n#manifest blobs/sha256/cc/cc1/data\nnot json\n#end\n", + map[string]string{"blobs/sha256/aa/aa1/data": "sha256:aa1", "blobs/sha256/bb/bb1/data": "sha256:bb1", "blobs/sha256/cc/cc1/data": "sha256:cc1"}) + if err != nil { + t.Fatal(err) + } + if m := got["sha256:aa1"]; m == nil || len(m.Layers) != 1 || m.Layers[0] != d("l") { + t.Errorf("aa1 %+v", m) + } + if got["sha256:bb1"] != nil || got["sha256:cc1"] != nil { + t.Error("an absent or unreadable manifest was read as one") + } + if _, err := parseManifests("#manifest x\n{}\n", map[string]string{"x": "sha256:x"}); err == nil { + t.Error("a cut read was accepted") + } + + f := world() + delete(f.manifests, d("m-old")) + v := view(t, f) + if v.Unread[Key("app/server", d("m-old"))] == "" { + t.Error("a manifest whose content is gone was not said as unread") + } + if out := Usage(v, 3); out["unread"] == nil { + t.Error("the answer did not say a manifest was unread") + } +} + +func TestADigestNeverBecomesAPathOutsideTheBlobs(t *testing.T) { + for _, bad := range []string{"sha256:", "nocolon", "sha256:a/../../etc", "sha256:ab cd"} { + if p := blobPath(bad); p != "" { + t.Errorf("%q became %q", bad, p) + } + } +} diff --git a/modules/distribution/cmd/store-tools/records.go b/modules/artifact-store-tools/cmd/artifact-store-tools/records.go similarity index 100% rename from modules/distribution/cmd/store-tools/records.go rename to modules/artifact-store-tools/cmd/artifact-store-tools/records.go diff --git a/modules/distribution/cmd/store-tools/runner.go b/modules/artifact-store-tools/cmd/artifact-store-tools/runner.go similarity index 100% rename from modules/distribution/cmd/store-tools/runner.go rename to modules/artifact-store-tools/cmd/artifact-store-tools/runner.go diff --git a/modules/distribution/cmd/store-tools/store.go b/modules/artifact-store-tools/cmd/artifact-store-tools/store.go similarity index 60% rename from modules/distribution/cmd/store-tools/store.go rename to modules/artifact-store-tools/cmd/artifact-store-tools/store.go index 62086bac..1214aedf 100644 --- a/modules/distribution/cmd/store-tools/store.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/store.go @@ -1,24 +1,21 @@ package main import ( - "context" "encoding/json" "fmt" - "io" - "net/http" "sort" "strconv" "strings" - "sync" - "time" ) -// What the store holds, read from its own files and its own door, and nothing written to either. +// What the artifact store holds, read from its own files, and nothing written to them. // // **Why its files.** The store's door lists repositories and the tags in each, but not a manifest no tag // names — and since the mesh pins every machine by digest, that is almost every manifest the store -// holds. The files list them all, with every blob's size. They are read through the store's own -// container, which is the only process that has them, and only read (novox/hq ADR 0251 §1). +// holds. The files list them all, with every blob's size, and hold every manifest's content. They are +// read through the store's own container, which is the only process that has them, and only read +// (novox/hq ADR 0251 §1). So this module runs on the machine that holds the store; anywhere else its +// tools say the container is not there. // storageRoot is where the registry keeps its files, inside its container. const storageRoot = "/var/lib/registry/docker/registry/v2" @@ -181,133 +178,3 @@ func ParseManifest(raw []byte) (*Manifest, error) { } return m, nil } - -var manifestAccept = []string{ - "application/vnd.oci.image.manifest.v1+json", - "application/vnd.oci.image.index.v1+json", - "application/vnd.docker.distribution.manifest.v2+json", - "application/vnd.docker.distribution.manifest.list.v2+json", - "application/vnd.docker.distribution.manifest.v1+prettyjws", -} - -// Store reaches the store: its files through its container, its manifests through its door. -type Store struct { - URL string - Container string - Run Runner - UID int - HTTP *http.Client - // ReadBudget bounds reading manifests in one call; what is not read in it is said. - ReadBudget time.Duration - - mu sync.Mutex - // cache holds every manifest read: a manifest is named by its content's digest, so it never changes. - cache map[string]*Manifest -} - -// List reads the store's files. -func (s *Store) List(ctx context.Context) (*Listing, error) { - if s.Container == "" { - return nil, fmt.Errorf("this bundle was not told the store's container (MESH_STORE_CONTAINER)") - } - out, err := docker(ctx, s.Run, s.UID, "exec", s.Container, "sh", "-c", listScript) - if err != nil { - return nil, err - } - return ParseListing(out) -} - -// Key names one manifest in one repository. -func Key(repo, digest string) string { return repo + "@" + digest } - -// Manifests reads every manifest the listing names, from the cache or the door, eight at a time and -// within the read budget. Answers each one read, and each one not read with why. -func (s *Store) Manifests(ctx context.Context, l *Listing) (map[string]*Manifest, map[string]string) { - type job struct{ repo, digest string } - var jobs []job - got := map[string]*Manifest{} - unread := map[string]string{} - s.mu.Lock() - if s.cache == nil { - s.cache = map[string]*Manifest{} - } - for _, name := range l.RepoNames() { - for d := range l.Repos[name].Revisions { - if m, ok := s.cache[d]; ok { - got[Key(name, d)] = m - } else { - jobs = append(jobs, job{name, d}) - } - } - } - s.mu.Unlock() - - budget := s.ReadBudget - if budget == 0 { - budget = 12 * time.Second - } - ctx, cancel := context.WithTimeout(ctx, budget) - defer cancel() - var mu sync.Mutex - work := make(chan job) - var wg sync.WaitGroup - for w := 0; w < 8; w++ { - wg.Add(1) - go func() { - defer wg.Done() - for j := range work { - m, err := s.read(ctx, j.repo, j.digest) - mu.Lock() - if err != nil { - unread[Key(j.repo, j.digest)] = err.Error() - } else { - got[Key(j.repo, j.digest)] = m - } - mu.Unlock() - } - }() - } - for _, j := range jobs { - work <- j - } - close(work) - wg.Wait() - return got, unread -} - -func (s *Store) read(ctx context.Context, repo, digest string) (*Manifest, error) { - if err := ctx.Err(); err != nil { - return nil, fmt.Errorf("not read within this call's budget") - } - req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(s.URL, "/")+"/v2/"+repo+"/manifests/"+digest, nil) - if err != nil { - return nil, err - } - for _, a := range manifestAccept { - req.Header.Add("Accept", a) - } - client := s.HTTP - if client == nil { - client = &http.Client{Timeout: 10 * time.Second} - } - resp, err := client.Do(req) - if err != nil { - return nil, fmt.Errorf("the store's door did not answer: %v", err) - } - defer resp.Body.Close() - body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20)) - if err != nil { - return nil, err - } - if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("the store's door answered %s", resp.Status) - } - m, err := ParseManifest(body) - if err != nil { - return nil, err - } - s.mu.Lock() - s.cache[digest] = m - s.mu.Unlock() - return m, nil -} diff --git a/modules/distribution/cmd/store-tools/store_test.go b/modules/artifact-store-tools/cmd/artifact-store-tools/store_test.go similarity index 88% rename from modules/distribution/cmd/store-tools/store_test.go rename to modules/artifact-store-tools/cmd/artifact-store-tools/store_test.go index b9e51314..ad9adb72 100644 --- a/modules/distribution/cmd/store-tools/store_test.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/store_test.go @@ -4,8 +4,6 @@ import ( "context" "encoding/json" "fmt" - "net/http" - "net/http/httptest" "os" "reflect" "sort" @@ -21,7 +19,7 @@ func d(name string) string { func hexOf(digest string) string { return strings.TrimPrefix(digest, "sha256:") } -// fakeStore is a store's files and its manifests: what listScript would print, and a door. +// fakeStore is a store's files and its manifests: what listScript and readScript would print. type fakeStore struct { blobs map[string]int64 revisions map[string][]string // repo -> digests @@ -51,20 +49,37 @@ func (f *fakeStore) listing() string { return b.String() } -func (f *fakeStore) door() *httptest.Server { - return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - _, digest, ok := strings.Cut(r.URL.Path, "/manifests/") - if !ok || r.Method != http.MethodGet { - http.Error(w, "no", http.StatusMethodNotAllowed) - return +// run is the store's container as docker exec reaches it: the listing, or the manifests asked for, as +// the two scripts print them. Anything else is a test failure: these tools only read. +func (f *fakeStore) run(t *testing.T) Runner { + return func(_ context.Context, name string, args ...string) Ran { + if name != "docker" || len(args) < 5 || args[0] != "exec" || args[1] != "mesh-registry" || args[2] != "sh" || args[3] != "-c" { + t.Fatalf("ran %s %v", name, args) } - body, ok := f.manifests[digest] - if !ok { - http.NotFound(w, r) - return + switch args[4] { + case listScript: + return Ran{Stdout: f.listing()} + case readScript: + var b strings.Builder + for _, p := range args[6:] { + fmt.Fprintf(&b, "#manifest %s\n", p) + found := false + for dg, body := range f.manifests { + if blobPath(dg) == p { + b.WriteString(body + "\n") + found = true + } + } + if !found { + b.WriteString("#absent\n") + } + } + b.WriteString("#end\n") + return Ran{Stdout: b.String()} } - w.Write([]byte(body)) - })) + t.Fatalf("ran a script these tools do not have: %q", args[4]) + return Ran{} + } } func image(config string, layers ...string) string { @@ -135,14 +150,7 @@ func records() *Records { func view(t *testing.T, f *fakeStore) *View { t.Helper() - door := f.door() - t.Cleanup(door.Close) - s := &Store{URL: door.URL, Container: "mesh-registry", UID: 1000, Run: func(_ context.Context, name string, args ...string) Ran { - if name != "docker" || args[0] != "exec" || args[1] != "mesh-registry" { - t.Fatalf("ran %s %v", name, args) - } - return Ran{Stdout: f.listing()} - }} + s := &Store{Container: "mesh-registry", UID: 1000, Run: f.run(t)} v, err := s.View(context.Background()) if err != nil { t.Fatal(err) @@ -306,9 +314,7 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) { func tool(t *testing.T, a *asked, name string) func(map[string]any) (any, error) { f := world() - door := f.door() - t.Cleanup(door.Close) - s := &Store{URL: door.URL, Container: "mesh-registry", Run: func(context.Context, string, ...string) Ran { return Ran{Stdout: f.listing()} }} + s := &Store{Container: "mesh-registry", Run: f.run(t)} for _, tl := range Tools(s, Controller{Ask: a.ask}) { if tl.Name == name { return tl.Run @@ -321,7 +327,7 @@ func tool(t *testing.T, a *asked, name string) func(map[string]any) (any, error) func TestADryRunNeverConfirms(t *testing.T) { a := &asked{} for _, args := range []map[string]any{{}, {"dry_run": true}, {"dry_run": true, "why": "tidy"}} { - if _, err := tool(t, a, "store_collect")(args); err != nil { + if _, err := tool(t, a, "artifact_store_collect")(args); err != nil { t.Fatal(err) } } @@ -334,13 +340,13 @@ func TestADryRunNeverConfirms(t *testing.T) { func TestARealRunWithoutWhyIsRefusedAndAsksNothing(t *testing.T) { a := &asked{} - if _, err := tool(t, a, "store_collect")(map[string]any{"dry_run": false}); err == nil { + if _, err := tool(t, a, "artifact_store_collect")(map[string]any{"dry_run": false}); err == nil { t.Fatal("a real run without why was accepted") } if len(a.calls) != 0 { t.Errorf("asked %v", a.calls) } - out, err := tool(t, a, "store_collect")(map[string]any{"dry_run": false, "why": "the store is full", "most": float64(10)}) + out, err := tool(t, a, "artifact_store_collect")(map[string]any{"dry_run": false, "why": "the store is full", "most": float64(10)}) if err != nil { t.Fatal(err) } @@ -354,7 +360,7 @@ func TestARealRunWithoutWhyIsRefusedAndAsksNothing(t *testing.T) { func TestReferencesFallBackWhenCollectedCannotBeCarried(t *testing.T) { a := &asked{fail: map[string]bool{"collected": true}} - out, err := tool(t, a, "store_references")(map[string]any{"repository": "app/server"}) + out, err := tool(t, a, "artifact_store_references")(map[string]any{"repository": "app/server"}) if err != nil { t.Fatal(err) } @@ -362,7 +368,7 @@ func TestReferencesFallBackWhenCollectedCannotBeCarried(t *testing.T) { if m["note"] == nil || m["manifests"] == nil { t.Errorf("answer %v", m) } - if _, err := tool(t, a, "store_references")(map[string]any{"repository": "nope"}); err == nil { + if _, err := tool(t, a, "artifact_store_references")(map[string]any{"repository": "nope"}); err == nil { t.Error("an unknown repository was answered") } } @@ -381,7 +387,7 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) { } var served []string for _, tl := range Tools(&Store{}, Controller{}) { - if !strings.HasPrefix(tl.Name, "store_") || tl.Description == "" || tl.Run == nil { + if !strings.HasPrefix(tl.Name, "artifact_store_") || tl.Description == "" || tl.Run == nil { t.Errorf("tool %q", tl.Name) } served = append(served, tl.Name) diff --git a/modules/distribution/cmd/store-tools/tools.go b/modules/artifact-store-tools/cmd/artifact-store-tools/tools.go similarity index 95% rename from modules/distribution/cmd/store-tools/tools.go rename to modules/artifact-store-tools/cmd/artifact-store-tools/tools.go index 936ad784..2e03b143 100644 --- a/modules/distribution/cmd/store-tools/tools.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/tools.go @@ -11,18 +11,18 @@ import ( stdio "git.novox.be/novox/mesh-sdk/go" ) -// Tools are the store module's tools (novox/hq ADR 0251 §1). The first three only read; store_collect +// Tools are the artifact store's tools (novox/hq ADR 0251 §1). The first three only read; artifact_store_collect // only reads too, unless it is a real run, and then it asks the controller, which decides and deletes. func Tools(s *Store, c Controller) []stdio.Tool { ctx := context.Background() repoArg := map[string]any{"type": "string", "description": "one repository, as the store names it (/)"} return []stdio.Tool{ { - Name: "store_repositories", + Name: "artifact_store_repositories", Description: "Every repository the artifact store holds: its tags and what each names, how many manifests it holds " + "(tagged or not — the mesh pins by digest, so most are untagged) and its size, the blobs its manifests mark. " + "A blob two repositories share is counted in each, and shared_bytes says how much of a size that is. Read from " + - "the store's own files and its door; changes nothing. Replaces curl /v2/_catalog and /v2//tags/list. (r)", + "the store's own files; changes nothing. Replaces curl /v2/_catalog and /v2//tags/list. (r)", Input: map[string]any{"repository": repoArg}, Run: func(args map[string]any) (any, error) { v, err := s.View(ctx) @@ -33,7 +33,7 @@ func Tools(s *Store, c Controller) []stdio.Tool { }, }, { - Name: "store_usage", + Name: "artifact_store_usage", Description: "How large the artifact store is: every blob's bytes together, the largest repositories, the bytes " + "more than one repository's manifests mark, and the bytes no manifest marks — what the store's nightly collector " + "frees next. Read from the store's own files; changes nothing. Replaces du on the store's directory. (r)", @@ -51,7 +51,7 @@ func Tools(s *Store, c Controller) []stdio.Tool { }, }, { - Name: "store_references", + Name: "artifact_store_references", Description: "What the controller's records say of each manifest the artifact store holds: kept (a definition " + "names it, or one of the five most recent builds of a module the mesh holds), the holder of a kept archive, " + "eligible (the mesh made it and keeps it for no reason), the holder of an eligible archive, let go yet present, " + @@ -73,7 +73,7 @@ func Tools(s *Store, c Controller) []stdio.Tool { }, }, { - Name: "store_collect", + Name: "artifact_store_collect", Description: "Collect what the mesh made and keeps for no reason (novox/hq ADR 0189, ADR 0251 §3). A dry run unless " + "dry_run is false: it asks the controller's collect what it would let go of, and works out how many bytes the " + "store's nightly collector would free once those are gone, beside what it frees tonight anyway. A real run needs " + @@ -143,7 +143,7 @@ func unreadNote(v *View) map[string]any { } } -// Repositories answers store_repositories. +// Repositories answers artifact_store_repositories. func Repositories(v *View, only string) (map[string]any, error) { shared := v.Shared() var repos []map[string]any @@ -175,7 +175,7 @@ func Repositories(v *View, only string) (map[string]any, error) { return out, nil } -// Usage answers store_usage. +// Usage answers artifact_store_usage. func Usage(v *View, top int) map[string]any { type sized struct { name string @@ -216,7 +216,7 @@ func Usage(v *View, top int) map[string]any { return out } -// References answers store_references. +// References answers artifact_store_references. func References(v *View, recs *Records, only string) (map[string]any, error) { if only != "" && v.L.Repos[only] == nil { return nil, fmt.Errorf("the store holds no repository %q", only) @@ -261,7 +261,7 @@ func References(v *View, recs *Records, only string) (map[string]any, error) { return out, nil } -// Collected answers store_collect: the controller's answer, and what the store's collector frees. +// Collected answers artifact_store_collect: the controller's answer, and what the store's collector frees. func Collected(v *View, a *CollectAnswer, dry bool) map[string]any { refs := a.LetGo if dry { diff --git a/modules/distribution/cmd/store-tools/view.go b/modules/artifact-store-tools/cmd/artifact-store-tools/view.go similarity index 100% rename from modules/distribution/cmd/store-tools/view.go rename to modules/artifact-store-tools/cmd/artifact-store-tools/view.go diff --git a/modules/distribution/go.mod b/modules/artifact-store-tools/go.mod similarity index 68% rename from modules/distribution/go.mod rename to modules/artifact-store-tools/go.mod index dc21c025..74ccd152 100644 --- a/modules/distribution/go.mod +++ b/modules/artifact-store-tools/go.mod @@ -1,4 +1,4 @@ -module distribution +module artifactstoretools go 1.22 diff --git a/modules/distribution/go.sum b/modules/artifact-store-tools/go.sum similarity index 100% rename from modules/distribution/go.sum rename to modules/artifact-store-tools/go.sum diff --git a/modules/artifact-store-tools/module.json b/modules/artifact-store-tools/module.json new file mode 100644 index 00000000..b9c008f9 --- /dev/null +++ b/modules/artifact-store-tools/module.json @@ -0,0 +1,32 @@ +{ + "module": "artifact-store-tools", + "version": "1", + "invokes": [ + "seat:mesh-controller.artifacts", + "seat:mesh-controller.collect" + ], + "tools": [ + "artifact_store_repositories", + "artifact_store_usage", + "artifact_store_references", + "artifact_store_collect" + ], + "build": { + "artifacts": [ + { + "name": "tools-go", + "kind": "bundle", + "language": "go", + "system": "arch", + "from": "cmd/artifact-store-tools", + "binary": "artifact-store-tools", + "loads": [ + "artifact-store-tools" + ], + "env": { + "MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry" + } + } + ] + } +} diff --git a/modules/distribution/README.md b/modules/distribution/README.md deleted file mode 100644 index 70d49c73..00000000 --- a/modules/distribution/README.md +++ /dev/null @@ -1,76 +0,0 @@ -# distribution - -The mesh's artifact store: an OCI registry that holds every image, mirrored upstream image, bundle and -archive the mesh delivers, by digest (novox/hq ADR 0156). It claims the mesh seat `mesh-artifact-store` -and provides `artifact-store`. - -## What it declares - -| resource | what | -|---|---| -| `state`, `registry-data` | the module's state directory and the store's files | -| `store` | the registry, with deletion enabled on its one door (ADR 0189 §1) | -| `collect` | the registry's own collector, nightly at 03:30, with `store` held still while it runs (ADR 0189 §4) | -| `tools-go` | the Go bundle `store-tools`, below | - -The mesh decides what the store may let go of, from its build records, and lets go of it after each -build it records (ADR 0189). The store's collector reclaims the bytes each night. - -## Tools (novox/hq ADR 0251, to-be 51) - -| tool | | what | -|---|---|---| -| `store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size | -| `store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) | -| `store_references` | r | what the controller's records say of each manifest the store holds, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold | -| `store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` | - -### How the store is read - -The store's door lists repositories and tags, but not a manifest no tag names, and the mesh pins every -machine by digest, so that is most of them. So the bundle lists the store's own files through its own -container (`docker exec mesh-registry`, busybox `find` and `stat`), read-only: every blob with its size, -every manifest each repository holds, what each tag names. It reads each manifest's content through -the door, eight at a time and within a budget per call, and keeps what it read (a manifest never -changes: it is named by its content). A manifest that could not be read is counted and said; what it -marks beyond itself is then unknown, so sizes may read low and freed bytes high, and the answer says so. - -The docker command runs as the tool runner's account; a socket that refuses it is asked again through -`sudo -n`, never with a prompt, as the container runtime's own tools do. - -A manifest *marks* its own content, its configuration and its layers, and an index marks the manifests -it lists. The store's collector removes every blob no manifest marks, so these numbers are its own. - -### The states of a manifest - -| state | what | removable | -|---|---|---| -| `kept` | a definition names it, or one of the five most recent builds of a module the mesh holds; `why` says which | no | -| `holder-of-kept-archive` | the manifest that keeps a kept archive's blob (hq issue 253) | no | -| `eligible` | the mesh made it and keeps it for no reason | through the controller's `collect` | -| `holder-of-eligible-archive` | the manifest that keeps an eligible archive's blob | with its archive | -| `let-go-yet-present` | the controller recorded letting go of it, and the store still holds it | no — a finding | -| `named-document` | a one-layer manifest the controller keeps under a tag (hq to-be 45 §9) | no | -| `unrecorded` | no record names it | **never**, by any tool (ADR 0189 §3) | - -The records come from the controller's `artifacts` verb, asked with the references already let go of; -when that answer is too large to carry, it is asked without them and the answer says so. - -### Collecting - -`store_collect` never deletes through the store's door. The controller decides and records what it -lets go of (ADR 0189 §2), so a real run asks its `collect` with `why` and `confirm`; the controller holds -every kept archive first and records the run as a hand-act. Bytes come back at the nightly collection. - -## Tests - -``` -go test ./... -``` - -Against a fake listing and a fake door: the listing parsed (a repository name with slashes, a cut -listing refused), the mark (an index, a shared blob, a blob nothing marks), every state, what the -records keep that the store lacks, bytes freed counting a blob shared with a kept manifest as kept, a -dry run never confirming, a real run without `why` refused before anything is asked, the fall-back when -the references let go of cannot be carried, escalation through `sudo -n`, and that the tools served are -exactly the manifest's `tools` and its `invokes` exactly the two controller verbs. diff --git a/modules/distribution/cmd/store-tools/main.go b/modules/distribution/cmd/store-tools/main.go deleted file mode 100644 index df45280f..00000000 --- a/modules/distribution/cmd/store-tools/main.go +++ /dev/null @@ -1,31 +0,0 @@ -// The distribution module's Go bundle (novox/hq ADR 0251 §1–3, to-be 51): a process the node's runtime -// launches and speaks MCP over stdio to. It says what the artifact store holds — its repositories, its -// size, and what the controller's records say of each manifest — read from the store's own files -// through its own container and from its door, and writing to neither. A collection is asked of the -// controller, which decides and records what it lets go of (ADR 0189). stdout is the protocol; what -// this bundle says, it says on stderr. -package main - -import ( - "fmt" - "os" - - stdio "git.novox.be/novox/mesh-sdk/go" -) - -func main() { - s := &Store{ - URL: os.Getenv("MESH_STORE_URL"), - Container: os.Getenv("MESH_STORE_CONTAINER"), - Run: ExecRunner, - UID: os.Getuid(), - } - if s.URL == "" { - fmt.Fprintln(os.Stderr, "[store-tools] MESH_STORE_URL is not set: every manifest will be said as unread") - } - // An empty name serves as the module the runtime names (MESH_SERVED_MODULE): distribution. - if err := stdio.Serve("", Tools(s, Controller{Ask: stdio.Ask})); err != nil { - fmt.Fprintln(os.Stderr, err) - os.Exit(1) - } -} diff --git a/modules/distribution/module.json b/modules/distribution/module.json index f97e9a98..e368b824 100644 --- a/modules/distribution/module.json +++ b/modules/distribution/module.json @@ -16,16 +16,6 @@ "capabilities": [ "container-runtime" ], - "invokes": [ - "seat:mesh-controller.artifacts", - "seat:mesh-controller.collect" - ], - "tools": [ - "store_repositories", - "store_usage", - "store_references", - "store_collect" - ], "own-secrets": { "broker": "/var/lib/mesh/registry/broker" }, @@ -100,24 +90,5 @@ "store" ] } - ], - "build": { - "artifacts": [ - { - "name": "tools-go", - "kind": "bundle", - "language": "go", - "system": "arch", - "from": "cmd/store-tools", - "binary": "store-tools", - "loads": [ - "store-tools" - ], - "env": { - "MESH_STORE_URL": "http://127.0.0.1:${port:5000}", - "MESH_STORE_CONTAINER": "mesh-registry" - } - } - ] - } + ] }