From 1bf98663c67b6e183f32c91bbf681b7fe141a443 Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 4 Nov 2026 21:11:15 +0100 Subject: [PATCH] =?UTF-8?q?artifact-store-tools:=20read=20the=20store=20th?= =?UTF-8?q?rough=20the=20registry=20seats'=20list,=20never=20its=20files?= =?UTF-8?q?=20or=20its=20read=20side=20(ADR=200632=20=C2=A71,=20=C2=A74;?= =?UTF-8?q?=20issue=201966)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docker exec listing and the HTTP door go; held and the sweep's index skip ask the seat's list too. The store's address file is the image seat's reach, used for the catalogue's definitions only. Claude-Session: https://claude.ai/code/session_01JG73KJNrH5EgFXMKKYSrig --- modules/artifact-store-tools/README.md | 40 ++-- .../artifact-store-tools/definitions_test.go | 2 +- .../cmd/artifact-store-tools/door.go | 49 +++-- .../cmd/artifact-store-tools/holder.go | 10 +- .../cmd/artifact-store-tools/holder_test.go | 4 +- .../cmd/artifact-store-tools/listed.go | 166 ++++++++++++++ .../cmd/artifact-store-tools/main.go | 17 +- .../cmd/artifact-store-tools/store_test.go | 206 ++++++++---------- .../cmd/artifact-store-tools/tools.go | 22 +- .../cmd/artifact-store-tools/view.go | 4 +- .../artifact-store-tools/files/store-address | 2 +- modules/artifact-store-tools/module.json | 5 +- 12 files changed, 333 insertions(+), 194 deletions(-) create mode 100644 modules/artifact-store-tools/cmd/artifact-store-tools/listed.go diff --git a/modules/artifact-store-tools/README.md b/modules/artifact-store-tools/README.md index 2cfbce8cf..cc662262c 100644 --- a/modules/artifact-store-tools/README.md +++ b/modules/artifact-store-tools/README.md @@ -10,34 +10,30 @@ The store is the `distribution` module, the registry seats' holder. These tools indexer's judge refused a store holder that builds an artifact; that rule is gone (issue 1966), and the sweep stays its own module, holding the seat `mesh-artifact-sweep`. It requires no provision: what it holds, releases and removes it asks the registry seats `oci-artifact-registry` and `container-image-registry` (see *The sweep through the registry -seats*). **Assign it to the machine that holds the store**: its read tools read the store's files through the store's -own container, and anywhere else they answer that the container is not on that machine. +seats*), and what the store holds it reads through the same seats' `list` (novox/hq ADR 0632 §4): it reads neither the +store's files nor its read side, so it runs on any machine. ## Tools | tool | | what | |---|---|---| | `artifact_store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size | -| `artifact_store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) | +| `artifact_store_usage` | r | the store's size, each manifest's as its seat says it, and the largest repositories | | `artifact_store_references` | r | what the controller's records and the catalogue's definitions say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold | | `artifact_store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` | ### How the store is read -The store's door lists repositories and tags, but not a manifest no tag names, and the mesh pins every -machine by digest, so that is most of them. So the bundle reads the store's own files through its own -container (`docker exec mesh-registry`, busybox `find`, `stat` and `cat`), and only reads them: every -blob with its size, every manifest each repository holds, what each tag names, and each manifest's -content, four hundred to an exec. What it read it keeps (a manifest is named by its content, so it never -changes), and the next call reads only what is new. A manifest that could not be read is counted and -said; what it marks beyond itself is then unknown, so sizes may read low and freed bytes high, and the -answer says so. +The store's read side lists repositories and tags, but not a manifest no tag names, and the mesh pins every +machine by digest, so that is most of them. So the bundle asks the two registry seats' `list` +(`container-image-registry` for images and indexes, `oci-artifact-registry` for bundles and archives), which the +store's holder answers from the store's own files: each manifest's digest, whether it is an index and the manifests it +names, the tags naming it, its size and whether a hold keeps it (novox/hq ADR 0632 §4). -docker runs as the tool runner's account; a socket that refuses it is asked again through `sudo -n`, -never with a prompt, as the container runtime's own tools do. - -A manifest *marks* its own content, its configuration and its layers, and an index marks the manifests -it lists. The store's collector removes every blob no manifest marks, so these numbers are its own. +A manifest's size is its own content and every blob it names, as its seat says it, so a layer two manifests share +is counted in each, and the bytes the store's nightly collector frees are the sizes of the manifests that go, an +upper bound. The address the bundle says the store is at comes from `${seat:container-image-registry:reach}`, the +image seat holder's listen (ADR 0632 §1), and is used for the catalogue's definitions only, never to reach the store. ### The states of a manifest @@ -99,16 +95,14 @@ manifest that stays and would lose a platform it names — to be read before a r go test ./... ``` -Against a fake container that prints what the two scripts would: the listing parsed (a repository name -with slashes, a cut listing refused), manifests read with one absent or unreadable said as unread, a -digest never becoming a path outside the blobs, the mark (an index, a shared blob, a blob nothing marks), -every state, what the records keep that the store lacks, bytes freed counting a blob shared with a kept -manifest as kept, a dry run never confirming, a real run without `why` refused before anything is asked, +Against a fake of the two registry seats' `list`: the store read through it (an index and the manifests it names, a +bundle's holder, a hold, the seat's sizes), every state, what the records keep that the store lacks, bytes freed as the +sizes of what goes, a dry run never confirming, a real run without `why` refused before anything is asked, the fall-back when the references let go of cannot be carried, a manifest a definition pins under the -store's current or an older name, a definition not read said first, escalation through `sudo -n`, and that the +store's current or an older name, a definition not read said first, and that the tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs and the catalogue's two tools. -Any command other than the two read-only scripts fails the test. +Any ask other than a seat's `list` fails the read tools' tests. ## Files it does not know diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/definitions_test.go b/modules/artifact-store-tools/cmd/artifact-store-tools/definitions_test.go index c46cb12ca..739c904d2 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/definitions_test.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/definitions_test.go @@ -152,7 +152,7 @@ func TestOlderNamesNotReadAreSaidFirst(t *testing.T) { // references is one artifact_store_references tool, kept across calls as the runtime keeps the bundle. func references(t *testing.T, a *asked) func(map[string]any) (any, error) { f := world() - s := &Store{Container: "mesh-registry", Run: f.run(t)} + s := &Store{Ask: f.ask(t)} for _, tl := range Tools(s, recordsAsked{a}, &Catalogue{Ask: a.ask}) { if tl.Name == "artifact_store_references" { return tl.Run diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/door.go b/modules/artifact-store-tools/cmd/artifact-store-tools/door.go index 7e7177c3d..e9292f5fe 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/door.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/door.go @@ -1,10 +1,11 @@ package main -// The artifact store's door and the bus, as the holder reaches them through this module's runtime. +// The artifact store, through its seats, and the bus, as the holder reaches them through this module's runtime. import ( "context" "encoding/json" + "errors" "fmt" "os" "strings" @@ -14,23 +15,43 @@ import ( stdio "git.novox.be/novox/mesh-sdk/go" "git.novox.be/novox/mesh-sdk/go/views" - - "artifactstoretools/internal/registry" ) -// door is registry.Store at the store's address as this machine reaches it. -type door struct{ s registry.Store } +// Gone is the answer when the store does not hold it: the outcome wanted, already true. +var Gone = errors.New("the store does not hold it") -// It is read only: what the sweep holds and removes it asks the registry seats (sweepseats.go). -func (d *door) Address() string { return d.s.Address } -func (d *door) Holds(ctx context.Context, r string) error { return d.s.Holds(ctx, r) } -func (d *door) IsIndex(ctx context.Context, r string) (bool, error) { - return d.s.IsIndex(ctx, r) +// ErrNotOurs is a reference this sweep will not address: not the mesh's own, or naming nothing the store holds by digest. +// +// **A fact about the record, not about the store** (novox/hq issue 226). The two deserve opposite responses (skip one and +// go on, abandon the sweep for the other), and collapsing them into "an error" is how a cautious loop became one that did +// nothing while reporting the right number. +var ErrNotOurs = errors.New("not a reference into the mesh's artifact store") + +// door is the store as its seats' list answers it (ADR 0632 §4), at the address this machine reaches it by: it reads +// only; what the sweep holds and removes it asks the registry seats too (sweepseats.go). +type door struct { + address string + store *Store } -// DoorAt reads the store's address (host:port) from the file the manifest fills from the seat mesh-artifact-store's -// reach, each time, so a store that moved is reached where it is now. -func DoorAt(file string) func() (Door, error) { +func (d *door) Address() string { return d.address } + +// Holds is whether the store holds what a reference names: an image's manifest, a bundle's or an archive's blob. Gone when +// its seat lists none (ADR 0526 clause 3: a registration by hand is refused at bytes that are not there). +func (d *door) Holds(_ context.Context, r string) error { + _, err := d.store.listed(r) + return err +} + +// IsIndex is whether a reference is an index the store holds, as its seat lists it. Gone when it does not hold it. +func (d *door) IsIndex(_ context.Context, r string) (bool, error) { + e, err := d.store.listed(r) + return e.Index, err +} + +// DoorAt reads the store's address (host:port) from the file the manifest fills from the seat container-image-registry's +// reach, each time, so a store that moved is said where it is now; the store is read through s. +func DoorAt(file string, s *Store) func() (Door, error) { return func() (Door, error) { if file == "" { return nil, fmt.Errorf("this bundle was given no file naming the artifact store's address") @@ -43,7 +64,7 @@ func DoorAt(file string) func() (Door, error) { if address == "" { return nil, fmt.Errorf("the artifact store's address is empty in %s", file) } - return &door{s: registry.Store{Address: address}}, nil + return &door{address: address, store: s}, nil } } diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/holder.go b/modules/artifact-store-tools/cmd/artifact-store-tools/holder.go index 2293c86ad..9169218bc 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/holder.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/holder.go @@ -30,8 +30,6 @@ import ( "git.novox.be/novox/mesh-sdk/go/health" "git.novox.be/novox/mesh-sdk/go/views" - - "artifactstoretools/internal/registry" ) // What the holder reads and keeps. @@ -71,7 +69,7 @@ type Bus interface { Publish(subject string, body any) error } -// Door is the artifact store's own door, read only: registry.Store, behind an interface so a test needs no store. What +// Door is the artifact store as its seats' list answers it, read only, behind an interface so a test needs no store. What // the sweep holds and removes it asks the registry seats (Seats, sweepseats.go). type Door interface { Address() string @@ -83,7 +81,7 @@ type Door interface { type Holder struct { Bus Bus Door func() (Door, error) - // Listed is the store's own listing now (Store.View); nil reads none, and then nothing is said gone or collected. + // Listed is the store as its seats list it now (Store.View); nil reads none, and then nothing is said gone or collected. Listed func(ctx context.Context) (*View, error) Acts handacts.Recorder // Seats asks a registry seat's verb, `seat:.` (hold, remove): the runtime's stdio.Ask in main. @@ -576,7 +574,7 @@ func (h *Holder) Sweep(ctx context.Context, s Sweeping) (*CollectAnswer, error) if !s.Confirmed { // An index waits for a confirmed collect (ADR 0251 §7, ADR 0257 §4). index, err := door.IsIndex(within, r) - if err != nil && !errors.Is(err, registry.Gone) && !errors.Is(err, registry.ErrNotOurs) { + if err != nil && !errors.Is(err, Gone) && !errors.Is(err, ErrNotOurs) { a.Stopped = fmt.Sprintf("the store could not say whether %s is an index, so nothing more was asked: %v", r, err) a.Left = len(eligible) - i break @@ -690,7 +688,7 @@ func (h *Holder) Held(refs []string) (delivery.HeldAnswer, error) { switch err := door.Holds(asking, canonical(r)); { case err == nil: a.Held = append(a.Held, r) - case errors.Is(err, registry.Gone): + case errors.Is(err, Gone): a.Gone = append(a.Gone, r) default: if a.Unknown == nil { diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/holder_test.go b/modules/artifact-store-tools/cmd/artifact-store-tools/holder_test.go index 37b7d553b..533258cc3 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/holder_test.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/holder_test.go @@ -18,8 +18,6 @@ import ( "git.novox.be/novox/mesh-sdk/go/health" "git.novox.be/novox/mesh-sdk/go/catalogue" - - "artifactstoretools/internal/registry" ) // memBus is the holder's bus in memory: states by name, each key's history, and what was published. @@ -141,7 +139,7 @@ func (d *memDoor) seat(key string, body any) (json.RawMessage, error) { func (d *memDoor) Address() string { return "10.0.0.5:5000" } func (d *memDoor) Holds(_ context.Context, r string) error { if !d.holds[r] { - return registry.Gone + return Gone } return nil } diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/listed.go b/modules/artifact-store-tools/cmd/artifact-store-tools/listed.go new file mode 100644 index 000000000..131dca3a1 --- /dev/null +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/listed.go @@ -0,0 +1,166 @@ +package main + +// **The store, read through its seats** (novox/hq ADR 0632 §4): the read tools and the sweep read what the store holds +// from the seats' `list`, container-image-registry's images and indexes and oci-artifact-registry's bundles and archives, +// never from the store's files or its read side. What a manifest marks beyond itself is the seat's size of it, so a +// blob two manifests share is counted in each, and what the store's nightly collector frees is not known before it runs. + +import ( + "context" + "encoding/json" + "fmt" + "sort" + + ociseat "git.novox.be/novox/mesh-contracts/go/seats/ociartifactregistry" + + reg "git.novox.be/novox/mesh-contracts/go/seats/containerimageregistry" +) + +// Repo is one repository as the seats' list answers it. +type Repo struct { + Name string + // Revisions are the digests of every manifest it holds, tagged or not. + Revisions map[string]bool + // Tags are what each tag names now. + Tags map[string]string +} + +// Listing is the store as the seats' list answers it: every repository, and a size for each manifest and each +// bundle's or archive's blob, keyed by its digest. A manifest's size is its own content and every blob it names, as the +// seat answers it, so a layer two manifests share is counted in each. +type Listing struct { + Blobs map[string]int64 + Repos map[string]*Repo +} + +func (l *Listing) repo(name string) *Repo { + r := l.Repos[name] + if r == nil { + r = &Repo{Name: name, Revisions: map[string]bool{}, Tags: map[string]string{}} + l.Repos[name] = r + } + return r +} + +// RepoNames are the repositories in order. +func (l *Listing) RepoNames() []string { + out := make([]string, 0, len(l.Repos)) + for n := range l.Repos { + out = append(out, n) + } + sort.Strings(out) + return out +} + +// Manifest is what one manifest names. +type Manifest struct { + MediaType string + ConfigMedia string + // Refs are the blobs it names: its configuration and its layers. + Refs []string + // Layers are its layers alone, in order. + Layers []string + // Children are the manifests an index names. + Children []string +} + +// Holder is the shape of a manifest that keeps one blob: an empty configuration and one layer. The +// controller holds every archive the mesh keeps this way (novox/hq issue 253), and keeps a named +// document the same way, under a tag. +func (m *Manifest) Holder() bool { + return m != nil && m.ConfigMedia == "application/vnd.oci.empty.v1+json" && len(m.Layers) == 1 +} + +// Key names one manifest in one repository. +func Key(repo, digest string) string { return repo + "@" + digest } + +// Store is the artifact store as its two seats' `list` answers it: Ask is the runtime's stdio.Ask in main. +type Store struct { + Ask func(key string, body any) (json.RawMessage, error) +} + +// ListOf asks seat's list, of one repository or, with name empty, of every one. +func (s *Store) ListOf(seat, name string) ([]reg.Repository, error) { + if s == nil || s.Ask == nil { + return nil, fmt.Errorf("this process reaches no registry seat") + } + raw, err := s.Ask("seat:"+seat+"."+reg.VerbList, reg.ListInput{Name: name}) + if err != nil { + return nil, fmt.Errorf("%s's list: %w", seat, err) + } + var a reg.ListAnswer + if err := json.Unmarshal(raw, &a); err != nil { + return nil, fmt.Errorf("%s's answer to list: %w", seat, err) + } + return a.Repositories, nil +} + +// View reads the store now, through both seats. +func (s *Store) View(context.Context) (*View, error) { + images, err := s.ListOf(reg.Seat, "") + if err != nil { + return nil, err + } + artifacts, err := s.ListOf(ociseat.Seat, "") + if err != nil { + return nil, err + } + return ViewOf(images, artifacts), nil +} + +// ViewOf is the store as the two seats list it: each manifest a repository's revision, with its tags; an index naming its +// platform manifests; a bundle's or archive's manifest keeping its one blob, so it reads as the holder of that archive. +func ViewOf(seats ...[]reg.Repository) *View { + l := &Listing{Blobs: map[string]int64{}, Repos: map[string]*Repo{}} + m := map[string]*Manifest{} + held := map[string]bool{} + for _, repos := range seats { + for _, r := range repos { + repo := l.repo(r.Name) + for _, e := range r.Manifests { + repo.Revisions[e.Digest] = true + for _, t := range e.Tags { + repo.Tags[t] = e.Digest + } + doc := &Manifest{Children: append([]string(nil), e.Platforms...)} + l.Blobs[e.Digest] = e.Size + if e.Artifact != "" { + // The holder's size is the archive's with it: the blob itself counts nothing more. + doc.ConfigMedia, doc.Layers, doc.Refs = emptyConfig, []string{e.Artifact}, []string{e.Artifact} + if _, sized := l.Blobs[e.Artifact]; !sized { + l.Blobs[e.Artifact] = 0 + } + } + m[Key(r.Name, e.Digest)] = doc + if e.Held { + held[Key(r.Name, e.Digest)] = true + } + } + } + } + return &View{L: l, M: m, Unread: map[string]string{}, Held: held} +} + +// emptyConfig is the configuration media type of a manifest keeping one blob. +const emptyConfig = "application/vnd.oci.empty.v1+json" + +// listed is the one entry of a reference's seat list naming it: by its manifest's digest, or a bundle's or an archive's by +// the blob its reference names. Gone when the seat lists none. +func (s *Store) listed(reference string) (reg.Listed, error) { + seat, name, digest, err := seatOf(reference) + if err != nil { + return reg.Listed{}, fmt.Errorf("%w: %v", ErrNotOurs, err) + } + repos, err := s.ListOf(seat, name) + if err != nil { + return reg.Listed{}, err + } + for _, r := range repos { + for _, e := range r.Manifests { + if (seat == ociseat.Seat && e.Artifact == digest) || (seat != ociseat.Seat && e.Digest == digest) { + return e, nil + } + } + } + return reg.Listed{}, Gone +} diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/main.go b/modules/artifact-store-tools/cmd/artifact-store-tools/main.go index 8d53a2088..05615df12 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/main.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/main.go @@ -1,11 +1,10 @@ // The artifact-store-tools module's Go bundle (novox/hq ADR 0251 §1–3, ADR 0571 clause 2, to-be 51): a process the // node's runtime launches and speaks MCP over stdio to. It says what the artifact store holds — its repositories, its -// size, and what the mesh keeps of each manifest — read from the store's own files through the store's own container, -// and it holds the seat mesh-artifact-sweep: the keep set, the sweep at each build's end, and S25's image-missing. +// size, and what the mesh keeps of each manifest — read through the registry seats' list (novox/hq ADR 0632 §4), and it +// holds the seat mesh-artifact-sweep: the keep set, the sweep at each build's end, and S25's image-missing. // -// **A module beside the store, not the store's own.** It requires no provision: what it holds and removes it asks the -// registry seats (sweepseats.go), and it is assigned to the machine that holds the store, whose container its read -// tools read through. stdout is the protocol; what this bundle says, it says on stderr. +// **A module beside the store, not the store's own.** It requires no provision: what it reads, holds and removes it asks +// the registry seats (listed.go, sweepseats.go), never the store's files or its read side. stdout is the protocol; what this bundle says, it says on stderr. package main import ( @@ -24,15 +23,11 @@ import ( func logf(format string, a ...any) { fmt.Fprintf(os.Stderr, format+"\n", a...) } func main() { - s := &Store{ - Container: os.Getenv("MESH_ARTIFACT_STORE_CONTAINER"), - Run: ExecRunner, - UID: os.Getuid(), - } + s := &Store{Ask: stdio.Ask} v := &views.Reader{Module: "artifact-store-tools", Logf: logf, Findings: views.OwnFindings{}, Acts: handacts.Recorder{Ask: stdio.Ask}} v.Follow(delivery.AssignerModule, "wants", WantsTool) v.Follow(delivery.IndexerModule, "manifests", ManifestsTool) - h := &Holder{Bus: runtimeBus{views: v}, Door: DoorAt(os.Getenv("MESH_ARTIFACT_STORE_ADDRESS_FILE")), Listed: s.View, + h := &Holder{Bus: runtimeBus{views: v}, Door: DoorAt(os.Getenv("MESH_ARTIFACT_STORE_ADDRESS_FILE"), s), Listed: s.View, Seats: stdio.Ask, Now: time.Now, Logf: logf} h.Acts = handacts.Recorder{Ask: stdio.Ask} v.Start(context.Background()) diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/store_test.go b/modules/artifact-store-tools/cmd/artifact-store-tools/store_test.go index 3d9c743c4..93b7cfbc5 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/store_test.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/store_test.go @@ -1,6 +1,7 @@ package main import ( + reg "git.novox.be/novox/mesh-contracts/go/seats/containerimageregistry" "git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/views" @@ -24,66 +25,72 @@ func d(name string) string { func hexOf(digest string) string { return strings.TrimPrefix(digest, "sha256:") } -// fakeStore is a store's files and its manifests: what listScript and readScript would print. +// fakeStore is a store's blobs and manifests, as the two registry seats' list answers them (ADR 0632 §4). type fakeStore struct { blobs map[string]int64 revisions map[string][]string // repo -> digests tags map[string]map[string]string manifests map[string]string // digest -> content + held map[string]bool // Key(repo, digest) -> held } -func (f *fakeStore) listing() string { - var b strings.Builder - b.WriteString("#blobs\n") - for dg, n := range f.blobs { - fmt.Fprintf(&b, "%d blobs/sha256/%s/%s/data\n", n, hexOf(dg)[:2], hexOf(dg)) - } - b.WriteString("#revisions\n") - for repo, ds := range f.revisions { - for _, dg := range ds { - fmt.Fprintf(&b, "repositories/%s/_manifests/revisions/sha256/%s/link\n", repo, hexOf(dg)) +// ask is the seats as the runtime asks them: list on each, every repository or one; anything else fails the test, as +// these tools only read. +func (f *fakeStore) ask(t *testing.T) func(string, any) (json.RawMessage, error) { + return func(key string, body any) (json.RawMessage, error) { + seat, verb, _ := strings.Cut(strings.TrimPrefix(key, "seat:"), ".") + if verb != "list" || (seat != "container-image-registry" && seat != "oci-artifact-registry") { + t.Fatalf("asked %s", key) } - } - b.WriteString("#tags\n") - for repo, ts := range f.tags { - for t, dg := range ts { - fmt.Fprintf(&b, "repositories/%s/_manifests/tags/%s/current/link:%s\n", repo, t, dg) + raw, _ := json.Marshal(body) + var in reg.ListInput + _ = json.Unmarshal(raw, &in) + repos := []reg.Repository{} + names := make([]string, 0, len(f.revisions)) + for n := range f.revisions { + names = append(names, n) } - } - b.WriteString("#end\n") - return b.String() -} - -// run is the store's container as docker exec reaches it: the listing, or the manifests asked for, as -// the two scripts print them. Anything else is a test failure: these tools only read. -func (f *fakeStore) run(t *testing.T) Runner { - return func(_ context.Context, name string, args ...string) Ran { - if name != "docker" || len(args) < 5 || args[0] != "exec" || args[1] != "mesh-registry" || args[2] != "sh" || args[3] != "-c" { - t.Fatalf("ran %s %v", name, args) - } - switch args[4] { - case listScript: - return Ran{Stdout: f.listing()} - case readScript: - var b strings.Builder - for _, p := range args[6:] { - fmt.Fprintf(&b, "#manifest %s\n", p) - found := false - for dg, body := range f.manifests { - if blobPath(dg) == p { - b.WriteString(body + "\n") - found = true + sort.Strings(names) + for _, name := range names { + if in.Name != "" && in.Name != name { + continue + } + var ms []reg.Listed + for _, dg := range f.revisions[name] { + var doc struct { + Config *struct{ MediaType, Digest string } + Layers []struct{ Digest string } + Manifests []struct{ Digest string } + } + _ = json.Unmarshal([]byte(f.manifests[dg]), &doc) + e := reg.Listed{Digest: dg, Size: f.blobs[dg], Held: f.held[Key(name, dg)]} + if doc.Config != nil { + e.Size += f.blobs[doc.Config.Digest] + } + for _, l := range doc.Layers { + e.Size += f.blobs[l.Digest] + } + for _, c := range doc.Manifests { + e.Platforms = append(e.Platforms, c.Digest) + } + e.Index = len(e.Platforms) > 0 + if doc.Config != nil && doc.Config.MediaType == "application/vnd.oci.empty.v1+json" && len(doc.Layers) == 1 { + e.Artifact = doc.Layers[0].Digest + } + for tag, at := range f.tags[name] { + if at == dg { + e.Tags = append(e.Tags, tag) } } - if !found { - b.WriteString("#absent\n") + if (e.Artifact != "") == (seat == "oci-artifact-registry") { + ms = append(ms, e) } } - b.WriteString("#end\n") - return Ran{Stdout: b.String()} + if len(ms) > 0 { + repos = append(repos, reg.Repository{Name: name, Manifests: ms}) + } } - t.Fatalf("ran a script these tools do not have: %q", args[4]) - return Ran{} + return json.Marshal(reg.ListAnswer{Repositories: repos}) } } @@ -155,7 +162,7 @@ func records() *Records { func view(t *testing.T, f *fakeStore) *View { t.Helper() - s := &Store{Container: "mesh-registry", UID: 1000, Run: f.run(t)} + s := &Store{Ask: f.ask(t)} v, err := s.View(context.Background()) if err != nil { t.Fatal(err) @@ -170,50 +177,34 @@ func withFill(r *Records) *Records { return r } -func TestTheListingIsRead(t *testing.T) { - l, err := ParseListing("#blobs\n12 blobs/sha256/ab/abcd/data\n#revisions\nrepositories/a/b/c/_manifests/revisions/sha256/ff/link\n#tags\nrepositories/a/b/c/_manifests/tags/v1/current/link:sha256:ff\n#end\n") - if err != nil { - t.Fatal(err) +// The store is read through its two seats' list (ADR 0632 §4): every manifest of every repository, tagged or not, an +// index with the manifests it names, a bundle's or archive's manifest as the holder of its blob; each manifest's size is +// the seat's, so nothing is unmarked before a manifest goes, and a manifest gone frees its size. +func TestTheStoreIsReadThroughItsSeats(t *testing.T) { + f := world() + f.held = map[string]bool{Key("app/tools", d("h-kept")): true} + v := view(t, f) + server := v.L.Repos["app/server"] + if server == nil || len(server.Revisions) != 4 || server.Tags["latest"] != d("m-stranger") { + t.Fatalf("app/server reads %+v", server) } - if l.Blobs["sha256:abcd"] != 12 { - t.Errorf("blobs %v", l.Blobs) + if m := v.M[Key("app/server", d("idx"))]; m == nil || !reflect.DeepEqual(m.Children, []string{d("m-kept")}) { + t.Errorf("the index names %+v", m) } - r := l.Repos["a/b/c"] - if r == nil || !r.Revisions["sha256:ff"] || r.Tags["v1"] != "sha256:ff" { - t.Errorf("repository with slashes misread: %+v", r) + if m := v.M[Key("app/tools", d("h-kept"))]; !m.Holder() || m.Layers[0] != d("a-kept") { + t.Errorf("the holder of a-kept reads %+v", m) } -} - -func TestACutListingIsRefused(t *testing.T) { - for _, out := range []string{"#blobs\n12 blobs/sha256/ab/abcd/data\n", "#blobs\nnot a line\n#end\n", "stray\n#end\n"} { - if _, err := ParseListing(out); err == nil { - t.Errorf("accepted %q", out) - } + if _, held := v.L.Blobs[d("a-kept")]; !held || !v.Held[Key("app/tools", d("h-kept"))] { + t.Errorf("a-kept is not read as held by the store, or its hold is not read: %v", v.Held) } -} - -func TestTheMarkIsTheCollectorsIncludingAnIndexAndSharedBlobs(t *testing.T) { - v := view(t, world()) - marked := v.Marked(nil) - for _, name := range []string{"cfg", "shared", "old-only", "stranger-layer", "a-kept", "a-old", "doc-body", "empty", "m-kept", "idx"} { - if !marked[d(name)] { - t.Errorf("%s not marked", name) - } + if want := int64(len(f.manifests[d("m-old")])) + 10 + 1000 + 500; v.L.Blobs[d("m-old")] != want { + t.Errorf("m-old's size %d, want %d (its content and every blob it names)", v.L.Blobs[d("m-old")], want) } - if marked[d("orphan")] { - t.Error("a blob no manifest names is marked") + if freed, n := v.Unmarked(nil); freed != 0 || n != 0 { + t.Errorf("unmarked %d in %d before anything goes", freed, n) } - freed, n := v.Unmarked(nil) - if freed != 7000 || n != 1 { - t.Errorf("the collector frees %d in %d, want 7000 in 1", freed, n) - } - // The index alone gone frees its own content and nothing m-kept still marks. - _, n = v.Unmarked(map[string]bool{Key("app/server", d("idx")): true}) - if n != 2 { - t.Errorf("without the index %d blobs unmarked, want 2 (orphan and the index itself)", n) - } - if sh := v.Shared(); !sh[d("empty")] || sh[d("cfg")] { - t.Error("shared misjudged") + if freed, n := v.Unmarked(map[string]bool{Key("app/server", d("idx")): true}); freed != v.L.Blobs[d("idx")] || n != 1 { + t.Errorf("the index gone frees %d in %d, want its size", freed, n) } } @@ -257,16 +248,15 @@ func TestAStateThisBundleDoesNotKnowIsKept(t *testing.T) { } } -func TestBytesFreedCountASharedBlobAsKept(t *testing.T) { +func TestBytesFreedAreTheSizesOfWhatGoes(t *testing.T) { v := view(t, world()) a := &CollectAnswer{DryRun: true, Eligible: 2, WouldLetGo: []string{ "artifact-store://app/server@" + d("m-old"), "artifact-store://app/tools/blobs/" + d("a-old")}} out := Collected(v, a, true, nil) now, after := out["collector_frees_now_bytes"].(int64), out["collector_frees_after_bytes"].(int64) - // m-old's own content, old-only and a-old and its holder go; shared and cfg stay with m-kept. - want := int64(7000) + 500 + 4000 + v.L.Blobs[d("m-old")] + v.L.Blobs[d("h-old")] - if now != 7000 || after != want { - t.Errorf("now %d after %d, want 7000 and %d", now, after, want) + // m-old and the holder of a-old go, each with the size its seat says. + if want := v.L.Blobs[d("m-old")] + v.L.Blobs[d("h-old")]; now != 0 || after != want { + t.Errorf("now %d after %d, want 0 and %d", now, after, want) } if out["manifests_gone"].(int) != 2 { t.Errorf("gone %v", out["manifests_gone"]) @@ -277,8 +267,7 @@ func TestOnlyTheirBytesLeaveOutWhatAnotherStateMarks(t *testing.T) { v := view(t, world()) entries := Classify(v, withFill(records()), nil) sums := Summarise(v, entries, entries) - el := sums[StateEligible] - if el.OnlyTheirBytes != 500+v.L.Blobs[d("m-old")] { + if el := sums[StateEligible]; el.OnlyTheirBytes != v.L.Blobs[d("m-old")] { t.Errorf("eligible only-theirs %d", el.OnlyTheirBytes) } } @@ -395,7 +384,7 @@ func (a *asked) ask(key string, body any) (json.RawMessage, error) { func tool(t *testing.T, a *asked, name string) func(map[string]any) (any, error) { f := world() - s := &Store{Container: "mesh-registry", Run: f.run(t)} + s := &Store{Ask: f.ask(t)} for _, tl := range Tools(s, recordsAsked{a}, &Catalogue{Ask: a.ask}) { if tl.Name == name { return tl.Run @@ -476,7 +465,9 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) { // CAT-AS1: what it holds and removes it asks the two registry seats the artifact store's holder serves. want := []string{"seat:mesh-hand-acts.hand-act", CatalogueModules, CatalogueModule, WantsTool, ManifestsTool, "seat:oci-artifact-registry.hold", "seat:oci-artifact-registry.release", "seat:oci-artifact-registry.remove", - "seat:container-image-registry.hold", "seat:container-image-registry.release", "seat:container-image-registry.remove"} + "seat:container-image-registry.hold", "seat:container-image-registry.release", "seat:container-image-registry.remove", + // ADR 0632 §4: what the store holds it reads through the two seats' list. + "seat:container-image-registry.list", "seat:oci-artifact-registry.list"} if !reflect.DeepEqual(m.Invokes, want) { t.Errorf("invokes %v", m.Invokes) } @@ -490,6 +481,11 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) { if len(m.Requires) != 0 { t.Errorf("requires %v, want nothing: the store is reached through the registry seats", m.Requires) } + // Its address file is filled from the image seat holder's listen (ADR 0632 §1), never by the retired provision. + address, err := os.ReadFile("../../files/store-address") + if err != nil || strings.TrimSpace(string(address)) != "${seat:container-image-registry:reach}" { + t.Errorf("the store's address file reads %q (%v), want the image seat's reach", address, err) + } var verbs []string for _, tl := range SeatTools(&Holder{}, &Store{}) { verbs = append(verbs, strings.TrimPrefix(tl.Name, catalogue.ArtifactSweepSeat+".")) @@ -509,25 +505,3 @@ func TestTheToolsServedAreTheToolsTheManifestNames(t *testing.T) { } } } - -func TestASocketRefusalIsAskedAgainThroughSudo(t *testing.T) { - var ran []string - run := func(_ context.Context, name string, args ...string) Ran { - ran = append(ran, name) - if name == "docker" { - return Ran{Status: 1, Stderr: "permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock"} - } - return Ran{Stdout: "#end\n"} - } - if _, err := docker(context.Background(), run, 1000, "exec", "x"); err != nil { - t.Fatal(err) - } - if !reflect.DeepEqual(ran, []string{"docker", "sudo"}) { - t.Errorf("ran %v", ran) - } - if _, err := docker(context.Background(), func(context.Context, string, ...string) Ran { - return Ran{Status: 1, Stderr: "Error: No such container: mesh-registry"} - }, 1000, "exec"); err == nil || !strings.Contains(err.Error(), "not on this machine") { - t.Errorf("err %v", err) - } -} diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/tools.go b/modules/artifact-store-tools/cmd/artifact-store-tools/tools.go index daaab4859..7125aa5b7 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/tools.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/tools.go @@ -27,9 +27,9 @@ func Tools(s *Store, c Recorded, cat *Catalogue) []stdio.Tool { { Name: "artifact_store_repositories", Description: "Every repository the artifact store holds: its tags and what each names, how many manifests it holds " + - "(tagged or not — the mesh pins by digest, so most are untagged) and its size, the blobs its manifests mark. " + - "A blob two repositories share is counted in each, and shared_bytes says how much of a size that is. Read from " + - "the store's own files; changes nothing. Replaces curl /v2/_catalog and /v2//tags/list. (r)", + "(tagged or not — the mesh pins by digest, so most are untagged) and its size, each manifest's as its seat says it. " + + "A layer two manifests share is counted in each. Read through the registry seats' list (ADR 0632 §4); changes " + + "nothing. Replaces curl /v2/_catalog and /v2//tags/list. (r)", Input: map[string]any{"repository": repoArg}, Run: func(args map[string]any) (any, error) { v, err := s.View(ctx) @@ -41,9 +41,9 @@ func Tools(s *Store, c Recorded, cat *Catalogue) []stdio.Tool { }, { Name: "artifact_store_usage", - Description: "How large the artifact store is: every blob's bytes together, the largest repositories, the bytes " + - "more than one repository's manifests mark, and the bytes no manifest marks — what the store's nightly collector " + - "frees next. Read from the store's own files; changes nothing. Replaces du on the store's directory. (r)", + Description: "How large the artifact store is: every manifest's size together, as its seat says it (a layer two " + + "manifests share is counted in each), and the largest repositories. Read through the registry seats' list (ADR " + + "0632 §4); changes nothing. Replaces du on the store's directory. (r)", Input: map[string]any{"top": map[string]any{"type": "integer", "description": "how many of the largest repositories to name (default 10, at most 200)"}}, Run: func(args map[string]any) (any, error) { top, err := count(args, "top", 10, 1, 200) @@ -139,16 +139,6 @@ func Tools(s *Store, c Recorded, cat *Catalogue) []stdio.Tool { } } -// View reads the store now. -func (s *Store) View(ctx context.Context) (*View, error) { - l, err := s.List(ctx) - if err != nil { - return nil, err - } - m, unread := s.Manifests(ctx, l) - return &View{L: l, M: m, Unread: unread}, nil -} - // unreadNote says what manifests that could not be read do to the numbers, when there are any. func unreadNote(v *View) map[string]any { if len(v.Unread) == 0 { diff --git a/modules/artifact-store-tools/cmd/artifact-store-tools/view.go b/modules/artifact-store-tools/cmd/artifact-store-tools/view.go index 9d853a0df..71578d42a 100644 --- a/modules/artifact-store-tools/cmd/artifact-store-tools/view.go +++ b/modules/artifact-store-tools/cmd/artifact-store-tools/view.go @@ -4,13 +4,15 @@ import ( "sort" ) -// View is the store at one moment: its files, and every manifest's content that could be read. +// View is the store at one moment, as its seats list it (listed.go). type View struct { L *Listing // M is each manifest read, by Key. M map[string]*Manifest // Unread is each manifest that could not be read, by Key, with why. Unread map[string]string + // Held is each manifest a hold on its seat keeps from retention, by Key. + Held map[string]bool } // Marks are the blobs one manifest keeps from the store's collector: its own content, its diff --git a/modules/artifact-store-tools/files/store-address b/modules/artifact-store-tools/files/store-address index 1bcadab26..71418ef2a 100644 --- a/modules/artifact-store-tools/files/store-address +++ b/modules/artifact-store-tools/files/store-address @@ -1 +1 @@ -${seat:mesh-artifact-store:reach} +${seat:container-image-registry:reach} diff --git a/modules/artifact-store-tools/module.json b/modules/artifact-store-tools/module.json index 35a5e0f35..5abdca6c5 100644 --- a/modules/artifact-store-tools/module.json +++ b/modules/artifact-store-tools/module.json @@ -25,7 +25,9 @@ "seat:oci-artifact-registry.remove", "seat:container-image-registry.hold", "seat:container-image-registry.release", - "seat:container-image-registry.remove" + "seat:container-image-registry.remove", + "seat:container-image-registry.list", + "seat:oci-artifact-registry.list" ], "reads": [ "mesh-delivery-builder.builds", @@ -93,7 +95,6 @@ "artifact-store-tools" ], "env": { - "MESH_ARTIFACT_STORE_CONTAINER": "mesh-registry", "MESH_ARTIFACT_STORE_NAMES_FILE": "${dir:state}/store-names", "MESH_ARTIFACT_STORE_ADDRESS_FILE": "${dir:state}/store-address" }