mesh-catalog/modules/artifact-store-tools

artifact-store-tools

Tools that say what the mesh's artifact store holds, and what the controller's records say of it (novox/hq ADR 0251 §1–3, to-be 51). It changes nothing on its machine; its only resources are its state directory and the file the mesh writes there from its setting older-store-names.

Why a module beside the store

The store is the distribution module, the registry seats' holder. These tools became a module of their own when the indexer's judge refused a store holder that builds an artifact; that rule is gone (issue 1966), and the sweep stays its own module, holding the seat mesh-artifact-sweep. It requires no provision: what it holds, releases and removes it asks the registry seats oci-artifact-registry and container-image-registry (see The sweep through the registry seats), and what the store holds it reads through the same seats' list (novox/hq ADR 0632 §4): it reads neither the store's files nor its read side, so it runs on any machine.

Tools

tool		what
artifact_store_repositories	r	every repository: tags and what each names, how many manifests (tagged or not), size
artifact_store_usage	r	the store's size, each manifest's as its seat says it, and the largest repositories
artifact_store_references	r	what the controller's records and the catalogue's definitions say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold
artifact_store_collect	a	a dry run unless dry_run is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs why and asks the controller's collect

How the store is read

The store's read side lists repositories and tags, but not a manifest no tag names, and the mesh pins every machine by digest, so that is most of them. So the bundle asks the two registry seats' list (container-image-registry for images and indexes, oci-artifact-registry for bundles and archives), which the store's holder answers from the store's own files: each manifest's digest, whether it is an index and the manifests it names, the tags naming it, its size and whether a hold keeps it (novox/hq ADR 0632 §4).

A manifest's size is its own content and every blob it names, as its seat says it, so a layer two manifests share is counted in each, and the bytes the store's nightly collector frees are the sizes of the manifests that go, an upper bound. The address the bundle says the store is at comes from ${seat:container-image-registry:reach}, the image seat holder's listen (ADR 0632 §1), and is used for the catalogue's definitions only, never to reach the store.

The states of a manifest

state	what	removable
kept	a definition names it, or one of the five most recent builds of a module the mesh holds; why says which	no
named-by-a-definition	no record names it, and a definition in the catalogue pins it by digest, under the store's current name or an older one; why names the module (hq issue 323)	no
holder-of-kept-archive	the manifest that keeps a kept archive's blob (hq issue 253)	no
eligible	the mesh made it and keeps it for no reason	through the controller's collect
holder-of-eligible-archive	the manifest that keeps an eligible archive's blob	with its archive
let-go-yet-present	the controller recorded letting go of it, and the store still holds it	no — a finding
named-document	a one-layer manifest the controller keeps under a tag (hq to-be 45 §9)	no
unrecorded	no record names it	never, by any tool (ADR 0189 §3)

A platform manifest takes the state of the recorded index naming it (hq ADR 0257). An upstream image is an index over several platforms, and each platform manifest is a manifest of the repository in its own right; the controller lets an index go with the platform manifests only it names, so they are counted as the index's. Two indexes naming one platform give it the more keeping state. An index no record names leaves its platform manifests unrecorded with it.

What the definitions pin (hq issue 323). A definition may pin an image of the store the mesh did not make: pushed before the mesh built it, and pinned under the store's current name or a name it had before. No record of the controller names it, yet a machine runs it, so artifact_store_references reads every definition the catalogue holds (the seat mesh-catalog's verbs modules, then module for each, eight at a time and within twenty seconds) and finds every <name>/<repository>@sha256:… in any of its strings, under each of the store's names: artifact-store://, the store's address as the controller answers it, and the names it had before. Those are the mesh's setting older-store-names (space-separated, none by default, set with the controller's settings), which the mesh writes to the module's store-names file: a catalogue names no installation. When they cannot be read, the answer's first line says so. A manifest no record names that a definition pins is named-by-a-definition; one the records name keeps the records' state. A definition read at its commit is kept, so the next call asks only for the list. Definitions that could not be read are listed under definitions.unread, and the answer's first line says that what they pin reads as unrecorded until they are.

Unrecorded copies of upstream images are listed in unrecorded_mirrors: each manifest at the top of a repository only the build's mirror writes (upstream/<host>/<path>, or a module's <module>/on-<argument> from before hq ADR 0257) that no record names, with what it and its platforms mark alone. They are the input to the controller's mirrors verb, which records them as the mesh's on a person's word; the sweep then decides about them as about every other copy. This tool removes none of them.

The records come from the controller's artifacts verb, asked with the references already let go of; when that answer is too large to carry, it is asked without them and the answer says so.

Collecting

artifact_store_collect never deletes anything itself. The controller decides and records what it lets go of (ADR 0189 §2), so a real run asks its collect with why and confirm; the controller holds every kept archive first and records the run as a hand-act. Bytes come back at the store's nightly collection, which runs with the store held still.

An index goes with the platform manifests it names that no kept index of its repository also names, as the controller's sweep lets them go (hq ADR 0257). The controller reads only its records; the store's files show what else names a platform, so a dry run lists in platforms_named_by_what_stays every manifest that stays and would lose a platform it names — to be read before a real run.

Tests

go test ./...

Against a fake of the two registry seats' list: the store read through it (an index and the manifests it names, a bundle's holder, a hold, the seat's sizes), every state, what the records keep that the store lacks, bytes freed as the sizes of what goes, a dry run never confirming, a real run without why refused before anything is asked, the fall-back when the references let go of cannot be carried, a manifest a definition pins under the store's current or an older name, a definition not read said first, and that the tools served are exactly the manifest's tools and its invokes exactly the two controller verbs and the catalogue's two tools. Any ask other than a seat's list fails the read tools' tests.

Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

• ${dir:state}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).

The sweep through the registry seats

The sweep writes nothing to the store's door (CAT-AS1, ADR 0617 §2): it asks the registry seats the store's holder serves. Each kept bundle or archive is held on oci-artifact-registry and each kept copy of an upstream base on container-image-registry (hold, with why it is kept) before anything goes; each eligible one is released, then removed for retention on its seat (release, remove). It reads a reference in both forms, artifact-store://<path> and <seat>/<module>/<artifact>@<digest>, as one artifact, and says artifacts-gone in the form the build named it. A call a seat refuses stops the sweep and writes the finding sweep-refused in the module's findings; the next sweep no seat refuses clears it (sweepseats.go).