jochen
d745a4d31e
mesh/merge-gate pass: builds artifact-store-tools → novox; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)
mesh/delivery delivered
No record names an image the mesh did not make, so one a definition pins under an older name of the store read as unrecorded, the same word as an image nothing uses. The tool now reads every definition through the catalogue and says named-by-a-definition with the module. The store's older names are the mesh's setting older-store-names, since a catalogue names no installation.
112 lines
7.8 KiB
Markdown
112 lines
7.8 KiB
Markdown
# artifact-store-tools
|
||
|
||
Tools that say what the mesh's artifact store holds, and what the controller's records say of it
|
||
(novox/hq ADR 0251 §1–3, to-be 51). It changes nothing on its machine; its only resources are its state
|
||
directory and the file the mesh writes there from its setting `older-store-names`.
|
||
|
||
## Why a module beside the store
|
||
|
||
The store is the `distribution` module. A module that provides the artifact store cannot also build an
|
||
artifact: building publishes to the store, so the store would be needed to create itself, and the
|
||
controller's module check refuses that manifest. So the store's tools are a second module, as the
|
||
controller's refusal says to do. **Assign it to the machine that holds the store**: its tools read the
|
||
store's files through the store's own container, and anywhere else they answer that the container is not
|
||
on that machine.
|
||
|
||
## Tools
|
||
|
||
| tool | | what |
|
||
|---|---|---|
|
||
| `artifact_store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size |
|
||
| `artifact_store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) |
|
||
| `artifact_store_references` | r | what the controller's records and the catalogue's definitions say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
|
||
| `artifact_store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` |
|
||
|
||
### How the store is read
|
||
|
||
The store's door lists repositories and tags, but not a manifest no tag names, and the mesh pins every
|
||
machine by digest, so that is most of them. So the bundle reads the store's own files through its own
|
||
container (`docker exec mesh-registry`, busybox `find`, `stat` and `cat`), and only reads them: every
|
||
blob with its size, every manifest each repository holds, what each tag names, and each manifest's
|
||
content, four hundred to an exec. What it read it keeps (a manifest is named by its content, so it never
|
||
changes), and the next call reads only what is new. A manifest that could not be read is counted and
|
||
said; what it marks beyond itself is then unknown, so sizes may read low and freed bytes high, and the
|
||
answer says so.
|
||
|
||
docker runs as the tool runner's account; a socket that refuses it is asked again through `sudo -n`,
|
||
never with a prompt, as the container runtime's own tools do.
|
||
|
||
A manifest *marks* its own content, its configuration and its layers, and an index marks the manifests
|
||
it lists. The store's collector removes every blob no manifest marks, so these numbers are its own.
|
||
|
||
### The states of a manifest
|
||
|
||
| state | what | removable |
|
||
|---|---|---|
|
||
| `kept` | a definition names it, or one of the five most recent builds of a module the mesh holds; `why` says which | no |
|
||
| `named-by-a-definition` | no record names it, and a definition in the catalogue pins it by digest, under the store's current name or an older one; `why` names the module (hq issue 323) | no |
|
||
| `holder-of-kept-archive` | the manifest that keeps a kept archive's blob (hq issue 253) | no |
|
||
| `eligible` | the mesh made it and keeps it for no reason | through the controller's `collect` |
|
||
| `holder-of-eligible-archive` | the manifest that keeps an eligible archive's blob | with its archive |
|
||
| `let-go-yet-present` | the controller recorded letting go of it, and the store still holds it | no — a finding |
|
||
| `named-document` | a one-layer manifest the controller keeps under a tag (hq to-be 45 §9) | no |
|
||
| `unrecorded` | no record names it | **never**, by any tool (ADR 0189 §3) |
|
||
|
||
**A platform manifest takes the state of the recorded index naming it** (hq ADR 0257). An upstream
|
||
image is an index over several platforms, and each platform manifest is a manifest of the repository in
|
||
its own right; the controller lets an index go with the platform manifests only it names, so they are
|
||
counted as the index's. Two indexes naming one platform give it the more keeping state. An index no
|
||
record names leaves its platform manifests unrecorded with it.
|
||
|
||
**What the definitions pin** (hq issue 323). A definition may pin an image of the store the mesh did not
|
||
make: pushed before the mesh built it, and pinned under the store's current name or a name it had before.
|
||
No record of the controller names it, yet a machine runs it, so `artifact_store_references` reads every
|
||
definition the catalogue holds (`mesh-catalog.catalog_modules`, then `catalog_module` for each, eight at
|
||
a time and within twenty seconds) and finds every `<name>/<repository>@sha256:…` in any of its strings,
|
||
under each of the store's names: `artifact-store://`, the store's address as the controller answers it,
|
||
and the names it had before. Those are the mesh's setting `older-store-names` (space-separated, `none` by
|
||
default, set with the controller's `settings`), which the mesh writes to the module's `store-names` file: a
|
||
catalogue names no installation. When they cannot be read, the answer's first line says so. A manifest no record
|
||
names that a definition pins is `named-by-a-definition`; one the records name keeps the records' state. A
|
||
definition read at its commit is kept, so the next call asks only for the list. Definitions that could
|
||
not be read are listed under `definitions.unread`, and the answer's first line says that what they pin
|
||
reads as unrecorded until they are.
|
||
|
||
**Unrecorded copies of upstream images** are listed in `unrecorded_mirrors`: each manifest at the top of
|
||
a repository only the build's mirror writes (`upstream/<host>/<path>`, or a module's `<module>/on-<argument>`
|
||
from before hq ADR 0257) that no record names, with what it and its platforms mark alone. They are the
|
||
input to the controller's `mirrors` verb, which records them as the mesh's on a person's word; the sweep
|
||
then decides about them as about every other copy. This tool removes none of them.
|
||
|
||
The records come from the controller's `artifacts` verb, asked with the references already let go of;
|
||
when that answer is too large to carry, it is asked without them and the answer says so.
|
||
|
||
### Collecting
|
||
|
||
`artifact_store_collect` never deletes anything itself. The controller decides and records what it
|
||
lets go of (ADR 0189 §2), so a real run asks its `collect` with `why` and `confirm`; the controller holds
|
||
every kept archive first and records the run as a hand-act. Bytes come back at the store's nightly
|
||
collection, which runs with the store held still.
|
||
|
||
An index goes with the platform manifests it names that no kept index of its repository also names, as
|
||
the controller's sweep lets them go (hq ADR 0257). The controller reads only its records; the store's
|
||
files show what else names a platform, so a dry run lists in `platforms_named_by_what_stays` every
|
||
manifest that stays and would lose a platform it names — to be read before a real run.
|
||
|
||
## Tests
|
||
|
||
```
|
||
go test ./...
|
||
```
|
||
|
||
Against a fake container that prints what the two scripts would: the listing parsed (a repository name
|
||
with slashes, a cut listing refused), manifests read with one absent or unreadable said as unread, a
|
||
digest never becoming a path outside the blobs, the mark (an index, a shared blob, a blob nothing marks),
|
||
every state, what the records keep that the store lacks, bytes freed counting a blob shared with a kept
|
||
manifest as kept, a dry run never confirming, a real run without `why` refused before anything is asked,
|
||
the fall-back when the references let go of cannot be carried, a manifest a definition pins under the
|
||
store's current or an older name, a definition not read said first, escalation through `sudo -n`, and that the
|
||
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs and the
|
||
catalogue's two tools.
|
||
Any command other than the two read-only scripts fails the test.
|