jochen
2dffc48c81
The store's tools counted each platform of an upstream image apart from its index, so a kept copy read as unrecorded and a let-go index's bytes were never counted as freed. The copies no record names are now listed for the controller's mirrors verb, and a dry run says what stays yet names a platform collection would take (hq ADR 0257, issue 321).
94 lines
6.1 KiB
Markdown
94 lines
6.1 KiB
Markdown
# artifact-store-tools
|
||
|
||
Tools that say what the mesh's artifact store holds, and what the controller's records say of it
|
||
(novox/hq ADR 0251 §1–3, to-be 51). It declares no resources and changes nothing on its machine.
|
||
|
||
## Why a module beside the store
|
||
|
||
The store is the `distribution` module. A module that provides the artifact store cannot also build an
|
||
artifact: building publishes to the store, so the store would be needed to create itself, and the
|
||
controller's module check refuses that manifest. So the store's tools are a second module, as the
|
||
controller's refusal says to do. **Assign it to the machine that holds the store**: its tools read the
|
||
store's files through the store's own container, and anywhere else they answer that the container is not
|
||
on that machine.
|
||
|
||
## Tools
|
||
|
||
| tool | | what |
|
||
|---|---|---|
|
||
| `artifact_store_repositories` | r | every repository: tags and what each names, how many manifests (tagged or not), size |
|
||
| `artifact_store_usage` | r | the store's size, the largest repositories, bytes shared between repositories, bytes no manifest marks (what the nightly collector frees next) |
|
||
| `artifact_store_references` | r | what the controller's records say of each manifest, counted and sized by state and repository; each manifest listed when one repository is asked; what the records keep that the store does not hold |
|
||
| `artifact_store_collect` | a | a dry run unless `dry_run` is false: what the controller would let go of, and the bytes the nightly collector would free then and now. A real run needs `why` and asks the controller's `collect` |
|
||
|
||
### How the store is read
|
||
|
||
The store's door lists repositories and tags, but not a manifest no tag names, and the mesh pins every
|
||
machine by digest, so that is most of them. So the bundle reads the store's own files through its own
|
||
container (`docker exec mesh-registry`, busybox `find`, `stat` and `cat`), and only reads them: every
|
||
blob with its size, every manifest each repository holds, what each tag names, and each manifest's
|
||
content, four hundred to an exec. What it read it keeps (a manifest is named by its content, so it never
|
||
changes), and the next call reads only what is new. A manifest that could not be read is counted and
|
||
said; what it marks beyond itself is then unknown, so sizes may read low and freed bytes high, and the
|
||
answer says so.
|
||
|
||
docker runs as the tool runner's account; a socket that refuses it is asked again through `sudo -n`,
|
||
never with a prompt, as the container runtime's own tools do.
|
||
|
||
A manifest *marks* its own content, its configuration and its layers, and an index marks the manifests
|
||
it lists. The store's collector removes every blob no manifest marks, so these numbers are its own.
|
||
|
||
### The states of a manifest
|
||
|
||
| state | what | removable |
|
||
|---|---|---|
|
||
| `kept` | a definition names it, or one of the five most recent builds of a module the mesh holds; `why` says which | no |
|
||
| `holder-of-kept-archive` | the manifest that keeps a kept archive's blob (hq issue 253) | no |
|
||
| `eligible` | the mesh made it and keeps it for no reason | through the controller's `collect` |
|
||
| `holder-of-eligible-archive` | the manifest that keeps an eligible archive's blob | with its archive |
|
||
| `let-go-yet-present` | the controller recorded letting go of it, and the store still holds it | no — a finding |
|
||
| `named-document` | a one-layer manifest the controller keeps under a tag (hq to-be 45 §9) | no |
|
||
| `unrecorded` | no record names it | **never**, by any tool (ADR 0189 §3) |
|
||
|
||
**A platform manifest takes the state of the recorded index naming it** (hq ADR 0257). An upstream
|
||
image is an index over several platforms, and each platform manifest is a manifest of the repository in
|
||
its own right; the controller lets an index go with the platform manifests only it names, so they are
|
||
counted as the index's. Two indexes naming one platform give it the more keeping state. An index no
|
||
record names leaves its platform manifests unrecorded with it.
|
||
|
||
**Unrecorded copies of upstream images** are listed in `unrecorded_mirrors`: each manifest at the top of
|
||
a repository only the build's mirror writes (`upstream/<host>/<path>`, or a module's `<module>/on-<argument>`
|
||
from before hq ADR 0257) that no record names, with what it and its platforms mark alone. They are the
|
||
input to the controller's `mirrors` verb, which records them as the mesh's on a person's word; the sweep
|
||
then decides about them as about every other copy. This tool removes none of them.
|
||
|
||
The records come from the controller's `artifacts` verb, asked with the references already let go of;
|
||
when that answer is too large to carry, it is asked without them and the answer says so.
|
||
|
||
### Collecting
|
||
|
||
`artifact_store_collect` never deletes anything itself. The controller decides and records what it
|
||
lets go of (ADR 0189 §2), so a real run asks its `collect` with `why` and `confirm`; the controller holds
|
||
every kept archive first and records the run as a hand-act. Bytes come back at the store's nightly
|
||
collection, which runs with the store held still.
|
||
|
||
An index goes with the platform manifests it names that no kept index of its repository also names, as
|
||
the controller's sweep lets them go (hq ADR 0257). The controller reads only its records; the store's
|
||
files show what else names a platform, so a dry run lists in `platforms_named_by_what_stays` every
|
||
manifest that stays and would lose a platform it names — to be read before a real run.
|
||
|
||
## Tests
|
||
|
||
```
|
||
go test ./...
|
||
```
|
||
|
||
Against a fake container that prints what the two scripts would: the listing parsed (a repository name
|
||
with slashes, a cut listing refused), manifests read with one absent or unreadable said as unread, a
|
||
digest never becoming a path outside the blobs, the mark (an index, a shared blob, a blob nothing marks),
|
||
every state, what the records keep that the store lacks, bytes freed counting a blob shared with a kept
|
||
manifest as kept, a dry run never confirming, a real run without `why` refused before anything is asked,
|
||
the fall-back when the references let go of cannot be carried, escalation through `sudo -n`, and that the
|
||
tools served are exactly the manifest's `tools` and its `invokes` exactly the two controller verbs.
|
||
Any command other than the two read-only scripts fails the test.
|