From f1e2ebce7abb144e272b96a81f0f7e142dd09e30 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 11:51:59 +0200 Subject: [PATCH 1/2] Record how the registries keep what is named, so nothing piles up unseen (ADR 0251, to-be 51) --- ...ps-the-image-it-runs-and-the-one-before.md | 203 ++++++++++++++++++ .../51-the-registries-keep-what-is-named.md | 149 +++++++++++++ 03-DESIGN/01-to-be/README.md | 1 + 3 files changed, 353 insertions(+) create mode 100644 02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md create mode 100644 03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md diff --git a/02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md b/02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md new file mode 100644 index 00000000..76bec15a --- /dev/null +++ b/02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md @@ -0,0 +1,203 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-08 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md +--- + +# 251. The registries say what they hold and keep what is named, and a machine keeps the image it runs and the one before + +## Context + +The mesh has three places where built things pile up, and only one of them collects. + +**The artifact store collects, and nobody can see it do so.** Since +[ADR 0189](0189-the-store-keeps-what-the-records-name.md) the controller lets go of every artifact it +made and keeps for no reason, after each build, and the store's collector reclaims the bytes each +night. Measured on 2026-10-08 through the controller's `collection` command: 207 archives kept, 207 +held by a manifest, 0 eligible to let go. That is the records' half. The store's half has no tool at +all. The module that holds the store carries a TypeScript client and three tools that no build +produces, so nothing on the mesh answers *which repositories are there, how large, and which of what +is there no record names*. The one thing ADR 0189 forbids collection to touch — a digest the mesh did +not record making — is exactly the thing nobody can count. + +**The package registry never collects.** Every publish of a package to the forge's npm registry adds a +version, and nothing removes one. Nothing says which versions any build still resolves to. + +**A machine collects only what nothing names.** The container runtime's weekly prune takes dangling +images and unused build cache, and nothing else. An image a machine pulled by digest is not dangling, +so every version of every module a machine ever ran stays on it. The build node is the worst case: on +2026-10-08 it held 355 images no container used, about 58 GB summed (images share layers, so the disk +they take together is less). + +The operator approved three improvements on 2026-10-08: tools for the store, retention for the package +registry, and pruning of unused images on a machine — each a dry run unless asked otherwise. + +## Considered Options + +**For the store's tools.** + +1. *The store seat's verbs.* The tools a role answers are the seat's, and a second holder of the store + would serve them too. But a verb added to a held seat whose holder lives in another repository lands + in three steps ([ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md)), and the + store has one holder and no second in view. Not now; the tools can be promoted when a second holder + exists. +2. **The store module's own tools, in Go.** Chosen. They replace the TypeScript client nothing builds. +3. *The controller's verbs.* The controller holds the records but not the store's files; the store may + move to a machine the controller is not on. Rejected for everything except what only the records can + answer. + +**For what the store tools read.** The store's door lists repositories and tags, but not a manifest no +tag names — and since the mesh pins by digest, that is almost every manifest. Only the store's files +list them, with each blob's size. So the tools read the store's own files, read-only, through the +store's own container, and read each manifest's content through the door. The registry's own +`garbage-collect --dry-run` was weighed and rejected as the source: it says what is unmarked, but not +how large, and not what letting go of a manifest would free. + +**For collecting on demand.** *Letting the store tool delete what the records do not keep* was rejected: +the controller deletes and records that it did, and two deleters means two records. **The same sweep +as after a build, asked on demand,** is chosen. + +**For what is unrecorded.** *Removing by age what no record names* was rejected: it is the rule of ADR +0189 §3 broken, and the images genesis pushed are exactly that. The tools count it and size it; a person +decides. + +**For the package registry.** + +1. *Keep the newest N of each package, and nothing else.* Rejected: a range pinned to an older line + (`^0.1.0` while `0.2.x` is the newest) resolves to a version this drops. +2. *Keep what lockfiles name.* Not enough alone: a module without a lockfile resolves its range at + build time. +3. **Keep what is referenced, and the newest N.** Chosen. Referenced means a version a lockfile on a + repository's default branch names, the highest version satisfying a range a package manifest there + names, and a version a dist-tag names. + +**For a machine's images.** + +1. *`docker image prune --all`.* Rejected: it removes the previous version, so going back needs a pull, + and it removes the image of a scheduled step between two of its runs. +2. *By age alone.* Rejected: a module not rebuilt for a while would lose its previous version. +3. **Keep what a container uses, what the machine's declaration names, and the newest other image of + each line.** Chosen. + +## Decision + +**1. The store module answers for the store, with four tools of its own.** The distribution module, +holder of `mesh-artifact-store`, serves a Go bundle in place of its unbuilt TypeScript: + +- `store_repositories` — every repository, its tags, how many manifests it holds and its size; +- `store_usage` — the store's size, the largest repositories, what blobs are shared, and what the + store's nightly collector would free now; +- `store_references` — what the records say of each manifest the store holds (below), and what the + records keep that the store does not hold; +- `store_collect` — a dry run unless `dry_run` is false; a real run needs `why`. + +The tools read the store's files through the store's own container and only read them. The first three +change nothing; `store_collect` changes nothing either, unless it is a real run, and then only by asking +the controller (point 3). + +**2. The records are asked, never copied.** A new controller verb, `artifacts`, answers every +reference the mesh recorded making, with its state: *kept* because a definition names it, *kept* +because it belongs to one of the five most recent builds of a module the mesh holds, *eligible* (made, +not kept, not yet let go), or — when asked — *collected*. `store_references` sets that beside what the +store holds and says, for every manifest, one of: + +- **kept**, with the reason; +- **a holder of a kept archive** (the manifest that keeps an archive's blob, [issue 253](../04-ISSUES/253-the-stores-collector-would-delete-every-archive-the-mesh-keeps/00-report.md)); +- **eligible**, or a holder of an eligible archive; +- **a named document** — something the controller keeps under a tag (to-be 45 §9); +- **unrecorded** — no record names it. + +Unrecorded manifests are counted and sized, by repository. **They are never removed by any tool in this +record.** ADR 0189 §3 stands as written. + +**3. Collecting on demand is the same sweep, asked by a person.** A new controller verb, `collect`, +says what the sweep would let go of, and does it only with `confirm` and a `why`. A real run holds every +kept archive before it lets anything go, exactly as the sweep after a build does; it is bounded by a +count and a time, says what it left; and it is recorded as a hand-act with its `why`. `store_collect`'s +dry run adds what only the store knows: how many bytes the nightly collector would free once those +manifests are gone, beside what it would free tonight anyway. Its real run asks `collect`. Bytes are +still reclaimed by the nightly collector, because only it runs with the store held still. + +**4. The package registry keeps what is referenced and the newest five of each package.** The forge +module, holder of `npm-package-registry`, gains a Go bundle beside its TypeScript one, with two tools: + +- `npm_packages` — every package of the registry's owner, each version with its publish date and size, + and why each is kept; +- `npm_retention` — what retention would delete and how many bytes; a dry run unless `dry_run` is + false, and a real run needs `why`. + +A version is kept when: + +- a lockfile on the default branch of any repository on the forge names it; +- it is the highest version satisfying a range that a package manifest there names; +- a dist-tag names it; or +- it is among the newest five of its package (`keep` changes the number; it is never below one). + +**A repository whose files could not be read stops a real run before anything is deleted.** Retention +that does not know what is referenced does not delete. + +**5. A machine removes images no declaration uses, and keeps the one each module runs and the one +before.** The container runtime's module gains `docker_prune_images`. It keeps: + +- every image a container on the machine uses, in any state; +- every image the machine's declaration names — both what the mesh would send it now and what it was + last sent — asked of a new controller verb, `images`; +- for each line of images, the newest image besides those. A line is the images sharing a repository + name, joined across names when one image carries several (a build's local tag and the store's name); +- every image younger than a floor, seven days by default — the same week the weekly prune leaves. + +It removes the rest by id, without force, so the runtime itself refuses an image a container uses. **It +refuses to remove anything when the controller cannot say what the declaration names.** It is a dry run +unless `dry_run` is false, and a real run needs `why`. + +**6. Nothing here runs by itself.** No timer is added: the store's nightly collector and the weekly +dangling prune stay as they are. Each tool is a dry run until a person runs it for real, and whether any +of them becomes scheduled is a later decision, made from what their dry runs said. + +## Consequences + +- The question "what is in the store that nothing records" has an answer, with a size. Before, it could + only be estimated by reading the store's disk by hand. +- Each controller verb answers in one bus message, and an answer larger than the bus carries is lost + on its way back (issue 314, open while this was written). So `artifacts` leaves out the collected references unless asked, + and `images` answers only images. +- Five builds are kept by two rules now: the store's (ADR 0189 §3) and the package registry's. They + agree on purpose, so "how far back can this go" has one answer. +- On a machine, the previous version of a module is a local image, and the one before that is a pull + from the store. The store keeps five, so a pull succeeds. +- A build node removes base images no declaration names, and pulls them again at its next build. That + is a cost in time, and it is visible in the dry run before anybody accepts it. +- The store module stops carrying code no build produces, and the `image.pushed` event it declared + and never emitted goes with it. + +## How it is checked + +- **The controller.** `artifacts` names every reference the keep set holds with its reason, and every + eligible reference; a collected reference only when asked. `collect` without `confirm` deletes nothing, + against a fake store that records what it was asked; with `confirm` and no `why`, it is refused; a + real run holds kept archives before letting anything go. `images` names every container image of a + declaration, and of the last one sent. +- **The store's tools.** Against a fake store and a fake listing of its files: classification of each + manifest (kept, holder, eligible, named document, unrecorded); the bytes the collector would free, + counting a blob shared with a kept manifest as kept; a dry run deletes nothing; a real run without + `why` is refused. +- **Package retention.** Range satisfaction (`^`, `~`, exact, `x`, comparison, `||`, hyphen); a version + a lockfile names is kept, whatever its age; the newest five are kept; an unreadable repository stops + a real run; a dry run deletes nothing. +- **Image pruning.** Against a fake runtime: what a container uses, what the declaration names, the + newest other image of each line and anything under the floor are kept; an image with two names is one + line; an unanswered controller refuses; a dry run removes nothing; removal is never forced. +- **Live.** The dry runs against the running store, the package registry and each machine, read before + anything is run for real. Their numbers are recorded in to-be 51. + +## References + +- [ADR 0189 — the store keeps what the records name](0189-the-store-keeps-what-the-records-name.md) +- [ADR 0246 — a seat's new verb is promised before it is required](0246-a-seats-new-verb-is-promised-before-it-is-required.md) +- [ADR 0109 — a package registry seat is one per ecosystem](0109-a-package-registry-seat-is-one-per-ecosystem.md) +- [ADR 0156 — an artifact is what a build produces](0156-an-artifact-is-what-a-build-produces-and-the-store-is-named-for-its-scope.md) +- [issue 253 — the store's collector would delete every archive the mesh keeps](../04-ISSUES/253-the-stores-collector-would-delete-every-archive-the-mesh-keeps/00-report.md) +- [to-be 51 — the registries keep what is named](../03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md) diff --git a/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md b/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md new file mode 100644 index 00000000..11e863a4 --- /dev/null +++ b/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md @@ -0,0 +1,149 @@ +--- +layer: to-be +status: in-progress +code: + - mesh-controller cmd/mesh-controller (the verbs artifacts, collect and images) + - mesh-catalog modules/distribution (the store's tools) + - mesh-catalog modules/gitea (package retention) + - mesh-catalog modules/docker (image pruning) +updated: 2026-10-08 +decisions: + - 02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md + - 02-DECISIONS/0189-the-store-keeps-what-the-records-name.md +--- + +# 51 — The registries keep what is named + +**Three places hold built things: the artifact store, the package registry and each machine's +container runtime. Each gets tools that say what it holds, what names each thing, and what could go. +Each removal is a dry run unless a person asks otherwise and says why. What decides is always a +record: the controller's build records for the store, the forge's repositories for the package +registry, and the machine's declaration for its images.** +([ADR 0251](../../02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md), +extending [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md).) + +## The parts + +``` + controller (records) + artifacts ──┐ collect ──┐ images ──┐ + │ │ │ + store module ─────┴─────────────┘ │ + (store_repositories, store_usage, │ + store_references, store_collect) │ + │ reads the store's files │ + │ through its own container │ + ▼ │ + artifact store │ + │ + forge module (npm_packages, npm_retention) │ + │ reads every repository's manifests │ + ▼ │ + package registry │ + │ + container runtime module ─────────────────────┘ + (docker_prune_images), on every machine +``` + +## The artifact store + +### What the store's tools read + +The store keeps each blob once, named by its digest, with its size on disk. Each repository lists the +manifests it holds (every one, whether a tag names it or not), its tags, and the layers it may serve. +The tools list those files through the store's own container, read-only, and read each manifest's +content through the store's door. Nothing is written. + +From that, a manifest *marks* its own blob, its configuration and each layer, and an index marks the +manifests it lists. The store's nightly collector removes every blob no manifest marks. The tools work +the same mark, so their numbers are the collector's. + +### What each tool answers + +- **`store_repositories`** — each repository with its tags, its manifest count and its size: the sum of + the blobs its manifests mark. A blob shared by two repositories is counted in each, and the answer + says so. +- **`store_usage`** — the store's total size; the largest repositories; the bytes in blobs that more + than one repository marks; and the bytes in blobs no manifest marks, which is what the nightly + collector frees next. +- **`store_references`** — for every manifest, one state (below), counted and sized by repository, and + listed when one repository is asked about. Then what the records keep that the store does not hold. +- **`store_collect`** — the dry run: what the controller would let go of, and how many bytes the nightly + collector would free once those are gone. A real run with `dry_run` false and a `why` asks the + controller's `collect`, and answers what it did. + +### The states of a manifest + +| state | what it means | can collection remove it? | +|---|---|---| +| kept — definition | a module's recorded definition names it | no | +| kept — recent build | one of the five most recent builds of a module the mesh holds | no | +| holder of a kept archive | the manifest that keeps a kept archive's blob | no | +| eligible | the mesh made it and keeps it for no reason | yes, through the controller | +| holder of an eligible archive | the manifest that keeps an eligible archive's blob | yes, with its archive | +| named document | a manifest the controller keeps under a tag (to-be 45 §9) | no | +| unrecorded | no record names it | no — counted and sized only | + +A holder is recognised by its shape: a manifest with an empty configuration and one layer, which is +the blob of an archive the records name. + +### The controller's two verbs for the store + +- **`artifacts`** reads only. It answers every reference the mesh recorded making, with its state and, + for kept ones, the reason; collected ones only when `collected` is asked for; narrowed to one + repository when one is named. +- **`collect`** says what the sweep would let go of. With `confirm` and a `why` it runs the sweep that + runs after a build: every kept archive held first, then each eligible reference let go and recorded + as collected, stopping at the first refusal by the store. It is bounded by a count and a time, says + what is left, and is a hand-act recorded with its `why`. + +## The package registry + +The forge module's Go bundle reads the registry of the owner its seat serves, and the files of every +repository on the forge. + +- A **referenced version** is one a lockfile on a repository's default branch names; the highest version + satisfying a dependency range a package manifest there names (dependencies, development, peer and + optional); or one a dist-tag names. +- **Kept** is referenced, or among the newest `keep` versions of its package (five by default). +- **`npm_packages`** answers each package with its versions, each version's publish time and size, and + why it is kept or not. +- **`npm_retention`** answers what it would delete, version by version, and the bytes. With `dry_run` + false and a `why` it deletes those versions through the forge's own interface. A repository whose + files could not be read is named in every answer, and stops a real run before anything is deleted. + +Ranges are read as npm reads them: exact, `^`, `~`, `x` and `*`, comparisons, hyphen ranges and `||`. +A range that cannot be read is named, and its package keeps every version until it can. + +## A machine's images + +The container runtime module's tool `docker_prune_images` works on the machine it runs on. + +1. It lists every image, with its names, its digests, its age and its size, and every container, in + any state. +2. It asks the controller's verb **`images`** for this machine: every container image the machine's + declaration names, both the one the mesh would send now and the one it was last sent. No answer + means nothing is removed. +3. An image is **kept** when a container uses it, when either declaration names it, or when it is + younger than the floor (`older_than_days`, seven by default). +4. Images are grouped into **lines**: the images sharing a repository name, joined across names when + one image carries more than one. In each line that holds a kept image, the newest image that is not + already kept is kept too: the previous version. +5. Everything else is removed by id, never forced. The runtime refuses an image a container uses, and a + refusal is reported, not retried. + +The answer lists each image with whether it is kept and why, the bytes that would be freed (summed per +image, and stated as an upper bound because images share layers), and, for a real run, what was +removed and what the runtime refused. + +## What stays as it is + +- The store's nightly collector, held still while it runs (ADR 0189 §4). +- The sweep after each build, bounded (ADR 0189 §5). +- The container runtime's weekly prune of dangling images and build cache. +- No new timer: each tool runs when a person runs it. + +## Where it stands + +*In progress, 2026-10-08.* The decision is recorded; the code is being built in the controller and in +the store, forge and container runtime modules. The live dry runs are recorded here once the tools run. diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index e7f746f4..215b5137 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -50,6 +50,7 @@ document is written and this one's status becomes `implemented`. | [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the walk, and the core's own updates never wait for the module | [ADR 0239](../../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md), [ADR 0238](../../02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md), [ADR 0236](../../02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md) | | [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of ten domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | [ADR 0244](../../02-DECISIONS/0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md), [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) | | [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | [ADR 0247](../../02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md), [ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) | +| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | [ADR 0251](../../02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md), [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md) | ## Not yet written From f91735b62944199bc33f34a45e5bbed2b3a039de Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 12:08:02 +0200 Subject: [PATCH 2/2] Put the store's tools in a module beside it, since the store's module may not build (ADR 0251) --- ...ps-the-image-it-runs-and-the-one-before.md | 31 ++++++++++++------- .../51-the-registries-keep-what-is-named.md | 21 +++++++------ 2 files changed, 31 insertions(+), 21 deletions(-) diff --git a/02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md b/02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md index 76bec15a..ca205303 100644 --- a/02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md +++ b/02-DECISIONS/0251-the-registries-say-what-they-hold-and-keep-what-is-named-and-a-machine-keeps-the-image-it-runs-and-the-one-before.md @@ -44,7 +44,10 @@ registry, and pruning of unused images on a machine — each a dry run unless as in three steps ([ADR 0246](0246-a-seats-new-verb-is-promised-before-it-is-required.md)), and the store has one holder and no second in view. Not now; the tools can be promoted when a second holder exists. -2. **The store module's own tools, in Go.** Chosen. They replace the TypeScript client nothing builds. +2. **A module's own tools, in Go, on the machine that holds the store.** Chosen. Not the store module + itself: a module that provides the artifact store may not build an artifact, because building + publishes to the store, so the tools are a second module, `artifact-store-tools`, assigned beside the + store. The store module's unbuilt TypeScript client goes. 3. *The controller's verbs.* The controller holds the records but not the store's files; the store may move to a machine the controller is not on. Rejected for everything except what only the records can answer. @@ -52,7 +55,7 @@ registry, and pruning of unused images on a machine — each a dry run unless as **For what the store tools read.** The store's door lists repositories and tags, but not a manifest no tag names — and since the mesh pins by digest, that is almost every manifest. Only the store's files list them, with each blob's size. So the tools read the store's own files, read-only, through the -store's own container, and read each manifest's content through the door. The registry's own +store's own container, manifests' content included. The registry's own `garbage-collect --dry-run` was weighed and rejected as the source: it says what is unmarked, but not how large, and not what letting go of a manifest would free. @@ -84,29 +87,33 @@ decides. ## Decision -**1. The store module answers for the store, with four tools of its own.** The distribution module, -holder of `mesh-artifact-store`, serves a Go bundle in place of its unbuilt TypeScript: +**1. A module beside the store answers for it, with four tools.** The module `artifact-store-tools`, +assigned to the machine that holds `mesh-artifact-store`, serves a Go bundle. The store module itself +cannot carry it, because a module providing the artifact store may not build an artifact; and the +tools are named for the artifact store, because *store* alone is the database server: -- `store_repositories` — every repository, its tags, how many manifests it holds and its size; -- `store_usage` — the store's size, the largest repositories, what blobs are shared, and what the +- `artifact_store_repositories` — every repository, its tags, how many manifests it holds and its size; +- `artifact_store_usage` — the store's size, the largest repositories, what blobs are shared, and what the store's nightly collector would free now; -- `store_references` — what the records say of each manifest the store holds (below), and what the +- `artifact_store_references` — what the records say of each manifest the store holds (below), and what the records keep that the store does not hold; -- `store_collect` — a dry run unless `dry_run` is false; a real run needs `why`. +- `artifact_store_collect` — a dry run unless `dry_run` is false; a real run needs `why`. -The tools read the store's files through the store's own container and only read them. The first three -change nothing; `store_collect` changes nothing either, unless it is a real run, and then only by asking +The tools read the store's files, manifests included, through the store's own container and only read +them. The first three +change nothing; `artifact_store_collect` changes nothing either, unless it is a real run, and then only by asking the controller (point 3). **2. The records are asked, never copied.** A new controller verb, `artifacts`, answers every reference the mesh recorded making, with its state: *kept* because a definition names it, *kept* because it belongs to one of the five most recent builds of a module the mesh holds, *eligible* (made, -not kept, not yet let go), or — when asked — *collected*. `store_references` sets that beside what the +not kept, not yet let go), or — when asked — *collected*. `artifact_store_references` sets that beside what the store holds and says, for every manifest, one of: - **kept**, with the reason; - **a holder of a kept archive** (the manifest that keeps an archive's blob, [issue 253](../04-ISSUES/253-the-stores-collector-would-delete-every-archive-the-mesh-keeps/00-report.md)); - **eligible**, or a holder of an eligible archive; +- **let go of, yet present** — the controller recorded letting it go and the store still holds it; - **a named document** — something the controller keeps under a tag (to-be 45 §9); - **unrecorded** — no record names it. @@ -116,7 +123,7 @@ record.** ADR 0189 §3 stands as written. **3. Collecting on demand is the same sweep, asked by a person.** A new controller verb, `collect`, says what the sweep would let go of, and does it only with `confirm` and a `why`. A real run holds every kept archive before it lets anything go, exactly as the sweep after a build does; it is bounded by a -count and a time, says what it left; and it is recorded as a hand-act with its `why`. `store_collect`'s +count and a time, says what it left; and it is recorded as a hand-act with its `why`. `artifact_store_collect`'s dry run adds what only the store knows: how many bytes the nightly collector would free once those manifests are gone, beside what it would free tonight anyway. Its real run asks `collect`. Bytes are still reclaimed by the nightly collector, because only it runs with the store held still. diff --git a/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md b/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md index 11e863a4..86518bf0 100644 --- a/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md +++ b/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md @@ -3,7 +3,7 @@ layer: to-be status: in-progress code: - mesh-controller cmd/mesh-controller (the verbs artifacts, collect and images) - - mesh-catalog modules/distribution (the store's tools) + - mesh-catalog modules/artifact-store-tools (the store's tools), modules/distribution (its unbuilt code removed) - mesh-catalog modules/gitea (package retention) - mesh-catalog modules/docker (image pruning) updated: 2026-10-08 @@ -28,9 +28,10 @@ extending [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-na controller (records) artifacts ──┐ collect ──┐ images ──┐ │ │ │ - store module ─────┴─────────────┘ │ - (store_repositories, store_usage, │ - store_references, store_collect) │ + ┌─────────────────┴─────────────┘ │ + artifact-store-tools │ + (artifact_store_repositories, _usage, │ + _references, _collect) │ │ reads the store's files │ │ through its own container │ ▼ │ @@ -52,7 +53,8 @@ extending [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-na The store keeps each blob once, named by its digest, with its size on disk. Each repository lists the manifests it holds (every one, whether a tag names it or not), its tags, and the layers it may serve. The tools list those files through the store's own container, read-only, and read each manifest's -content through the store's door. Nothing is written. +content the same way. Nothing is written. The tools are their own module, `artifact-store-tools`, +assigned to the machine that holds the store: the store's module may not build an artifact itself. From that, a manifest *marks* its own blob, its configuration and each layer, and an index marks the manifests it lists. The store's nightly collector removes every blob no manifest marks. The tools work @@ -60,15 +62,15 @@ the same mark, so their numbers are the collector's. ### What each tool answers -- **`store_repositories`** — each repository with its tags, its manifest count and its size: the sum of +- **`artifact_store_repositories`** — each repository with its tags, its manifest count and its size: the sum of the blobs its manifests mark. A blob shared by two repositories is counted in each, and the answer says so. -- **`store_usage`** — the store's total size; the largest repositories; the bytes in blobs that more +- **`artifact_store_usage`** — the store's total size; the largest repositories; the bytes in blobs that more than one repository marks; and the bytes in blobs no manifest marks, which is what the nightly collector frees next. -- **`store_references`** — for every manifest, one state (below), counted and sized by repository, and +- **`artifact_store_references`** — for every manifest, one state (below), counted and sized by repository, and listed when one repository is asked about. Then what the records keep that the store does not hold. -- **`store_collect`** — the dry run: what the controller would let go of, and how many bytes the nightly +- **`artifact_store_collect`** — the dry run: what the controller would let go of, and how many bytes the nightly collector would free once those are gone. A real run with `dry_run` false and a `why` asks the controller's `collect`, and answers what it did. @@ -80,6 +82,7 @@ the same mark, so their numbers are the collector's. | kept — recent build | one of the five most recent builds of a module the mesh holds | no | | holder of a kept archive | the manifest that keeps a kept archive's blob | no | | eligible | the mesh made it and keeps it for no reason | yes, through the controller | +| let go of, yet present | the controller recorded letting it go, and the store still holds it | no — said, for a person to look at | | holder of an eligible archive | the manifest that keeps an eligible archive's blob | yes, with its archive | | named document | a manifest the controller keeps under a tag (to-be 45 §9) | no | | unrecorded | no record names it | no — counted and sized only |