diff --git a/02-DECISIONS/0257-a-copied-base-is-recorded-where-it-is-copied-kept-by-the-builds-that-stood-on-it-and-held-once-per-image.md b/02-DECISIONS/0257-a-copied-base-is-recorded-where-it-is-copied-kept-by-the-builds-that-stood-on-it-and-held-once-per-image.md index f11f461a..cea384a6 100644 --- a/02-DECISIONS/0257-a-copied-base-is-recorded-where-it-is-copied-kept-by-the-builds-that-stood-on-it-and-held-once-per-image.md +++ b/02-DECISIONS/0257-a-copied-base-is-recorded-where-it-is-copied-kept-by-the-builds-that-stood-on-it-and-held-once-per-image.md @@ -78,10 +78,13 @@ Windows included. 1. *Let the index go alone.* Rejected for fact 3. 2. *Let the index go with every platform manifest it names.* Rejected, because a kept index of the same repository may name one of them. -3. **Let the index go with the platform manifests it names that no kept index of the repository also - names, platforms first.** Chosen. The controller reads only its records, so it cannot see an - unrecorded index that also names a platform. The store tools can see that, and their dry run lists - every manifest that stays and would lose a platform, before anybody runs collection for real. +3. *The same, in every sweep.* Rejected. The sweep after each build is real collection with no dry run + in front of it. The controller reads only its records, so it cannot see an unrecorded index, or a copy + a build is making at that moment, that names the same platform. +4. **Let the index go with the platform manifests it names that no kept index of the repository also + names, platforms first, only in a collect a person confirmed.** Chosen. The store tools can see every + manifest in a repository, and their dry run lists each one that stays and would lose a platform. A + person reads that before confirming. The sweep after a build lets an index go alone, as it did before. **The copies no record names**, made before this record by a failed build, or before builds kept what they stood on. @@ -93,7 +96,7 @@ they stood on. tools list each unrecorded copy at the top of a copy's repository, with what it alone marks. A new controller verb, `mirrors`, records the ones a person names, with a reason. It refuses any reference outside a repository only the copy step writes, and any reference the store does not hold. Recording - deletes nothing. + deletes nothing itself; a recorded copy then goes at the next sweep unless a kept build stood on it. **Copying only the platforms the mesh runs** was weighed and is left for a later record. It would remove most of the route proxy's toolchain copy, which is Windows images, but the reference a build is handed @@ -105,7 +108,9 @@ partly absent depends on its version and settings, and that has not been measure **1. One repository per upstream image.** A copied base is put in `upstream//` (the public hub is `docker.io`, and a port's colon becomes a dash). The module's own repository from before is offered as a source: where it holds the image by digest, the copy is made from it, and nothing is asked -of upstream. A module's build arguments name the new copy. Its source fingerprint is unchanged, because +of upstream. A copy counts as held only when its index and every platform manifest the index names are +held; otherwise it is copied again, which puts back what is missing. A module's build arguments name the +new copy. Its source fingerprint is unchanged, because the fingerprint reads a base by digest, never by where it was copied to. **2. A copy is recorded where it is copied.** The builder says, in every build outcome, what it copied. @@ -117,19 +122,23 @@ named it. Only the copy step ever wrote a repository so named. **3. A copy is kept by the kept builds that stood on it, and by nothing else.** A copy is kept, with the reason `stood-on`, while one of the five most recent successful builds of a module the mesh holds stood on it. Otherwise it is eligible, and the sweep lets it go as it lets any eligible artifact go. A -definition never keeps a copy. A copy already let go of stays let go of. The reason `stood-on` belongs to +definition never keeps a copy. A copy already let go of stays let go of, until a build copies it again: +the build then holds it, and its record of being let go of is removed. The reason `stood-on` belongs to copies alone: another module's artifact is still kept by that module's own builds. -**4. An index goes with its platforms.** When the sweep lets an index go, it first deletes the platform -manifests the index names that no kept index of the same repository names, and deletes the index last. A -sweep cut short leaves the index to be asked about again. If the sweep cannot read which platforms the -kept indexes name, it lets nothing go. This applies to every index the mesh let go of, upstream artifacts -included. +**4. An index goes with its platforms only in a collect a person confirmed.** Before its first delete, +a confirmed `collect` reads, for every repository it will touch, which platform manifests the kept +indexes there name. If any of those reads fails, it lets nothing go. Then, for each index it lets go of, +it first deletes the platform manifests that no kept index of the same repository names, and deletes the +index last. A collect cut short leaves the index to be asked about again. The sweep after a build lets an +index go alone. This applies to every index the mesh lets go of, upstream artifacts included. **5. A person may record a copy that no record names.** The controller's `mirrors` verb lists the copies the records hold, with their states. Given references, it records the ones that are in a repository only the copy step writes and that the store holds. It is a dry run without `confirm`. With `confirm` it needs -a `why`, and it is recorded as a hand act. It deletes nothing. `artifact_store_references` lists the +a `why`, and it is recorded as a hand act once the copies are recorded. It deletes nothing itself, but a +recorded copy is eligible, and the next sweep lets it go unless a kept build stood on it; the dry run says +so before anything is recorded. `artifact_store_references` lists the candidates in `unrecorded_mirrors`, largest first. **6. The store tools count a platform as its index's.** A platform manifest takes the state of the @@ -153,6 +162,11 @@ ADR 0097 is superseded in one sentence only: a base is copied into the image's o - Most of the 23.8 GiB is copies no record names, and pre-mesh images (research [035](../01-RESEARCH/035-the-images-from-before-the-mesh/00-overview.md)). Those wait for a person, through `mirrors` and, for the pre-mesh images, a decision of its own. +- **Rolling the controller back is not free once old copies are gone.** After this is rolled out, the + modules' former `/on-` copies stop being stood on and are let go of. A controller + rolled back to the version before this record names those repositories again. It finds them empty and + copies every base from the public registry again, which rations anonymous pulls. A rollback after that + point should be planned around the pull limit. - The move to one repository per image costs one copy per image, taken from the store itself. After it, each module's former copies stop being stood on, become eligible after five more builds of that module, and go, freeing their manifests. @@ -169,7 +183,10 @@ ADR 0097 is superseded in one sentence only: a base is copied into the image's o - **The builder.** - The repository is named for the image: the hub, other hosts, a port, upper case. - A copy held under the module's former repository is mounted, every blob, with nothing uploaded - again and nothing asked of an upstream that is gone. + again and nothing asked of an upstream that is gone. A store that answers a mount with an upload + gets each blob uploaded. + - An index missing a platform, in the image's repository or the former one, is not taken as held. + - Builds copying while platforms are let go of under them end with the copy whole. - A second module standing on the image copies nothing. - A failed build and a working build both say what they copied. - **The records.** @@ -177,16 +194,19 @@ ADR 0097 is superseded in one sentence only: a base is copied into the image's o - A copy is kept, with `stood-on`, by the five kept builds and by nothing else, and a definition naming the image keeps no copy. - Another module's artifact is not kept as a copy. - - A collected copy stays collected. + - A collected copy stays collected, and a copy a build copies again is held again; a person's record + never brings a collected copy back. - A person's record stands over a later build's. - The migration's backfill records one copy, by its earliest build, and records nothing outside an `on-` repository. - A copy's repository draws no build edge. - **The sweep.** - - Against a fake store, an index goes with its own platforms, platforms first. - - A platform a kept index names stays. + - Through the records, a confirmed collect lets an eligible index go with its own platform, platforms + first, and leaves the platform a kept index of the same repository names. + - The sweep after a build lets an index go alone. - An image is one delete. - - A store that cannot say what a kept index names keeps the index. + - A kept index the store will not answer for stops a confirmed collect before its first delete, and an + index in a repository not read beforehand is kept. - **The `mirrors` verb.** - It is composed only with what it takes. - It takes only references in a copy's repository. diff --git a/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md b/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md index 66a90111..b91ea5c7 100644 --- a/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md +++ b/03-DESIGN/01-to-be/51-the-registries-keep-what-is-named.md @@ -111,8 +111,12 @@ A build that stands on an image published elsewhere copies it into the store fir build records already named were recorded by the migration that made the table. - **Kept by the kept builds that stood on it** (`stood-on`), and by nothing else: a definition never keeps a copy. Otherwise a copy is eligible, like any artifact the mesh made. -- **An index goes with its platforms.** The sweep deletes the platform manifests an index names that - no kept index of its repository also names, and deletes the index last. +- **Held whole.** A copy counts as held only when its index and every platform the index names are + held; otherwise it is copied again. A copy a build copies again is no longer recorded as let go of. +- **An index goes with its platforms only in a confirmed collect.** Before its first delete, it reads + which platforms the kept indexes of every repository it touches name, and lets nothing go if a read + fails. It deletes the platform manifests an index names that no kept index of its repository also + names, and deletes the index last. The sweep after a build lets an index go alone. - **A copy no record names** waits for a person. `artifact_store_references` lists each one in `unrecorded_mirrors`, with what it alone marks. The controller's `mirrors` verb records the ones a person names, and the sweep then decides about them. @@ -125,11 +129,13 @@ A build that stands on an image published elsewhere copies it into the store fir - **`collect`** says what the sweep would let go of. With `confirm` and a `why` it runs the sweep that runs after a build: every kept archive held first, then each eligible reference let go and recorded as collected, stopping at the first refusal by the store. It is bounded by a count and a time, says - what is left, and is a hand-act recorded with its `why`. An index is let go of with its own platform - manifests (ADR 0257). + what is left, and is a hand-act recorded with its `why`. A confirmed collect lets an index go with its + own platform manifests (ADR 0257); the sweep after a build lets it go alone. - **`mirrors`** lists the copied bases the records hold, with their states. Given references, it records the copies that are in a repository only the copy step writes and that the store holds. It is a dry - run without `confirm`; with `confirm` it needs a `why`, and it is a hand act. It deletes nothing. + run without `confirm`; with `confirm` it needs a `why`, and it is a hand act, logged once the copies + are recorded. It deletes nothing itself, but a recorded copy is eligible: the next sweep lets it go + unless a kept build stood on it, and both the dry run and the real run say so. `artifact_store_collect` counts the platforms that go with an index, and its dry run lists, in `platforms_named_by_what_stays`, every manifest that stays and names a platform that would go. The diff --git a/04-ISSUES/321-a-copied-base-was-recorded-by-nothing-and-a-let-go-index-kept-its-platforms/01-diagnosis.md b/04-ISSUES/321-a-copied-base-was-recorded-by-nothing-and-a-let-go-index-kept-its-platforms/01-diagnosis.md index 80d9149d..1d6f2100 100644 --- a/04-ISSUES/321-a-copied-base-was-recorded-by-nothing-and-a-let-go-index-kept-its-platforms/01-diagnosis.md +++ b/04-ISSUES/321-a-copied-base-was-recorded-by-nothing-and-a-let-go-index-kept-its-platforms/01-diagnosis.md @@ -51,5 +51,12 @@ unrecorded platform manifests left behind by the indexes collected before it. sweep reads), and the store client (what letting an index go deletes). The store's tools are located there too, because they count a platform manifest apart from its index. +**What the fix leaves.** The 16 platform manifests in the route proxy's upstream-artifact repository +stay. The indexes that named them were let go of before this fix, so no record names them, and neither +the sweep nor a confirmed collect will touch them. They can be removed in one of two ways, each a +person's act. One is the `mirrors` verb's way of recording them, if the decision is extended to the +repository of an upstream artifact. The other is a decision on how the store lets go of manifests no +record names (research 035 asks the same question for the pre-mesh images). + **The fix** is [ADR 0257](../../02-DECISIONS/0257-a-copied-base-is-recorded-where-it-is-copied-kept-by-the-builds-that-stood-on-it-and-held-once-per-image.md) and its two pull requests (the controller and the catalogue's store tools).