diff --git a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md index 7573c96..09a951e 100644 --- a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md +++ b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md @@ -1,7 +1,7 @@ --- status: located opened: 2026-09-24 -located-in: [hal modules/minio] +located-in: [mesh-catalog modules/minio] fixed-by: amended-design: --- @@ -19,11 +19,14 @@ pull with `401 UNAUTHORIZED`: /minio/mc 401 /minio/console 200 ``` -The module pins a **tag**, not a digest, and the default is four and a half years old: +The module pins the server image **by digest**, in its `module.json`: -``` -image: /minio/minio:${MINIO_VERSION:-RELEASE.2022-01-07T01-53-23Z} -``` + "image": "/minio/minio@sha256:14cea493…" + +*Corrected 2026-09-24. This first said the module pinned a **tag** whose default was four and a +half years old. That described the **predecessor** mesh's object-store module — a different file +in a different repository — not the module being cut over to. The conflation, and what it cost, +is retracted in full in [the diagnosis](01-diagnosis.md).* The cause is upstream and outside the mesh: the vendor **deleted** the community server and client repositories. It is not an access policy that a credential could answer, and nothing about the @@ -54,7 +57,12 @@ vendor namespace*. That was wrong in a way worth keeping, because the evidence l Sibling repositories in the same namespace pulling normally is what rules out a namespace-wide policy, and the registries' own APIs — `404` against `200` — are what establish deletion. -## The mesh was not blocked, which is the other half +## The predecessor mesh was not blocked, which is the other half + +*Scope, corrected 2026-09-24: everything in this section describes the **predecessor's** delivery +machinery and its object-store module. It is what made the instance harmless, and it is why +dropping the module from the queue was unnecessary. It says nothing about how the mesh being built +resolves images, which is a different mechanism.* A node that already holds the images runs the module normally. The node carrying the cutover holds the pinned server image, the client, and the load-balancer image the module composes with, all @@ -96,9 +104,10 @@ The instance is harmless; the standing condition is not. warning is not a rule. A mesh cannot state that its modules are installable while the only evidence is that they are already installed. -The third point is the general one, and it is not specific to this vendor: an image pinned by tag -against a registry the mesh does not control is a dependency with no guarantee behind it, and the -mesh currently learns it has lost one only by trying to use it. +The third point is the general one, and it is not specific to this vendor: an image pinned against +a registry the mesh does not control — **by tag or by digest, it makes no difference** — is a +dependency with no guarantee behind it, and the mesh currently learns it has lost one only by +trying to use it. ## Open questions @@ -107,8 +116,11 @@ mesh currently learns it has lost one only by trying to use it. goodwill? That is the fix that generalises. It costs storage and a policy about what to mirror, and it is a deliberate move **away** from references-over-payload for third-party images specifically — so it should be decided as such, not smuggled in as a fix. -- Should a module's images be pinned **by digest** rather than by tag? It makes the artifact - exact and auditable, but does nothing about withdrawal — a deleted digest is just as gone. +- ~~Should a module's images be pinned **by digest** rather than by tag?~~ **Answered, and the + premise was wrong.** This module already pins by digest, and it made no difference: the + repository was deleted, so the digest resolves to nothing. A digest buys an exact, auditable + artifact; it buys no protection whatever against withdrawal. Struck rather than deleted, because + the question was asked from a mistaken reading of the manifest and that is worth seeing. - What **checks** that every module in the catalogue is still obtainable from a node that holds nothing? Nothing does today. A periodic cold-pull of the catalogue would have caught this on 2026-09-11 rather than thirteen days later, mid-cutover. diff --git a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md index 77f1cdf..54b1d92 100644 --- a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md +++ b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md @@ -86,23 +86,49 @@ in question. So a deploy of this module on that node succeeds today. -## Claims in the original report that could not be substantiated +## RETRACTED — the four claims this diagnosis called unsubstantiated -Recorded because they were specific and load-bearing, and acting on them would have wasted time. +*Added 2026-09-24, the same day, after the error was pointed out.* -| Claim | Finding | +This diagnosis originally carried a table headed *"Claims in the original report that could not be +substantiated"*, asserting that a `module.json` did not exist, that no digest pin existed, that an +all-zeros runtime digest appeared nowhere, and that a readiness document was not on disk. **The +table was wrong and it is withdrawn in full.** The original report was accurate. + +| Claim, as reported | Actual finding | |---|---| -| The module's manifest is a `module.json`, pinning the server image by digest at line 83 | There is **no `module.json` anywhere** in the monorepo. The manifest is YAML and pins a **tag**. No digest pin exists. | -| The module's own runtime artifact is a placeholder with an all-zero digest | **Zero occurrences** of that image name or of an all-zero digest anywhere in the tree. The module declares no runtime or sidecar artifact. | -| A dated readiness document records six modules with placeholder digests | **No such file exists.** | -| The server image is cached locally at an older release than the module pins | The cached tag is **exactly** the pinned one, not an older release. | +| The manifest is a `module.json` pinning the server image by digest at line 83 | **True, and exactly.** `modules/minio/module.json`, digest pin, line 83. | +| The module's own runtime artifact carries an all-zeros placeholder digest | **True.** A second container resource pins a runtime sidecar at an all-zeros digest, meaning nothing was ever published for it. | +| A dated readiness document records several modules with placeholder digests | **Unverified, not disproven.** It is not on the machine searched. The migration record is a separate private repository that is not checked out there, so its absence locally is not evidence. | +| The cached server image is an older release than the module pins | **Unverified.** What was checked was the *predecessor's* tag pin against the cache, which did match. Whether the cached image is the digest this module pins was never checked. | -None of these change the real finding, which stands: the images are gone upstream. +### Why it went wrong, stated plainly + +**One repository was searched, and absence in it was reported as absence.** The catalogue of the +mesh being built is a **separate repository**, not checked out on the machine where the search ran. +Every one of the four claims was about that repository. The searches were real and their output was +reported honestly; the inference drawn from them was not warranted. + +Compounding it, the predecessor's object-store module and the one being cut over to were treated as +the same thing. They are different files, in different repositories, with different shapes: the +predecessor's is a compose file pinning a **tag** with a version variable, and it declares no +sidecar; the one being cut over to is a JSON manifest pinning a **digest**, and it declares two +container resources. Findings about the first were written up as findings about the second. + +**The lesson worth keeping, because it is not specific to this issue:** *"zero occurrences anywhere +in the tree"* is only ever as strong as the tree that was searched, and a diagnosis must name which +tree that was. This one did not, which is what let a one-repository search read as a mesh-wide fact. +A confident rebuttal of a correct report is worse than no diagnosis, because it sends the next +person looking in the wrong place with the authority of a written record behind them. + +What none of this changes: the images are gone upstream, and that finding stands on the registries' +own APIs. ## What is located, and what is not -**Located:** the module in the monorepo's catalogue — it pins, by tag, an image that no longer -exists anywhere public. +**Located:** the object-store module in the catalogue of the mesh being built — it pins, by digest, +a server image that no longer exists anywhere public, and a runtime sidecar that was never +published. **Not located, and deliberately left open:** the general condition. The mesh has no mirror of the third-party images its modules depend on and no check that a module is obtainable by a node