From 999636e2a8c72173d42c9c845ddcb2178d0b89c4 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 24 Sep 2026 18:46:47 +0200 Subject: [PATCH] =?UTF-8?q?Issue=20113:=20retract=20the=20diagnosis=20tabl?= =?UTF-8?q?e=20=E2=80=94=20the=20original=20report=20was=20right?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The diagnosis carried a table headed "claims that could not be substantiated", denying a module.json, a digest pin, and an all-zeros runtime digest. All three exist. The table is withdrawn in full and replaced with what is actually true, plus the two claims that remain genuinely unverified rather than disproven. The cause: one repository was searched and absence in it was written up as absence. The catalogue of the mesh being built is a separate repository, not checked out where the search ran, and all four claims were about that repository. Compounding it, the predecessor's object-store module and the one being cut over to were treated as one thing — they are different files in different repositories, one pinning a tag with no sidecar, the other a digest with two container resources. Also corrected in the report: located-in named the wrong repository; the "pins a tag" passage described the predecessor; the open question about pinning by digest is struck, because this module already does and it made no difference — a deleted digest resolves to nothing either way. The section on why nothing broke is now scoped explicitly to the predecessor's machinery. The lesson kept in the record: "zero occurrences anywhere in the tree" is only as strong as the tree searched, and a diagnosis must say which tree. A confident rebuttal of a correct report is worse than no diagnosis — it sends the next person to the wrong place with a written record behind them. --- .../00-report.md | 34 +++++++++----- .../01-diagnosis.md | 46 +++++++++++++++---- 2 files changed, 59 insertions(+), 21 deletions(-) diff --git a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md index 7573c96..09a951e 100644 --- a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md +++ b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md @@ -1,7 +1,7 @@ --- status: located opened: 2026-09-24 -located-in: [hal modules/minio] +located-in: [mesh-catalog modules/minio] fixed-by: amended-design: --- @@ -19,11 +19,14 @@ pull with `401 UNAUTHORIZED`: /minio/mc 401 /minio/console 200 ``` -The module pins a **tag**, not a digest, and the default is four and a half years old: +The module pins the server image **by digest**, in its `module.json`: -``` -image: /minio/minio:${MINIO_VERSION:-RELEASE.2022-01-07T01-53-23Z} -``` + "image": "/minio/minio@sha256:14cea493…" + +*Corrected 2026-09-24. This first said the module pinned a **tag** whose default was four and a +half years old. That described the **predecessor** mesh's object-store module — a different file +in a different repository — not the module being cut over to. The conflation, and what it cost, +is retracted in full in [the diagnosis](01-diagnosis.md).* The cause is upstream and outside the mesh: the vendor **deleted** the community server and client repositories. It is not an access policy that a credential could answer, and nothing about the @@ -54,7 +57,12 @@ vendor namespace*. That was wrong in a way worth keeping, because the evidence l Sibling repositories in the same namespace pulling normally is what rules out a namespace-wide policy, and the registries' own APIs — `404` against `200` — are what establish deletion. -## The mesh was not blocked, which is the other half +## The predecessor mesh was not blocked, which is the other half + +*Scope, corrected 2026-09-24: everything in this section describes the **predecessor's** delivery +machinery and its object-store module. It is what made the instance harmless, and it is why +dropping the module from the queue was unnecessary. It says nothing about how the mesh being built +resolves images, which is a different mechanism.* A node that already holds the images runs the module normally. The node carrying the cutover holds the pinned server image, the client, and the load-balancer image the module composes with, all @@ -96,9 +104,10 @@ The instance is harmless; the standing condition is not. warning is not a rule. A mesh cannot state that its modules are installable while the only evidence is that they are already installed. -The third point is the general one, and it is not specific to this vendor: an image pinned by tag -against a registry the mesh does not control is a dependency with no guarantee behind it, and the -mesh currently learns it has lost one only by trying to use it. +The third point is the general one, and it is not specific to this vendor: an image pinned against +a registry the mesh does not control — **by tag or by digest, it makes no difference** — is a +dependency with no guarantee behind it, and the mesh currently learns it has lost one only by +trying to use it. ## Open questions @@ -107,8 +116,11 @@ mesh currently learns it has lost one only by trying to use it. goodwill? That is the fix that generalises. It costs storage and a policy about what to mirror, and it is a deliberate move **away** from references-over-payload for third-party images specifically — so it should be decided as such, not smuggled in as a fix. -- Should a module's images be pinned **by digest** rather than by tag? It makes the artifact - exact and auditable, but does nothing about withdrawal — a deleted digest is just as gone. +- ~~Should a module's images be pinned **by digest** rather than by tag?~~ **Answered, and the + premise was wrong.** This module already pins by digest, and it made no difference: the + repository was deleted, so the digest resolves to nothing. A digest buys an exact, auditable + artifact; it buys no protection whatever against withdrawal. Struck rather than deleted, because + the question was asked from a mistaken reading of the manifest and that is worth seeing. - What **checks** that every module in the catalogue is still obtainable from a node that holds nothing? Nothing does today. A periodic cold-pull of the catalogue would have caught this on 2026-09-11 rather than thirteen days later, mid-cutover. diff --git a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md index 77f1cdf..54b1d92 100644 --- a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md +++ b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md @@ -86,23 +86,49 @@ in question. So a deploy of this module on that node succeeds today. -## Claims in the original report that could not be substantiated +## RETRACTED — the four claims this diagnosis called unsubstantiated -Recorded because they were specific and load-bearing, and acting on them would have wasted time. +*Added 2026-09-24, the same day, after the error was pointed out.* -| Claim | Finding | +This diagnosis originally carried a table headed *"Claims in the original report that could not be +substantiated"*, asserting that a `module.json` did not exist, that no digest pin existed, that an +all-zeros runtime digest appeared nowhere, and that a readiness document was not on disk. **The +table was wrong and it is withdrawn in full.** The original report was accurate. + +| Claim, as reported | Actual finding | |---|---| -| The module's manifest is a `module.json`, pinning the server image by digest at line 83 | There is **no `module.json` anywhere** in the monorepo. The manifest is YAML and pins a **tag**. No digest pin exists. | -| The module's own runtime artifact is a placeholder with an all-zero digest | **Zero occurrences** of that image name or of an all-zero digest anywhere in the tree. The module declares no runtime or sidecar artifact. | -| A dated readiness document records six modules with placeholder digests | **No such file exists.** | -| The server image is cached locally at an older release than the module pins | The cached tag is **exactly** the pinned one, not an older release. | +| The manifest is a `module.json` pinning the server image by digest at line 83 | **True, and exactly.** `modules/minio/module.json`, digest pin, line 83. | +| The module's own runtime artifact carries an all-zeros placeholder digest | **True.** A second container resource pins a runtime sidecar at an all-zeros digest, meaning nothing was ever published for it. | +| A dated readiness document records several modules with placeholder digests | **Unverified, not disproven.** It is not on the machine searched. The migration record is a separate private repository that is not checked out there, so its absence locally is not evidence. | +| The cached server image is an older release than the module pins | **Unverified.** What was checked was the *predecessor's* tag pin against the cache, which did match. Whether the cached image is the digest this module pins was never checked. | -None of these change the real finding, which stands: the images are gone upstream. +### Why it went wrong, stated plainly + +**One repository was searched, and absence in it was reported as absence.** The catalogue of the +mesh being built is a **separate repository**, not checked out on the machine where the search ran. +Every one of the four claims was about that repository. The searches were real and their output was +reported honestly; the inference drawn from them was not warranted. + +Compounding it, the predecessor's object-store module and the one being cut over to were treated as +the same thing. They are different files, in different repositories, with different shapes: the +predecessor's is a compose file pinning a **tag** with a version variable, and it declares no +sidecar; the one being cut over to is a JSON manifest pinning a **digest**, and it declares two +container resources. Findings about the first were written up as findings about the second. + +**The lesson worth keeping, because it is not specific to this issue:** *"zero occurrences anywhere +in the tree"* is only ever as strong as the tree that was searched, and a diagnosis must name which +tree that was. This one did not, which is what let a one-repository search read as a mesh-wide fact. +A confident rebuttal of a correct report is worse than no diagnosis, because it sends the next +person looking in the wrong place with the authority of a written record behind them. + +What none of this changes: the images are gone upstream, and that finding stands on the registries' +own APIs. ## What is located, and what is not -**Located:** the module in the monorepo's catalogue — it pins, by tag, an image that no longer -exists anywhere public. +**Located:** the object-store module in the catalogue of the mesh being built — it pins, by digest, +a server image that no longer exists anywhere public, and a runtime sidecar that was never +published. **Not located, and deliberately left open:** the general condition. The mesh has no mirror of the third-party images its modules depend on and no check that a module is obtainable by a node