hq/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md

113 — Diagnosis

2026-09-24 — the trail

The symptom arrived already carrying a diagnosis: the registry has disabled anonymous pulls for the entire vendor namespace. Everything below was an attempt to confirm that, and it did not survive.

Step 1 — the token is not the test

The reported evidence was the anonymous pull token's contents: "actions": [] and "$disabled". A token is an intermediate artifact. The test is whether a manifest can actually be fetched with it, so the first step was to request one:

GET /v2/minio/minio/manifests/<pinned tag>      -> 401 UNAUTHORIZED

That confirmed the failure but said nothing about its scope or cause.

Step 2 — a control ruled out the stated cause

The same request flow, in the same minute, against other repositories:

repository	token actions	manifest
the vendor's server	[]	401
the vendor's client	[]	401
the vendor's operator	['pull']	200
the vendor's console	['pull']	200
the vendor's sidecar proxy	['pull']	200
an unrelated public project	['pull']	200

A namespace-wide policy is ruled out. Two repositories fail; their siblings in the same namespace pull normally.

Step 3 — both pieces of the original evidence were red herrings

• "$disabled" is present on every repository on that registry, including all of the successful ones above. It belongs to the signing context, not to authorisation. It carries no information about this failure at all.
• "actions": [] with a 401 is indistinguishable from a repository that does not exist. A deliberately invented repository name in the vendor's namespace returned a byte-identical response — empty actions, 401. The signal cannot separate access revoked from not there, so it cannot support the conclusion it was used for.

That second point turned the question from who revoked access to is it still there.

Step 4 — the registries' own APIs establish deletion

Asked directly, rather than through the pull path:

• Primary registry: its repository API answers 404 for the server repository, and 200 for a sibling in the same namespace. The repository is gone, not private.
• Secondary registry: a listing of the vendor's namespace returns 68 public repositories. The server and the client are absent from the list. Present are the operator, console, sidecar proxy, benchmarking and key-management images — and a large, newer set belonging to the vendor's commercially licensed line.

Step 5 — upstream confirms, and dates it

The vendor deleted the community server and client from the primary registry on 2026-09-11. This was the last step of a staged withdrawal: free image publishing stopped in October 2025, the community console UI was removed mid-2025, and the open-source repository was archived in February 2026. The secondary registry was where the ecosystem repointed as a stopgap; it has since lost the two repositories as well.

Conclusion: the images were withdrawn, not restricted. No credential can answer this, because there is nothing left to authenticate against. A different image source is the only remedy.

Why the mesh kept working, checked rather than assumed

The claim that the module "genuinely isn't ready for cutover" was tested and is false for the node in question.

• The node holds the pinned server image, the client, and the load-balancer image the module composes with — all pulled years before the withdrawal.
• The module's resolved version variable on that node matches the cached tag exactly, so the composition references an image that is present.
• The composition declares no pull policy, so a present tag is used as-is.
• The deploy stage pulls best-effort, then asserts only that every declared image resolves locally. A pull error becomes a logged warning when all images are present.
• The start path restarts the unit and does not pull. The one --pull always in the tree sits inside a generated guide describing the superseded approach, not in the code that writes units.

So a deploy of this module on that node succeeds today.

RETRACTED — the four claims this diagnosis called unsubstantiated

Added 2026-09-24, the same day, after the error was pointed out.

This diagnosis originally carried a table headed "Claims in the original report that could not be substantiated", asserting that a module.json did not exist, that no digest pin existed, that an all-zeros runtime digest appeared nowhere, and that a readiness document was not on disk. The table was wrong and it is withdrawn in full. The original report was accurate.

Claim, as reported	Actual finding
The manifest is a module.json pinning the server image by digest at line 83	True, and exactly. modules/minio/module.json, digest pin, line 83.
The module's own runtime artifact carries an all-zeros placeholder digest	True. A second container resource pins a runtime sidecar at an all-zeros digest, meaning nothing was ever published for it.
A dated readiness document records several modules with placeholder digests	Unverified, not disproven. It is not on the machine searched. The migration record is a separate private repository that is not checked out there, so its absence locally is not evidence.
The cached server image is an older release than the module pins	Unverified. What was checked was the predecessor's tag pin against the cache, which did match. Whether the cached image is the digest this module pins was never checked.

Why it went wrong, stated plainly

One repository was searched, and absence in it was reported as absence. The catalogue of the mesh being built is a separate repository, not checked out on the machine where the search ran. Every one of the four claims was about that repository. The searches were real and their output was reported honestly; the inference drawn from them was not warranted.

Compounding it, the predecessor's object-store module and the one being cut over to were treated as the same thing. They are different files, in different repositories, with different shapes: the predecessor's is a compose file pinning a tag with a version variable, and it declares no sidecar; the one being cut over to is a JSON manifest pinning a digest, and it declares two container resources. Findings about the first were written up as findings about the second.

The lesson worth keeping, because it is not specific to this issue: "zero occurrences anywhere in the tree" is only ever as strong as the tree that was searched, and a diagnosis must name which tree that was. This one did not, which is what let a one-repository search read as a mesh-wide fact. A confident rebuttal of a correct report is worse than no diagnosis, because it sends the next person looking in the wrong place with the authority of a written record behind them.

What none of this changes: the images are gone upstream, and that finding stands on the registries' own APIs.

What is located, and what is not

Located: the object-store module in the catalogue of the mesh being built — it pins, by digest, a server image that no longer exists anywhere public, and a runtime sidecar that was never published.

Not located, and deliberately left open: the general condition. The mesh has no mirror of the third-party images its modules depend on and no check that a module is obtainable by a node holding nothing. That is a design gap rather than a defect in this module, and it is stated as an open question on the report rather than answered here.