From 17fb7c26d28a201ec8e529513862824895d5b9b6 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 24 Sep 2026 15:10:47 +0200 Subject: [PATCH] Issue 114: quay.io disabled anonymous pulls for minio's own images Found checking minio before assigning it on novox. Not a novox-side problem - DNS, daemon.json and the auth handshake are all fine; quay.io's own token for both quay.io/minio/mc and quay.io/minio/minio (the module's own pinned server image) comes back with actions:[] and $disabled. Worked four days ago at the 2026-09-20 rehearsal. Distinct from issue 064: that one was about the mesh's build environment reaching a declared vendor image. This is the vendor disabling the image for everyone, which a declared, pinned digest does nothing to protect against. Checks pass. --- .../00-report.md | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 04-ISSUES/114-quay-io-disabled-anonymous-pulls-for-minios-own-images/00-report.md diff --git a/04-ISSUES/114-quay-io-disabled-anonymous-pulls-for-minios-own-images/00-report.md b/04-ISSUES/114-quay-io-disabled-anonymous-pulls-for-minios-own-images/00-report.md new file mode 100644 index 0000000..5aa5fc5 --- /dev/null +++ b/04-ISSUES/114-quay-io-disabled-anonymous-pulls-for-minios-own-images/00-report.md @@ -0,0 +1,68 @@ +--- +status: located +opened: 2026-09-24 +located-in: [mesh-catalog modules/minio] +fixed-by: +amended-design: +--- + +# 114 — quay.io disabled anonymous pulls for minio's own images, breaking the module's pin + +## What was observed + +On the control-node, 2026-09-24, checking `minio` before assigning it — the next module in the +`novox` queue after the forge and analytics. `scripts/compare-module-with-running.py` reported no +name conflict; separately, the module needs a data migration from a 4-node HA cluster HAL runs +into the module's single-node server, which [`DATA-CUTOVER.md`](../../../migration/DATA-CUTOVER.md) +already has a rehearsed recipe for (`scripts/rehearse-minio-cutover.sh`, rehearsed 2026-09-20). + +Pulling `quay.io/minio/mc` to run that recipe failed: `unauthorized: access to the requested +resource is not authorized`. Not a `novox`-side problem — DNS resolves quay.io correctly, the +daemon's `insecure-registries` entries are unrelated, and the auth handshake completes. The token +quay.io itself issues is the tell: + +```json +{ + "access": [{"type": "repository", "name": "minio/mc", "actions": []}], + "context": {"com.apostille.roots": {"minio/mc": "$disabled"}, "com.apostille.root": "$disabled"} +} +``` + +Checked the module's own pinned server image too, not just the tool used to test it — +`modules/minio/module.json`'s `quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e` +gets the identical answer: token issues, zero actions, `$disabled`. Anonymous pull of **both** +images `minio` publishes on quay.io is disabled, upstream, sometime after the 2026-09-20 rehearsal +— four days is the outside bound on when this broke. + +A second, already-known gap sits under this one and is not this issue: the module's `runtime` +resource still carries a placeholder image +(`mesh-runtime-minio@sha256:0000000000000000000000000000000000000000000000000000000000000000`), +tracked since `MIGRATION-READINESS-2026-09-21.md` as deferred under +[issue 064](../064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md). + +## Why it matters beyond this instance + +[Issue 064](../064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md) answered +*can the mesh's build environment reach a declared vendor image* — a network-policy question, +resolved by requiring the image be declared as a build input. It assumed the image, once declared, +stays fetchable. This is the case that assumption doesn't cover: the vendor disabled the image +itself, for everyone, regardless of network policy or declaration. A pinned digest does not protect +against this — the digest still resolves, the registry just refuses to serve it anonymously. + +Any module pinning a third-party image by digest carries this risk silently until the day something +needs to pull it fresh — a rebuild, a new node, a cache eviction. Nothing in the catalogue today +distinguishes "pinned and known-good" from "pinned and quietly unfetchable." + +## Open questions + +- Is there an authenticated path to these images (a quay.io account, a mirror), or does `minio` + need to move to a different distribution channel entirely? HAL's own running cluster is on an + older release (`RELEASE.2022-01-07T01-53-23Z`, present in `novox`'s local image cache) — worth + checking whether that tag or a similar one is still fetchable from anywhere before assuming the + module must repin to whatever's locally cached by accident. +- Should the catalogue periodically re-verify that a pinned vendor image is still fetchable, rather + than discovering it at the moment a cutover needs it? +- Same question issue 109 asked about addresses, asked of images: is a locally-cached copy on one + node a fact the mesh can rely on, or a trap for the next node that doesn't have it? + +Handed off for a separate session to work; not blocking the rest of the `novox` module queue.