jschoubben
17fb7c26d2
Found checking minio before assigning it on novox. Not a novox-side problem - DNS, daemon.json and the auth handshake are all fine; quay.io's own token for both quay.io/minio/mc and quay.io/minio/minio (the module's own pinned server image) comes back with actions:[] and $disabled. Worked four days ago at the 2026-09-20 rehearsal. Distinct from issue 064: that one was about the mesh's build environment reaching a declared vendor image. This is the vendor disabling the image for everyone, which a declared, pinned digest does nothing to protect against. Checks pass.
3.8 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | |
|---|---|---|---|---|---|
| located | 2026-09-24 |
|
114 — quay.io disabled anonymous pulls for minio's own images, breaking the module's pin
What was observed
On the control-node, 2026-09-24, checking minio before assigning it — the next module in the
novox queue after the forge and analytics. scripts/compare-module-with-running.py reported no
name conflict; separately, the module needs a data migration from a 4-node HA cluster HAL runs
into the module's single-node server, which DATA-CUTOVER.md
already has a rehearsed recipe for (scripts/rehearse-minio-cutover.sh, rehearsed 2026-09-20).
Pulling quay.io/minio/mc to run that recipe failed: unauthorized: access to the requested resource is not authorized. Not a novox-side problem — DNS resolves quay.io correctly, the
daemon's insecure-registries entries are unrelated, and the auth handshake completes. The token
quay.io itself issues is the tell:
{
"access": [{"type": "repository", "name": "minio/mc", "actions": []}],
"context": {"com.apostille.roots": {"minio/mc": "$disabled"}, "com.apostille.root": "$disabled"}
}
Checked the module's own pinned server image too, not just the tool used to test it —
modules/minio/module.json's quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
gets the identical answer: token issues, zero actions, $disabled. Anonymous pull of both
images minio publishes on quay.io is disabled, upstream, sometime after the 2026-09-20 rehearsal
— four days is the outside bound on when this broke.
A second, already-known gap sits under this one and is not this issue: the module's runtime
resource still carries a placeholder image
(mesh-runtime-minio@sha256:0000000000000000000000000000000000000000000000000000000000000000),
tracked since MIGRATION-READINESS-2026-09-21.md as deferred under
issue 064.
Why it matters beyond this instance
Issue 064 answered can the mesh's build environment reach a declared vendor image — a network-policy question, resolved by requiring the image be declared as a build input. It assumed the image, once declared, stays fetchable. This is the case that assumption doesn't cover: the vendor disabled the image itself, for everyone, regardless of network policy or declaration. A pinned digest does not protect against this — the digest still resolves, the registry just refuses to serve it anonymously.
Any module pinning a third-party image by digest carries this risk silently until the day something needs to pull it fresh — a rebuild, a new node, a cache eviction. Nothing in the catalogue today distinguishes "pinned and known-good" from "pinned and quietly unfetchable."
Open questions
- Is there an authenticated path to these images (a quay.io account, a mirror), or does
minioneed to move to a different distribution channel entirely? HAL's own running cluster is on an older release (RELEASE.2022-01-07T01-53-23Z, present innovox's local image cache) — worth checking whether that tag or a similar one is still fetchable from anywhere before assuming the module must repin to whatever's locally cached by accident. - Should the catalogue periodically re-verify that a pinned vendor image is still fetchable, rather than discovering it at the moment a cutover needs it?
- Same question issue 109 asked about addresses, asked of images: is a locally-cached copy on one node a fact the mesh can rely on, or a trap for the next node that doesn't have it?
Handed off for a separate session to work; not blocking the rest of the novox module queue.