Issue 113 and research 015: the object store's images are gone upstream, not access-restricted #103

2026-09-24T13:27:31Z

jschoubben commented

2026-09-24 13:27:31 +00:00

Opens issue 113 (status: located, with diagnosis) and research 015 for the replacement.

What this corrects

The symptom arrived already diagnosed as "quay.io has disabled anonymous pulls for the entire minio org". That did not survive checking:

Evidence cited	What it actually is
`"actions": []` on the pull token	Byte-identical to what a deliberately invented repository name in the same namespace returns. Cannot distinguish revoked from absent.
`"$disabled"` in the token	Present on every repository on that registry, including all the ones pulling successfully. It describes image signing, not access.
"the entire org"	Two repositories fail. The operator, console and sidecar proxy in the same namespace return `200` with `actions: ['pull']`.

The registries' own APIs establish the real cause — deletion, not restriction. The primary registry answers 404 for the server repository against 200 for a sibling; the secondary registry's namespace listing has 68 public repositories and the server and client are simply absent. Upstream withdrew them on 2026-09-11, after archiving the OSS repository in February 2026. No credential can answer this — there is nothing left to authenticate against.

The mesh was not blocked

Also checked rather than assumed: the module deploys fine on a node that already holds the images, and that is by design. The deploy stage pulls best-effort then asserts only that every declared image resolves locally — with a code comment naming this exact precedent. Dropping the module from the cutover queue was not necessary.

Four further specific claims in the original report could not be substantiated at all (a module.json that does not exist, a digest pin that does not exist, an all-zero placeholder digest with zero occurrences tree-wide, and a readiness document that is not on disk). They are tabulated in the diagnosis because acting on them would have wasted time. None of them changes the real finding.

Why it still matters

The instance is harmless; the standing condition is not. No node that does not already hold the images can ever provision the module again — so the claim that a node can be rebuilt from its declarations is already false here, and silently. The pinned release is 4.5 years old and permanently unpatched. Nothing detects this class of failure, because the guard that correctly protects existing nodes also reduces the trace to a warning.

Research 015

Scopes SeaweedFS as primary, with Garage, RustFS and Ceph RGW recorded so the rejections are not rediscovered.

This is not a redesign. 03-DESIGN/01-to-be/07-the-foundation.md already states the dependency is on "the protocol, not the product: … S3 for the object store", and ADR 0028 makes the object store an ordinary module rather than a foundation service. The effort instantiates an existing principle.

The requirement that gates the choice is the live OIDC wiring (six populated variables plus a blocking entrypoint). Garage has no native IdP integration, so it is a real regression; how much of SeaweedFS's OIDC story is in the freely licensed build, and whether IAM/STS can stand in for a console redirect, is stated as the first thing to verify — with the honest possibility that no candidate preserves the current feature set.

Migration outline is side-by-side, rclone rather than the withdrawn vendor client, verify per bucket, flip the published connection URL, incumbent read-only as rollback.

Checks

cycle:   263 documents checked, the chain holds
records: 96 decision records checked, all checks passed
index:   current

Scanned clean for node names, domain names, absolute paths, hosts and IPs.

Left deliberately open rather than answered: whether the mesh should mirror third-party images into its own registry. That is the fix that generalises, and it is a separate decision.

Opens **issue 113** (`status: located`, with diagnosis) and **research 015** for the replacement. ## What this corrects The symptom arrived already diagnosed as *"quay.io has disabled anonymous pulls for the entire minio org"*. That did not survive checking: | Evidence cited | What it actually is | |---|---| | `"actions": []` on the pull token | Byte-identical to what a **deliberately invented** repository name in the same namespace returns. Cannot distinguish *revoked* from *absent*. | | `"$disabled"` in the token | Present on **every** repository on that registry, including all the ones pulling successfully. It describes image **signing**, not access. | | "the entire org" | Two repositories fail. The operator, console and sidecar proxy in the same namespace return `200` with `actions: ['pull']`. | The registries' own APIs establish the real cause — **deletion, not restriction**. The primary registry answers `404` for the server repository against `200` for a sibling; the secondary registry's namespace listing has 68 public repositories and the server and client are simply absent. Upstream withdrew them on 2026-09-11, after archiving the OSS repository in February 2026. **No credential can answer this** — there is nothing left to authenticate against. ## The mesh was not blocked Also checked rather than assumed: the module deploys fine on a node that already holds the images, and that is by design. The deploy stage pulls best-effort then asserts only that every declared image **resolves locally** — with a code comment naming this exact precedent. Dropping the module from the cutover queue was not necessary. Four further specific claims in the original report could not be substantiated at all (a `module.json` that does not exist, a digest pin that does not exist, an all-zero placeholder digest with zero occurrences tree-wide, and a readiness document that is not on disk). They are tabulated in the diagnosis because acting on them would have wasted time. None of them changes the real finding. ## Why it still matters The instance is harmless; the standing condition is not. **No node that does not already hold the images can ever provision the module again** — so the claim that a node can be rebuilt from its declarations is already false here, and silently. The pinned release is 4.5 years old and permanently unpatched. Nothing detects this class of failure, because the guard that correctly protects existing nodes also reduces the trace to a warning. ## Research 015 Scopes **SeaweedFS** as primary, with Garage, RustFS and Ceph RGW recorded so the rejections are not rediscovered. This is **not a redesign**. `03-DESIGN/01-to-be/07-the-foundation.md` already states the dependency is on *"the protocol, not the product: … S3 for the object store"*, and ADR 0028 makes the object store an ordinary module rather than a foundation service. The effort instantiates an existing principle. The requirement that gates the choice is the **live OIDC wiring** (six populated variables plus a blocking entrypoint). Garage has no native IdP integration, so it is a real regression; how much of SeaweedFS's OIDC story is in the freely licensed build, and whether IAM/STS can stand in for a console redirect, is stated as the first thing to verify — with the honest possibility that no candidate preserves the current feature set. Migration outline is side-by-side, `rclone` rather than the withdrawn vendor client, verify per bucket, flip the published connection URL, incumbent read-only as rollback. ## Checks ``` cycle: 263 documents checked, the chain holds records: 96 decision records checked, all checks passed index: current ``` Scanned clean for node names, domain names, absolute paths, hosts and IPs. Left deliberately open rather than answered: whether the mesh should mirror third-party images into its own registry. That is the fix that generalises, and it is a separate decision.

jschoubben added 1 commit 2026-09-24 13:27:32 +00:00

Issue 113 and research 015: the object store's images are gone upstream, not access-restricted d497b37e43

The symptom arrived diagnosed as "the registry disabled anonymous pulls for the
whole vendor namespace". It did not hold: sibling repositories in that namespace
pull normally, the "$disabled" token field appears on every repository including
working ones and describes signing rather than access, and "actions": [] with a
401 is byte-identical to what an invented repository name returns. Both registries'
own APIs establish deletion instead.

Recorded because the correction is the expensive part to rediscover, and because
the instance was harmless while the standing condition is not: no node that does
not already hold the images can ever provision the module again, and nothing
detects that until one tries.

Research 015 scopes the replacement. It is not a redesign — the foundation design
already commits to S3 the protocol rather than the product, and the object store
is an ordinary module, so this instantiates an existing principle. The live OIDC
wiring is the requirement that gates the choice, and it is checked first.

jschoubben added 1 commit 2026-09-24 13:44:47 +00:00

Issue 113: ground the rebuildability point in what the design actually says 4beb6629db

Review of my own text found an unattributed claim — "the mesh's claim that a node
can be rebuilt from its declarations" — which is not a stated principle anywhere.
Replaced with the design position that genuinely covers it: to-be 07 chooses
references over payload because "reproducibility comes from pinning the identity of
a thing rather than carrying its bytes". This incident is that choice's failure mode
when the identity stops resolving, which is a sharper point than the one I made.

Scope stated honestly: the passage is about the foundation bundle and this module is
not in it, but pin-identity-fetch-bytes is how every module gets third-party images.

Also names the tension the mirroring question actually carries — mirroring is a move
away from references-over-payload, so it is a decision, not a fix.

jschoubben commented

2026-09-24 13:45:16 +00:00

Review

Verified rather than re-read:

Links — all 3 relative links resolve; touches: all 9 paths exist.
Quotes — both quotations from 03-DESIGN/01-to-be/07-the-foundation.md are character-exact (checked with grep -F).
located-in: [hal modules/minio] — matches the established [repo path] convention ([mesh-host internal/apply], [mesh-lab test/integration]).
ADR 0028 — confirmed it says what I cited: "The object store is not substrate… An object store is an ordinary module, required through the module graph."
Checks — cycle 263 docs / records 96 / index current.
Disclosure — clean for node names, domains, absolute paths, hosts, IPs, registry hostnames.

One real gap found and fixed (`4beb662`)

I had written "the mesh's claim that a node can be rebuilt from its declarations is false for this module." Searching for that principle turned up nothing — I had asserted a claim the design does not make.

What the design does say is better and sharper, at 07-the-foundation.md:156-159: it chooses references over payload because "reproducibility comes from pinning the identity of a thing rather than carrying its bytes." This incident is precisely that choice's failure mode — the reasoning is sound, but it holds only while a pinned identity stays resolvable, and nothing in the mesh's control guarantees that for an image in someone else's registry.

Scope is now stated honestly rather than overclaimed: that passage is about the foundation bundle and this module is not in it, but pin-identity-fetch-bytes is how every module gets its third-party images, so the exposure is general even though the sentence is local.

Same commit also names the tension the mirroring question actually carries — mirroring third-party images is a deliberate move away from references-over-payload, so it should be decided as such rather than smuggled in as a fix.

Merging.

## Review Verified rather than re-read: - **Links** — all 3 relative links resolve; `touches:` all 9 paths exist. - **Quotes** — both quotations from `03-DESIGN/01-to-be/07-the-foundation.md` are character-exact (checked with `grep -F`). - **`located-in: [hal modules/minio]`** — matches the established `[repo path]` convention (`[mesh-host internal/apply]`, `[mesh-lab test/integration]`). - **ADR 0028** — confirmed it says what I cited: *"The object store is not substrate… An object store is an ordinary module, required through the module graph."* - **Checks** — `cycle` 263 docs / `records` 96 / `index` current. - **Disclosure** — clean for node names, domains, absolute paths, hosts, IPs, registry hostnames. ### One real gap found and fixed (4beb662) I had written *"the mesh's claim that a node can be rebuilt from its declarations is false for this module."* Searching for that principle turned up **nothing** — I had asserted a claim the design does not make. What the design does say is better and sharper, at `07-the-foundation.md:156-159`: it chooses **references over payload** because *"reproducibility comes from pinning the identity of a thing rather than carrying its bytes."* This incident is precisely that choice's failure mode — the reasoning is sound, but it holds only while a pinned identity stays **resolvable**, and nothing in the mesh's control guarantees that for an image in someone else's registry. Scope is now stated honestly rather than overclaimed: that passage is about the foundation bundle and this module is not in it, but pin-identity-fetch-bytes is how *every* module gets its third-party images, so the exposure is general even though the sentence is local. Same commit also names the tension the mirroring question actually carries — mirroring third-party images is a deliberate move *away* from references-over-payload, so it should be decided as such rather than smuggled in as a fix. Merging.

jschoubben merged commit f10dce4f9e into main

2026-09-24 13:45:35 +00:00

jschoubben deleted branch storage/113-the-object-store-lost-its-upstream

2026-09-24 13:45:36 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/hq#103