Issue 113 and research 015: the object store's images are gone upstream, not access-restricted

The symptom arrived diagnosed as "the registry disabled anonymous pulls for the
whole vendor namespace". It did not hold: sibling repositories in that namespace
pull normally, the "$disabled" token field appears on every repository including
working ones and describes signing rather than access, and "actions": [] with a
401 is byte-identical to what an invented repository name returns. Both registries'
own APIs establish deletion instead.

Recorded because the correction is the expensive part to rediscover, and because
the instance was harmless while the standing condition is not: no node that does
not already hold the images can ever provision the module again, and nothing
detects that until one tries.

Research 015 scopes the replacement. It is not a redesign — the foundation design
already commits to S3 the protocol rather than the product, and the object store
is an ordinary module, so this instantiates an existing principle. The live OIDC
wiring is the requirement that gates the choice, and it is checked first.

01-RESEARCH/015-the-object-store-after-minio/00-overview.md

@@ -0,0 +1,119 @@
+---
+status: active
+initiated: 2026-09-24
+touches:
+  - 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md
+  - 02-DECISIONS/0033-the-substrate-is-a-store-and-a-broker.md
+  - 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
+  - 02-DECISIONS/0049-a-consumers-identity-fits-the-tightest-backend.md
+  - 02-DECISIONS/0078-the-store-and-broker-are-modules.md
+  - 02-DECISIONS/0084-which-provider-serves-a-consumer.md
+  - 03-DESIGN/00-as-is/03-provisioning.md
+  - 03-DESIGN/01-to-be/07-the-foundation.md
+  - 04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md
+---
+
+# 015 — The object store after MinIO: which S3 implementation, and how the data moves
+
+**The question.** The mesh's object store is MinIO. Its community edition is archived upstream,
+its server and client images have been deleted from every public registry, and the pinned release
+is four and a half years old and will never be patched
+([issue 113](../../04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md)).
+Which S3-compatible implementation replaces it, and what is the migration track for the data and
+the provisioning model that sit on top of it?
+
+**Why now, and why not sooner.** Nothing is on fire: nodes that already hold the images keep
+running, and issue 113 establishes that the deploy path tolerates an unfetchable-but-present
+image by design. The forcing function is not an outage but a one-way door — **no node that does
+not already hold the images can ever provision the module again**, so the mesh's ability to stand
+a node up from its declarations is already broken for this module, and silently.
+
+**The direction is not a departure from the design; it is the design.** The foundation document
+already states the commitment:
+
+> The dependency is on the **protocol**, not the product: AMQP for the bus, S3 for the object
+> store, the OCI protocol for the registry. That is what keeps the naming safe rather than a
+> commitment that cannot be revisited.
+
+The object store is also **not** a foundation service — ADR 0028 removed it, and it is an
+ordinary module required through the module graph by whatever wants one. (The "exception that is
+not a swap" in that passage is the relational store, whose provisioning model borrows PostgreSQL's
+own meaning of databases, roles and schemas. The object store carries no such coupling: a bucket
+is a bucket.) So this effort is an instantiation of an existing principle, not a redesign — which
+is the cheapest kind of decision to make and the strongest kind to cite.
+
+## What the replacement has to carry, measured
+
+Taken from the module's manifest, its composition, its tool surface, and a search for its
+consumers across the catalogue — not from assumption.
+
+| Requirement | Evidence in the module today |
+|---|---|
+| S3 API | The protocol every consumer speaks; already the design's stated dependency. |
+| **OIDC login against the mesh's identity provider** | Six configuration variables are wired and populated in practice — discovery URL, client id, client secret, scopes, display name, redirect — plus a dedicated entrypoint script that blocks startup until the provider answers. This is live, not aspirational. |
+| Erasure-coded multi-node topology | Four server nodes with two data directories each, behind a load balancer. |
+| A single-node form | Declared as a flavour, for development and small nodes. |
+| Buckets as a typed provision | The module declares a provision type of `bucket` on a named network; the mesh mints the credential and the provider creates it (ADRs 0048, 0084). |
+| A tool surface | Bucket create/list/delete, object list/info/delete, presigned URL, and provisioning. |
+| A console | Published on its own subdomain through the reverse proxy, with an unlimited request-body middleware for uploads. |
+
+**Consumers, counted:** one application module, one capture module that takes a private bucket per
+node, one workflow module's tools, and the delivery/rescue internals of the shared library. The
+surface is small — the cost is concentrated in the provisioning handler, the tool handlers and the
+OIDC story, not spread across the catalogue.
+
+## Candidates
+
+Scoped to **SeaweedFS** as the primary, with the others recorded so the rejection is not
+rediscovered.
+
+- **SeaweedFS** — Apache-2.0, Go, twelve-plus years of development, erasure coding, and OIDC
+  support in its S3/STS layer. Chosen to scope because it is the only candidate that plausibly
+  preserves the OIDC requirement above, which is the one requirement that is live and least
+  substitutable.
+- **Garage** — the lightest to operate and the simplest model, but **no native identity-provider
+  integration**. Adopting it means losing OIDC console login or fronting it with a proxy. A real
+  functional regression against something currently in use.
+- **RustFS** — markets itself as a binary-level drop-in retaining existing data, buckets and
+  configuration, which would make the data migration close to trivial. Young, and that claim is
+  exactly the kind that must be verified on a copy before it is believed.
+- **Ceph RGW** — the most capable and the most operationally expensive; disproportionate to a mesh
+  where the object store is an ordinary module, not a platform.
+
+**The first thing to verify, because the choice turns on it:** how much of SeaweedFS's OIDC story
+is in the freely licensed build, and whether its shape — IAM/STS token exchange — can actually
+stand in for a console that redirects a human to an identity provider. If it cannot, the honest
+finding may be that **no** candidate preserves the current feature set, and the decision becomes
+which regression to accept. That question is worth answering before any migration work starts.
+
+## The migration track, in outline
+
+Data movement is the easy half, and deliberately reversible.
+
+1. **Stand the replacement up beside the incumbent**, on its own ports and its own provision type.
+   No downtime, nothing removed.
+2. **Copy bucket by bucket with a neutral tool.** `rclone` rather than the incumbent's own client
+   — the client has been withdrawn upstream too, so building the migration on it would inherit
+   the same dependency this effort exists to remove.
+3. **Verify per bucket** — object counts and checksums, not a transfer exit code.
+4. **Repoint consumers through the connection the module already publishes.** Consumers read an
+   API URL from the module's declared connections rather than addressing the store directly, so
+   the cutover surface is that value plus the provisioning and tool handlers.
+5. **Freeze writes, final incremental sync, flip**, and keep the incumbent read-only as the
+   rollback until confidence is earned.
+6. **Retire**, and only then remove the module.
+
+The genuinely new work is not the copy. It is the **provisioning handler** and the **tool
+handlers**, which are written against MinIO's admin API, and the OIDC wiring.
+
+## Open questions
+
+- How much of the OIDC requirement survives, and in which build? See above — this gates the
+  choice.
+- Does the mesh's bucket provision translate to the candidate's identity model without weakening
+  what ADR 0049 says about a consumer's identity fitting the tightest backend?
+- Should this effort also answer issue 113's general question — mirroring third-party images into
+  the mesh's own registry — or is that a separate decision? Replacing one withdrawn product with
+  another unmirrored upstream leaves the same one-way door in place, just further from the hinge.
+- Is the four-node erasure-coded topology still warranted, or was it inherited? Worth re-asking
+  while the product is being chosen, rather than reproducing a shape by default.