01-RESEARCH/015-the-object-store-after-minio/00-overview.md

@@ -64,27 +64,33 @@ OIDC story, not spread across the catalogue.
 
 ## Candidates
 
-Scoped to **SeaweedFS** as the primary, with the others recorded so the rejection is not
-rediscovered.
+**The comparison is open across three candidates.** It was briefly narrowed to SeaweedFS; that
+narrowing did not survive measurement and was reopened on 2026-09-24. The evidence, the full
+requirement-by-requirement table and what each option costs are in
+[01 — the candidates measured](01-candidate-comparison.md).
 
-- **SeaweedFS** — Apache-2.0, Go, twelve-plus years of development, erasure coding, and OIDC
-  support in its S3/STS layer. Chosen to scope because it is the only candidate that plausibly
-  preserves the OIDC requirement above, which is the one requirement that is live and least
-  substitutable.
-- **Garage** — the lightest to operate and the simplest model, but **no native identity-provider
-  integration**. Adopting it means losing OIDC console login or fronting it with a proxy. A real
-  functional regression against something currently in use.
-- **RustFS** — markets itself as a binary-level drop-in retaining existing data, buckets and
-  configuration, which would make the data migration close to trivial. Young, and that claim is
-  exactly the kind that must be verified on a copy before it is believed.
-- **Ceph RGW** — the most capable and the most operationally expensive; disproportionate to a mesh
-  where the object store is an ordinary module, not a platform.
+In short, and only in short:
 
-**The first thing to verify, because the choice turns on it:** how much of SeaweedFS's OIDC story
-is in the freely licensed build, and whether its shape — IAM/STS token exchange — can actually
-stand in for a console that redirects a human to an identity provider. If it cannot, the honest
-finding may be that **no** candidate preserves the current feature set, and the decision becomes
-which regression to accept. That question is worth answering before any migration work starts.
+- **SeaweedFS** — Apache-2.0, the longest field record, erasure coding. Its **console OIDC is an
+  Enterprise feature**; the free build authenticates the console with a local username and
+  password. This is what falsified the original narrowing.
+- **Garage** — the best match for how the mesh provisions, with a first-class admin REST API
+  scoped to exactly bucket and key creation. Costs: replication rather than erasure coding, no
+  full S3 endpoint span, and neither console nor identity integration.
+- **RustFS** — the only candidate preserving the live behaviour, with a MinIO-shaped console and
+  documented OIDC against Keycloak, under Apache-2.0. Costs: it reached GA eight days before this
+  was written, and an open defect is reported in the credential path the mesh's bucket provision
+  depends on.
+- **Ceph RGW** — remains rejected as disproportionate where the object store is an ordinary
+  module rather than a platform.
+
+**The question that decided the original narrowing has been answered** — SeaweedFS's console OIDC
+is not in the free build — and the answer was *"no candidate preserves the current feature set
+for free"*, exactly the outcome this effort said was possible. The decision is therefore not
+which product is best in the abstract but **which cost is acceptable**, and two measurements
+gate it: whether an authenticating proxy is an acceptable answer to console single sign-on, and
+which S3 endpoints consumers actually call. Both are named in
+[01](01-candidate-comparison.md#what-is-still-unmeasured). Nothing graduates before them.
 
 ## The migration track, in outline
 

01-RESEARCH/015-the-object-store-after-minio/01-candidate-comparison.md

@@ -0,0 +1,107 @@
+# 015 / 01 — The candidates measured, and why the first pick did not survive
+
+*Dated 2026-09-24. This document reopens the comparison the overview had narrowed to one
+candidate. It exists because the requirement the narrowing rested on turned out not to be met.*
+
+## What was asked, and what came back
+
+The overview named one question as deciding the choice: **how much of SeaweedFS's OIDC story is
+in the freely licensed build, and can its shape stand in for a console that redirects a human to
+an identity provider.** SeaweedFS was scoped as primary *because it was believed to be the only
+candidate that preserved that*. The answer is no, and the premise fails with it.
+
+| SeaweedFS capability | Apache-2.0 build | Enterprise (proprietary, per-TB) |
+|---|---|---|
+| OIDC on the S3 API, via STS/JWT token exchange | **yes** | yes |
+| Admin UI sign-in | **username and password only** | **OIDC SSO** (Keycloak and others) |
+
+The admin UI is Apache-2.0 and real; its **identity provider integration is not**. It sits behind
+the commercial licence together with point-in-time recovery, automatic erasure-code repair,
+native multi-tenancy and S3 QoS limiting.
+
+So the free build gives OIDC for *programmatic* access and a locally-authenticated console. The
+behaviour in use today — a human redirected to the mesh's identity provider to reach the object
+store's console — is not preserved. The narrowing was sound reasoning on wrong information.
+
+## The three-way comparison, as measured
+
+Against the requirement set in the overview, which was taken from the module rather than assumed.
+
+| | SeaweedFS | Garage | RustFS |
+|---|---|---|---|
+| Licence | Apache-2.0 | AGPL-3.0 | Apache-2.0 |
+| Maturity | twelve-plus years | several years, stable | **GA 2026-09-16** |
+| Redundancy | erasure coding | **replication only** — a replica count, commonly 3, so ~3× raw per byte | erasure coding |
+| S3 API | broad | **explicitly not the full span of endpoints** | broad, MinIO-shaped |
+| Console | `weed admin`, in the free build | **none official** (a third-party UI exists) | yes, modelled on MinIO's |
+| Console OIDC | Enterprise only | **none** | **yes** — documented against Keycloak |
+| Programmatic bucket + credential creation | `weed shell` / IAM API | **full admin REST API**, scoped tokens for `CreateBucket` and `CreateKey` | admin API under its own `v3` namespace; client-compatible with MinIO's |
+| Permission model | IAM users, groups, policies | **its own**: per access key, per bucket, read/write/owner — no AWS-style ACLs or bucket policies | MinIO-shaped IAM: users, groups, policies, service accounts |
+
+**No candidate is free of cost.** That is the finding, and it is why the comparison is reopened
+rather than resolved here.
+
+## What each one actually costs
+
+**SeaweedFS** — the safe engineering choice. Longest field record, erasure coding, permissive
+licence. The cost is the console regression: either accept local credentials for the console, pay
+per TB, or front it with an authenticating proxy (see below).
+
+**Garage** — the best fit for how the mesh *provisions*. Its admin API is a first-class REST
+surface with scoped tokens for exactly the two operations the bucket provision performs, which is
+a better match than any other candidate. Three costs, and they are not small: redundancy is
+replication, so the storage bill is a multiple rather than erasure coding's overhead; it does not
+implement the full S3 endpoint span, which has to be checked against what consumers actually call;
+and there is neither a console nor identity integration, so the console requirement is not
+regressed but *removed*.
+
+**RustFS** — the only candidate that preserves the live behaviour. Apache-2.0, erasure coding, a
+MinIO-shaped console with **documented OIDC against Keycloak**, and an admin API close enough to
+MinIO's that the existing tool handlers are the least work to port. Against that:
+
+- **It reached GA on 2026-09-16 — eight days before this was written.** For a component that
+  holds data, field record is a feature, and it does not have one yet.
+- **The overview's characterisation of it was wrong and is corrected here.** It was recorded as a
+  binary-level drop-in retaining existing data, which would make migration trivial. In fact API
+  compatibility and on-disk compatibility are *separate* paths, and the on-disk one is
+  preview-scoped with documented encryption limitations. It should not be relied on. This costs
+  nothing in practice — the migration track already moves data over the S3 API, not on disk — but
+  the claim should not survive into a decision record.
+- **There is an open defect in precisely the area the mesh depends on.** An access key created by
+  an OIDC/SSO user is reported denied on all S3 operations, the embedded policy being ignored.
+  The mesh mints credentials for every provisioned bucket. Whether this touches the mesh's path —
+  which provisions with administrative credentials rather than as an SSO user — is **unverified,
+  and must be tested before RustFS is chosen**, not after.
+
+**Ceph RGW** stays rejected, for the reason already recorded: disproportionate where the object
+store is an ordinary module rather than a platform.
+
+## The option that changes the trade
+
+The console regression is not necessarily the product's to solve. An authenticating proxy in
+front of whatever console exists — against the identity provider the mesh already runs, behind
+the reverse proxy it already runs — restores single sign-on at the proxy layer for **SeaweedFS or
+Garage**, without a commercial licence and without betting on the youngest candidate.
+
+If that holds, the decision stops being *which regression to accept* and becomes a straight
+comparison of storage properties, where SeaweedFS's field record and erasure coding are the
+strongest position. **It is unverified.** It is the cheapest next measurement available and it
+should be made before the record is written.
+
+## What is still unmeasured
+
+Named so the effort is not closed while pretending otherwise.
+
+1. Whether an authenticating proxy in front of a console is acceptable as the mesh's answer to
+   console SSO — a design question as much as a technical one, since it moves identity out of the
+   product and into the edge for this module.
+2. Which S3 endpoints the consumers actually call, checked against Garage's supported span rather
+   than assumed. Until that is counted, Garage cannot be fairly ranked.
+3. Whether RustFS's OIDC-credential defect touches the credential path the bucket provision uses.
+4. What the redundancy change costs in real terms — replication against erasure coding at the
+   sizes actually stored, rather than as a ratio in the abstract.
+5. Whether the four-node erasure-coded topology is still warranted at all, which the overview
+   already raised and nothing here answers.
+
+**No candidate should be graduated to a decision until 1 and 2 are measured.** 3 gates RustFS
+specifically. The effort stays open.