Research 015: reopen the comparison — the premise for narrowing to one candidate was false

SeaweedFS was scoped as primary because it looked like the only candidate preserving OIDC console login. Measured: its admin UI is Apache-2.0 but its identity-provider integration is not — console SSO sits behind the per-TB commercial licence, alongside point-in-time recovery and automatic EC repair. The free build gives OIDC on the S3 API via STS and a console authenticated by local username and password. So the answer to the gating question is that no candidate preserves the current feature set for free, which this effort had written down as a possible outcome. Reopened across three candidates with the requirement-by-requirement evidence in 01. Two corrections to what the overview recorded. RustFS is not a binary-level drop-in retaining existing data: API and on-disk compatibility are separate paths and the on-disk one is preview-scoped. And it carries an open defect in the credential path the bucket provision depends on, which gates it specifically. Nothing graduates before two measurements named in 01: whether an authenticating proxy is an acceptable answer to console SSO, and which S3 endpoints consumers actually call — the latter because Garage does not implement the full span and cannot be ranked until that is counted.
2026-09-24 16:10:33 +02:00
parent a458f751c6
commit c8cbbcfb8a
2 changed files with 132 additions and 19 deletions
@@ -64,27 +64,33 @@ OIDC story, not spread across the catalogue.
 ## Candidates
-Scoped to **SeaweedFS** as the primary, with the others recorded so the rejection is not
+**The comparison is open across three candidates.** It was briefly narrowed to SeaweedFS; that
-rediscovered.
+narrowing did not survive measurement and was reopened on 2026-09-24. The evidence, the full
 requirement-by-requirement table and what each option costs are in
 [01 — the candidates measured](01-candidate-comparison.md).
- **SeaweedFS** — Apache-2.0, Go, twelve-plus years of development, erasure coding, and OIDC
+In short, and only in short:
  support in its S3/STS layer. Chosen to scope because it is the only candidate that plausibly
  preserves the OIDC requirement above, which is the one requirement that is live and least
  substitutable.
 - **Garage** — the lightest to operate and the simplest model, but **no native identity-provider
  integration**. Adopting it means losing OIDC console login or fronting it with a proxy. A real
  functional regression against something currently in use.
 - **RustFS** — markets itself as a binary-level drop-in retaining existing data, buckets and
  configuration, which would make the data migration close to trivial. Young, and that claim is
  exactly the kind that must be verified on a copy before it is believed.
 - **Ceph RGW** — the most capable and the most operationally expensive; disproportionate to a mesh
  where the object store is an ordinary module, not a platform.
-**The first thing to verify, because the choice turns on it:** how much of SeaweedFS's OIDC story
+- **SeaweedFS** — Apache-2.0, the longest field record, erasure coding. Its **console OIDC is an
-is in the freely licensed build, and whether its shape — IAM/STS token exchange — can actually
+  Enterprise feature**; the free build authenticates the console with a local username and
-stand in for a console that redirects a human to an identity provider. If it cannot, the honest
+  password. This is what falsified the original narrowing.
-finding may be that **no** candidate preserves the current feature set, and the decision becomes
+- **Garage** — the best match for how the mesh provisions, with a first-class admin REST API
-which regression to accept. That question is worth answering before any migration work starts.
+  scoped to exactly bucket and key creation. Costs: replication rather than erasure coding, no
  full S3 endpoint span, and neither console nor identity integration.
 - **RustFS** — the only candidate preserving the live behaviour, with a MinIO-shaped console and
  documented OIDC against Keycloak, under Apache-2.0. Costs: it reached GA eight days before this
  was written, and an open defect is reported in the credential path the mesh's bucket provision
  depends on.
 - **Ceph RGW** — remains rejected as disproportionate where the object store is an ordinary
  module rather than a platform.
 **The question that decided the original narrowing has been answered** — SeaweedFS's console OIDC
 is not in the free build — and the answer was *"no candidate preserves the current feature set
 for free"*, exactly the outcome this effort said was possible. The decision is therefore not
 which product is best in the abstract but **which cost is acceptable**, and two measurements
 gate it: whether an authenticating proxy is an acceptable answer to console single sign-on, and
 which S3 endpoints consumers actually call. Both are named in
 [01](01-candidate-comparison.md#what-is-still-unmeasured). Nothing graduates before them.
 ## The migration track, in outline
@@ -0,0 +1,107 @@
 # 015 / 01 — The candidates measured, and why the first pick did not survive
 *Dated 2026-09-24. This document reopens the comparison the overview had narrowed to one
 candidate. It exists because the requirement the narrowing rested on turned out not to be met.*
 ## What was asked, and what came back
 The overview named one question as deciding the choice: **how much of SeaweedFS's OIDC story is
 in the freely licensed build, and can its shape stand in for a console that redirects a human to
 an identity provider.** SeaweedFS was scoped as primary *because it was believed to be the only
 candidate that preserved that*. The answer is no, and the premise fails with it.
 | SeaweedFS capability | Apache-2.0 build | Enterprise (proprietary, per-TB) |
 |---|---|---|
 | OIDC on the S3 API, via STS/JWT token exchange | **yes** | yes |
 | Admin UI sign-in | **username and password only** | **OIDC SSO** (Keycloak and others) |
 The admin UI is Apache-2.0 and real; its **identity provider integration is not**. It sits behind
 the commercial licence together with point-in-time recovery, automatic erasure-code repair,
 native multi-tenancy and S3 QoS limiting.
 So the free build gives OIDC for *programmatic* access and a locally-authenticated console. The
 behaviour in use today — a human redirected to the mesh's identity provider to reach the object
 store's console — is not preserved. The narrowing was sound reasoning on wrong information.
 ## The three-way comparison, as measured
 Against the requirement set in the overview, which was taken from the module rather than assumed.
 | | SeaweedFS | Garage | RustFS |
 |---|---|---|---|
 | Licence | Apache-2.0 | AGPL-3.0 | Apache-2.0 |
 | Maturity | twelve-plus years | several years, stable | **GA 2026-09-16** |
 | Redundancy | erasure coding | **replication only** — a replica count, commonly 3, so ~3× raw per byte | erasure coding |
 | S3 API | broad | **explicitly not the full span of endpoints** | broad, MinIO-shaped |
 | Console | `weed admin`, in the free build | **none official** (a third-party UI exists) | yes, modelled on MinIO's |
 | Console OIDC | Enterprise only | **none** | **yes** — documented against Keycloak |
 | Programmatic bucket + credential creation | `weed shell` / IAM API | **full admin REST API**, scoped tokens for `CreateBucket` and `CreateKey` | admin API under its own `v3` namespace; client-compatible with MinIO's |
 | Permission model | IAM users, groups, policies | **its own**: per access key, per bucket, read/write/owner — no AWS-style ACLs or bucket policies | MinIO-shaped IAM: users, groups, policies, service accounts |
 **No candidate is free of cost.** That is the finding, and it is why the comparison is reopened
 rather than resolved here.
 ## What each one actually costs
 **SeaweedFS** — the safe engineering choice. Longest field record, erasure coding, permissive
 licence. The cost is the console regression: either accept local credentials for the console, pay
 per TB, or front it with an authenticating proxy (see below).
 **Garage** — the best fit for how the mesh *provisions*. Its admin API is a first-class REST
 surface with scoped tokens for exactly the two operations the bucket provision performs, which is
 a better match than any other candidate. Three costs, and they are not small: redundancy is
 replication, so the storage bill is a multiple rather than erasure coding's overhead; it does not
 implement the full S3 endpoint span, which has to be checked against what consumers actually call;
 and there is neither a console nor identity integration, so the console requirement is not
 regressed but *removed*.
 **RustFS** — the only candidate that preserves the live behaviour. Apache-2.0, erasure coding, a
 MinIO-shaped console with **documented OIDC against Keycloak**, and an admin API close enough to
 MinIO's that the existing tool handlers are the least work to port. Against that:
 - **It reached GA on 2026-09-16 — eight days before this was written.** For a component that
  holds data, field record is a feature, and it does not have one yet.
 - **The overview's characterisation of it was wrong and is corrected here.** It was recorded as a
  binary-level drop-in retaining existing data, which would make migration trivial. In fact API
  compatibility and on-disk compatibility are *separate* paths, and the on-disk one is
  preview-scoped with documented encryption limitations. It should not be relied on. This costs
  nothing in practice — the migration track already moves data over the S3 API, not on disk — but
  the claim should not survive into a decision record.
 - **There is an open defect in precisely the area the mesh depends on.** An access key created by
  an OIDC/SSO user is reported denied on all S3 operations, the embedded policy being ignored.
  The mesh mints credentials for every provisioned bucket. Whether this touches the mesh's path —
  which provisions with administrative credentials rather than as an SSO user — is **unverified,
  and must be tested before RustFS is chosen**, not after.
 **Ceph RGW** stays rejected, for the reason already recorded: disproportionate where the object
 store is an ordinary module rather than a platform.
 ## The option that changes the trade
 The console regression is not necessarily the product's to solve. An authenticating proxy in
 front of whatever console exists — against the identity provider the mesh already runs, behind
 the reverse proxy it already runs — restores single sign-on at the proxy layer for **SeaweedFS or
 Garage**, without a commercial licence and without betting on the youngest candidate.
 If that holds, the decision stops being *which regression to accept* and becomes a straight
 comparison of storage properties, where SeaweedFS's field record and erasure coding are the
 strongest position. **It is unverified.** It is the cheapest next measurement available and it
 should be made before the record is written.
 ## What is still unmeasured
 Named so the effort is not closed while pretending otherwise.
 1. Whether an authenticating proxy in front of a console is acceptable as the mesh's answer to
   console SSO — a design question as much as a technical one, since it moves identity out of the
   product and into the edge for this module.
 2. Which S3 endpoints the consumers actually call, checked against Garage's supported span rather
   than assumed. Until that is counted, Garage cannot be fairly ranked.
 3. Whether RustFS's OIDC-credential defect touches the credential path the bucket provision uses.
 4. What the redundancy change costs in real terms — replication against erasure coding at the
   sizes actually stored, rather than as a ratio in the abstract.
 5. Whether the four-node erasure-coded topology is still warranted at all, which the overview
   already raised and nothing here answers.
 **No candidate should be graduated to a decision until 1 and 2 are measured.** 3 gates RustFS
 specifically. The effort stays open.