diff --git a/01-RESEARCH/015-the-object-store-after-minio/00-overview.md b/01-RESEARCH/015-the-object-store-after-minio/00-overview.md index fee62e3..6c628fb 100644 --- a/01-RESEARCH/015-the-object-store-after-minio/00-overview.md +++ b/01-RESEARCH/015-the-object-store-after-minio/00-overview.md @@ -64,27 +64,33 @@ OIDC story, not spread across the catalogue. ## Candidates -Scoped to **SeaweedFS** as the primary, with the others recorded so the rejection is not -rediscovered. +**The comparison is open across three candidates.** It was briefly narrowed to SeaweedFS; that +narrowing did not survive measurement and was reopened on 2026-09-24. The evidence, the full +requirement-by-requirement table and what each option costs are in +[01 — the candidates measured](01-candidate-comparison.md). -- **SeaweedFS** — Apache-2.0, Go, twelve-plus years of development, erasure coding, and OIDC - support in its S3/STS layer. Chosen to scope because it is the only candidate that plausibly - preserves the OIDC requirement above, which is the one requirement that is live and least - substitutable. -- **Garage** — the lightest to operate and the simplest model, but **no native identity-provider - integration**. Adopting it means losing OIDC console login or fronting it with a proxy. A real - functional regression against something currently in use. -- **RustFS** — markets itself as a binary-level drop-in retaining existing data, buckets and - configuration, which would make the data migration close to trivial. Young, and that claim is - exactly the kind that must be verified on a copy before it is believed. -- **Ceph RGW** — the most capable and the most operationally expensive; disproportionate to a mesh - where the object store is an ordinary module, not a platform. +In short, and only in short: -**The first thing to verify, because the choice turns on it:** how much of SeaweedFS's OIDC story -is in the freely licensed build, and whether its shape — IAM/STS token exchange — can actually -stand in for a console that redirects a human to an identity provider. If it cannot, the honest -finding may be that **no** candidate preserves the current feature set, and the decision becomes -which regression to accept. That question is worth answering before any migration work starts. +- **SeaweedFS** — Apache-2.0, the longest field record, erasure coding. Its **console OIDC is an + Enterprise feature**; the free build authenticates the console with a local username and + password. This is what falsified the original narrowing. +- **Garage** — the best match for how the mesh provisions, with a first-class admin REST API + scoped to exactly bucket and key creation. Costs: replication rather than erasure coding, no + full S3 endpoint span, and neither console nor identity integration. +- **RustFS** — the only candidate preserving the live behaviour, with a MinIO-shaped console and + documented OIDC against Keycloak, under Apache-2.0. Costs: it reached GA eight days before this + was written, and an open defect is reported in the credential path the mesh's bucket provision + depends on. +- **Ceph RGW** — remains rejected as disproportionate where the object store is an ordinary + module rather than a platform. + +**The question that decided the original narrowing has been answered** — SeaweedFS's console OIDC +is not in the free build — and the answer was *"no candidate preserves the current feature set +for free"*, exactly the outcome this effort said was possible. The decision is therefore not +which product is best in the abstract but **which cost is acceptable**, and two measurements +gate it: whether an authenticating proxy is an acceptable answer to console single sign-on, and +which S3 endpoints consumers actually call. Both are named in +[01](01-candidate-comparison.md#what-is-still-unmeasured). Nothing graduates before them. ## The migration track, in outline diff --git a/01-RESEARCH/015-the-object-store-after-minio/01-candidate-comparison.md b/01-RESEARCH/015-the-object-store-after-minio/01-candidate-comparison.md new file mode 100644 index 0000000..f2e00ab --- /dev/null +++ b/01-RESEARCH/015-the-object-store-after-minio/01-candidate-comparison.md @@ -0,0 +1,107 @@ +# 015 / 01 — The candidates measured, and why the first pick did not survive + +*Dated 2026-09-24. This document reopens the comparison the overview had narrowed to one +candidate. It exists because the requirement the narrowing rested on turned out not to be met.* + +## What was asked, and what came back + +The overview named one question as deciding the choice: **how much of SeaweedFS's OIDC story is +in the freely licensed build, and can its shape stand in for a console that redirects a human to +an identity provider.** SeaweedFS was scoped as primary *because it was believed to be the only +candidate that preserved that*. The answer is no, and the premise fails with it. + +| SeaweedFS capability | Apache-2.0 build | Enterprise (proprietary, per-TB) | +|---|---|---| +| OIDC on the S3 API, via STS/JWT token exchange | **yes** | yes | +| Admin UI sign-in | **username and password only** | **OIDC SSO** (Keycloak and others) | + +The admin UI is Apache-2.0 and real; its **identity provider integration is not**. It sits behind +the commercial licence together with point-in-time recovery, automatic erasure-code repair, +native multi-tenancy and S3 QoS limiting. + +So the free build gives OIDC for *programmatic* access and a locally-authenticated console. The +behaviour in use today — a human redirected to the mesh's identity provider to reach the object +store's console — is not preserved. The narrowing was sound reasoning on wrong information. + +## The three-way comparison, as measured + +Against the requirement set in the overview, which was taken from the module rather than assumed. + +| | SeaweedFS | Garage | RustFS | +|---|---|---|---| +| Licence | Apache-2.0 | AGPL-3.0 | Apache-2.0 | +| Maturity | twelve-plus years | several years, stable | **GA 2026-09-16** | +| Redundancy | erasure coding | **replication only** — a replica count, commonly 3, so ~3× raw per byte | erasure coding | +| S3 API | broad | **explicitly not the full span of endpoints** | broad, MinIO-shaped | +| Console | `weed admin`, in the free build | **none official** (a third-party UI exists) | yes, modelled on MinIO's | +| Console OIDC | Enterprise only | **none** | **yes** — documented against Keycloak | +| Programmatic bucket + credential creation | `weed shell` / IAM API | **full admin REST API**, scoped tokens for `CreateBucket` and `CreateKey` | admin API under its own `v3` namespace; client-compatible with MinIO's | +| Permission model | IAM users, groups, policies | **its own**: per access key, per bucket, read/write/owner — no AWS-style ACLs or bucket policies | MinIO-shaped IAM: users, groups, policies, service accounts | + +**No candidate is free of cost.** That is the finding, and it is why the comparison is reopened +rather than resolved here. + +## What each one actually costs + +**SeaweedFS** — the safe engineering choice. Longest field record, erasure coding, permissive +licence. The cost is the console regression: either accept local credentials for the console, pay +per TB, or front it with an authenticating proxy (see below). + +**Garage** — the best fit for how the mesh *provisions*. Its admin API is a first-class REST +surface with scoped tokens for exactly the two operations the bucket provision performs, which is +a better match than any other candidate. Three costs, and they are not small: redundancy is +replication, so the storage bill is a multiple rather than erasure coding's overhead; it does not +implement the full S3 endpoint span, which has to be checked against what consumers actually call; +and there is neither a console nor identity integration, so the console requirement is not +regressed but *removed*. + +**RustFS** — the only candidate that preserves the live behaviour. Apache-2.0, erasure coding, a +MinIO-shaped console with **documented OIDC against Keycloak**, and an admin API close enough to +MinIO's that the existing tool handlers are the least work to port. Against that: + +- **It reached GA on 2026-09-16 — eight days before this was written.** For a component that + holds data, field record is a feature, and it does not have one yet. +- **The overview's characterisation of it was wrong and is corrected here.** It was recorded as a + binary-level drop-in retaining existing data, which would make migration trivial. In fact API + compatibility and on-disk compatibility are *separate* paths, and the on-disk one is + preview-scoped with documented encryption limitations. It should not be relied on. This costs + nothing in practice — the migration track already moves data over the S3 API, not on disk — but + the claim should not survive into a decision record. +- **There is an open defect in precisely the area the mesh depends on.** An access key created by + an OIDC/SSO user is reported denied on all S3 operations, the embedded policy being ignored. + The mesh mints credentials for every provisioned bucket. Whether this touches the mesh's path — + which provisions with administrative credentials rather than as an SSO user — is **unverified, + and must be tested before RustFS is chosen**, not after. + +**Ceph RGW** stays rejected, for the reason already recorded: disproportionate where the object +store is an ordinary module rather than a platform. + +## The option that changes the trade + +The console regression is not necessarily the product's to solve. An authenticating proxy in +front of whatever console exists — against the identity provider the mesh already runs, behind +the reverse proxy it already runs — restores single sign-on at the proxy layer for **SeaweedFS or +Garage**, without a commercial licence and without betting on the youngest candidate. + +If that holds, the decision stops being *which regression to accept* and becomes a straight +comparison of storage properties, where SeaweedFS's field record and erasure coding are the +strongest position. **It is unverified.** It is the cheapest next measurement available and it +should be made before the record is written. + +## What is still unmeasured + +Named so the effort is not closed while pretending otherwise. + +1. Whether an authenticating proxy in front of a console is acceptable as the mesh's answer to + console SSO — a design question as much as a technical one, since it moves identity out of the + product and into the edge for this module. +2. Which S3 endpoints the consumers actually call, checked against Garage's supported span rather + than assumed. Until that is counted, Garage cannot be fairly ranked. +3. Whether RustFS's OIDC-credential defect touches the credential path the bucket provision uses. +4. What the redundancy change costs in real terms — replication against erasure coding at the + sizes actually stored, rather than as a ratio in the abstract. +5. Whether the four-node erasure-coded topology is still warranted at all, which the overview + already raised and nothing here answers. + +**No candidate should be graduated to a decision until 1 and 2 are measured.** 3 gates RustFS +specifically. The effort stays open.