Research 015: reopen the comparison — the premise for narrowing to one candidate was false #105

2026-09-24T14:10:52Z

jschoubben commented

2026-09-24 14:10:52 +00:00

Research 015 named one question as deciding the choice, and scoped SeaweedFS as primary because it appeared to be the only candidate preserving OIDC console login. That question has now been measured. The premise fails.

What was measured

SeaweedFS capability	Apache-2.0 build	Enterprise (proprietary, per-TB)
OIDC on the S3 API (STS/JWT)	yes	yes
Admin UI sign-in	username + password only	OIDC SSO

The weed admin UI is Apache-2.0. Its identity-provider integration is not — console SSO sits behind the commercial licence with point-in-time recovery, automatic EC repair, multi-tenancy and S3 QoS.

So the live behaviour — a human redirected to the mesh's identity provider to reach the object store's console — is not preserved by the free build. The narrowing was sound reasoning on wrong information.

The reopened three-way

Full table in 01-candidate-comparison.md. In brief, no candidate is free of cost:

SeaweedFS — longest field record, erasure coding, permissive licence. Costs the console regression.
Garage — best match for how the mesh provisions (first-class admin REST API, scoped tokens for exactly CreateBucket/CreateKey). Costs replication instead of erasure coding (~3× raw), no full S3 endpoint span, and neither console nor identity integration.
RustFS — the only candidate preserving the live behaviour: Apache-2.0, erasure coding, MinIO-shaped console with documented OIDC. Costs a GA dated 2026-09-16 — eight days ago, plus an open defect in the credential path.

Two corrections to what 015 previously recorded

RustFS is not a binary-level drop-in retaining existing data. API compatibility and on-disk compatibility are separate paths; the on-disk one is preview-scoped with documented encryption limits. Costs nothing in practice — the migration track already moves data over the S3 API — but the claim must not survive into a decision record.
RustFS has an open defect in exactly the area the mesh depends on — an access key created by an OIDC/SSO user reported denied on all S3 operations, embedded policy ignored. The mesh mints a credential for every provisioned bucket. Whether it touches the mesh's path (which provisions with administrative credentials, not as an SSO user) is unverified and gates RustFS specifically.

The option that changes the trade

An authenticating proxy in front of whatever console exists, against the identity provider and reverse proxy the mesh already runs, restores SSO at the edge for SeaweedFS or Garage — no commercial licence, no bet on the youngest candidate. If it holds, the decision stops being which regression to accept and becomes a straight comparison of storage properties. It is unverified, and it is the cheapest next measurement.

Effort stays open

status: active. Five items listed as unmeasured; nothing graduates before two of them — whether a proxy is an acceptable answer to console SSO, and which S3 endpoints consumers actually call (Garage cannot be fairly ranked until that is counted).

Checks

cycle:   263 documents checked, the chain holds
records: 96 decision records checked, all checks passed
index:   current

Links and the cross-document anchor resolve; disclosure scan clean. Product names (Keycloak, MinIO, SeaweedFS) appear only as facts about the candidates' own documentation, never as claims about this installation.

Research 015 named one question as deciding the choice, and scoped SeaweedFS as primary **because it appeared to be the only candidate preserving OIDC console login**. That question has now been measured. The premise fails. ## What was measured | SeaweedFS capability | Apache-2.0 build | Enterprise (proprietary, per-TB) | |---|---|---| | OIDC on the S3 API (STS/JWT) | **yes** | yes | | Admin UI sign-in | **username + password only** | **OIDC SSO** | The `weed admin` UI *is* Apache-2.0. Its **identity-provider integration is not** — console SSO sits behind the commercial licence with point-in-time recovery, automatic EC repair, multi-tenancy and S3 QoS. So the live behaviour — a human redirected to the mesh's identity provider to reach the object store's console — is not preserved by the free build. The narrowing was sound reasoning on wrong information. ## The reopened three-way Full table in `01-candidate-comparison.md`. In brief, **no candidate is free of cost**: - **SeaweedFS** — longest field record, erasure coding, permissive licence. Costs the console regression. - **Garage** — best match for how the mesh *provisions* (first-class admin REST API, scoped tokens for exactly `CreateBucket`/`CreateKey`). Costs replication instead of erasure coding (~3× raw), no full S3 endpoint span, and neither console nor identity integration. - **RustFS** — the only candidate preserving the live behaviour: Apache-2.0, erasure coding, MinIO-shaped console with documented OIDC. Costs a **GA dated 2026-09-16 — eight days ago**, plus an open defect in the credential path. ## Two corrections to what 015 previously recorded 1. **RustFS is not a binary-level drop-in retaining existing data.** API compatibility and on-disk compatibility are *separate* paths; the on-disk one is preview-scoped with documented encryption limits. Costs nothing in practice — the migration track already moves data over the S3 API — but the claim must not survive into a decision record. 2. **RustFS has an open defect in exactly the area the mesh depends on** — an access key created by an OIDC/SSO user reported denied on all S3 operations, embedded policy ignored. The mesh mints a credential for every provisioned bucket. Whether it touches the mesh's path (which provisions with administrative credentials, not as an SSO user) is **unverified and gates RustFS specifically**. ## The option that changes the trade An authenticating proxy in front of whatever console exists, against the identity provider and reverse proxy the mesh already runs, restores SSO at the edge for **SeaweedFS or Garage** — no commercial licence, no bet on the youngest candidate. If it holds, the decision stops being *which regression to accept* and becomes a straight comparison of storage properties. **It is unverified**, and it is the cheapest next measurement. ## Effort stays open `status: active`. Five items listed as unmeasured; **nothing graduates before two of them** — whether a proxy is an acceptable answer to console SSO, and which S3 endpoints consumers actually call (Garage cannot be fairly ranked until that is counted). ## Checks ``` cycle: 263 documents checked, the chain holds records: 96 decision records checked, all checks passed index: current ``` Links and the cross-document anchor resolve; disclosure scan clean. Product names (Keycloak, MinIO, SeaweedFS) appear only as facts about the candidates' own documentation, never as claims about this installation.

jschoubben added 1 commit 2026-09-24 14:10:52 +00:00

Research 015: reopen the comparison — the premise for narrowing to one candidate was false c8cbbcfb8a

SeaweedFS was scoped as primary because it looked like the only candidate preserving
OIDC console login. Measured: its admin UI is Apache-2.0 but its identity-provider
integration is not — console SSO sits behind the per-TB commercial licence, alongside
point-in-time recovery and automatic EC repair. The free build gives OIDC on the S3 API
via STS and a console authenticated by local username and password.

So the answer to the gating question is that no candidate preserves the current feature
set for free, which this effort had written down as a possible outcome. Reopened across
three candidates with the requirement-by-requirement evidence in 01.

Two corrections to what the overview recorded. RustFS is not a binary-level drop-in
retaining existing data: API and on-disk compatibility are separate paths and the on-disk
one is preview-scoped. And it carries an open defect in the credential path the bucket
provision depends on, which gates it specifically.

Nothing graduates before two measurements named in 01: whether an authenticating proxy
is an acceptable answer to console SSO, and which S3 endpoints consumers actually call —
the latter because Garage does not implement the full span and cannot be ranked until
that is counted.

jschoubben commented

2026-09-24 16:33:20 +00:00

Do not merge as-is. Holding open rather than closing, because it carries measured findings that exist nowhere else in the repo — but its conclusions are now superseded.

What in here still stands

SeaweedFS's console OIDC is an Enterprise feature. The weed admin UI is Apache-2.0; its identity-provider integration is not. Measured, and it falsified the original narrowing.
The two RustFS corrections. It is not a binary-level drop-in retaining existing data (API and on-disk compatibility are separate paths; on-disk is preview-scoped), and it has an open defect where an OIDC-created access key is denied all S3 operations.

What is now wrong

Console SSO is not a requirement. The operator's actual requirement is per-application access keys scoped to buckets — Nextcloud and friends as "users". This document is organised around a console requirement that does not exist, which is the axis the whole comparison is ranked on.
Garage is under-rated as a result. Its per-access-key-per-bucket model is the stated requirement rather than an approximation of it, its admin API has tokens scoped to exactly CreateBucket/CreateKey, and it is first-party documented for Nextcloud as primary storage.
The replication penalty is quantified now and it is immaterial: 230 GiB logical / 82,496 objects / 10 buckets, with 1.3 TiB free. Garage at rf=3 costs ~+222 GiB against MinIO's existing 2.03× (EC:4). Also, all eight "drives" are directories on one array, so EC-vs-replication is an overhead question, not a resilience one.
It does not mention the option that now leads — docker.io/pgsty/minio, a maintained fork of the community edition that keeps the on-disk format, S3 API and MINIO_* variables compatible, restores the stripped console, and tracks CVEs. Verified by pulling it. That makes "repoint and keep the data" a live candidate this document does not consider at all, and it changes the urgency of replacing the product entirely.
Nextcloud is already live on S3 (since 2023-01-15, opaque urn:oid: objects with metadata in its database), so migration needs a maintenance window — which the migration outline does not account for.

What it needs before it can merge

A rewrite against the real requirement set, with the fork as a candidate and the numbers folded in. The object-store question is paused by the operator, so this waits rather than being rewritten now.

Also unrelated and worth noting while this is open: issue 113's diagnosis is wrong and merged. It records four claims as unsubstantiated — a module.json, a digest pin at line 83, an all-zeros runtime digest — on the basis of a grep that only covered the monorepo. All three exist in mesh-catalog/modules/minio/module.json. That retraction is owed regardless of what happens to this PR.

**Do not merge as-is.** Holding open rather than closing, because it carries measured findings that exist nowhere else in the repo — but its conclusions are now superseded. ## What in here still stands - **SeaweedFS's console OIDC is an Enterprise feature.** The `weed admin` UI is Apache-2.0; its identity-provider integration is not. Measured, and it falsified the original narrowing. - **The two RustFS corrections.** It is not a binary-level drop-in retaining existing data (API and on-disk compatibility are separate paths; on-disk is preview-scoped), and it has an open defect where an OIDC-created access key is denied all S3 operations. ## What is now wrong 1. **Console SSO is not a requirement.** The operator's actual requirement is per-**application** access keys scoped to buckets — Nextcloud and friends as "users". This document is organised around a console requirement that does not exist, which is the axis the whole comparison is ranked on. 2. **Garage is under-rated as a result.** Its per-access-key-per-bucket model *is* the stated requirement rather than an approximation of it, its admin API has tokens scoped to exactly `CreateBucket`/`CreateKey`, and it is first-party documented for Nextcloud as primary storage. 3. **The replication penalty is quantified now and it is immaterial:** 230 GiB logical / 82,496 objects / 10 buckets, with 1.3 TiB free. Garage at rf=3 costs ~+222 GiB against MinIO's existing 2.03× (EC:4). Also, all eight "drives" are directories on one array, so EC-vs-replication is an overhead question, not a resilience one. 4. **It does not mention the option that now leads** — `docker.io/pgsty/minio`, a maintained fork of the community edition that keeps the on-disk format, S3 API and `MINIO_*` variables compatible, restores the stripped console, and tracks CVEs. Verified by pulling it. That makes "repoint and keep the data" a live candidate this document does not consider at all, and it changes the urgency of replacing the product entirely. 5. **Nextcloud is already live on S3** (since 2023-01-15, opaque `urn:oid:` objects with metadata in its database), so migration needs a maintenance window — which the migration outline does not account for. ## What it needs before it can merge A rewrite against the real requirement set, with the fork as a candidate and the numbers folded in. The object-store question is **paused** by the operator, so this waits rather than being rewritten now. Also unrelated and worth noting while this is open: **issue 113's diagnosis is wrong** and merged. It records four claims as unsubstantiated — a `module.json`, a digest pin at line 83, an all-zeros runtime digest — on the basis of a grep that only covered the monorepo. All three exist in `mesh-catalog/modules/minio/module.json`. That retraction is owed regardless of what happens to this PR.

jschoubben added 3 commits 2026-09-24 16:46:50 +00:00

Merge remote-tracking branch 'origin/main' into work/object-store-records 48abc36b5b

Issue 113: retract the diagnosis table — the original report was right 999636e2a8

The diagnosis carried a table headed "claims that could not be substantiated", denying a
module.json, a digest pin, and an all-zeros runtime digest. All three exist. The table is
withdrawn in full and replaced with what is actually true, plus the two claims that remain
genuinely unverified rather than disproven.

The cause: one repository was searched and absence in it was written up as absence. The
catalogue of the mesh being built is a separate repository, not checked out where the search
ran, and all four claims were about that repository. Compounding it, the predecessor's
object-store module and the one being cut over to were treated as one thing — they are
different files in different repositories, one pinning a tag with no sidecar, the other a
digest with two container resources.

Also corrected in the report: located-in named the wrong repository; the "pins a tag" passage
described the predecessor; the open question about pinning by digest is struck, because this
module already does and it made no difference — a deleted digest resolves to nothing either
way. The section on why nothing broke is now scoped explicitly to the predecessor's
machinery.

The lesson kept in the record: "zero occurrences anywhere in the tree" is only as strong as
the tree searched, and a diagnosis must say which tree. A confident rebuttal of a correct
report is worse than no diagnosis — it sends the next person to the wrong place with a
written record behind them.

Research 015: rewrite the comparison — wrong axis, and a missing candidate 1d524a1fa8

The previous version ranked candidates on whether they preserved single sign-on to the
object store's console. That is not a requirement: a "user" of the store is normally an
application, so the requirement is per-application keys scoped to buckets — which the mesh
already mints. And the console login it ranked on never worked; the module's own hook comment
records "policy claim missing", a failing login written up as progress.

It also omitted the incumbent's own maintained fork, which changes the question from "which
product replaces it" into two decisions: repoint, or migrate — and if migrating, to which.
Repointing costs an image reference; migrating costs a data copy, two handler rewrites and a
maintenance window. Repointing does not foreclose migrating, which is the argument for taking
it first.

On the corrected requirement Garage ranks first — its per-key-per-bucket model is the
requirement verbatim, its admin API matches how the mesh provisions, and the highest-risk
consumer is first-party documented against it. Its remaining gap (no versioning, no
server-side encryption, partial lifecycle) is unmeasured against the buckets and is the one
thing that could still disqualify it.

Measured and folded in: 230 GiB logical, 82,496 objects, 468 GiB raw at 2.03x, eight drive
directories on one filesystem on one machine. That last fact decides more than any feature —
the erasure coding is not buying independent-drive redundancy, so the redundancy model is
close to irrelevant and only storage overhead remains, which at this volume is a rounding
error against the headroom.

Both errors are recorded at the end of 01 rather than quietly fixed. A configured feature is
not an observed one; and when a dependency dies, "who took it over" precedes "what replaces
it" — searching for alternatives by construction returns things that are not the incumbent.

jschoubben commented

2026-09-24 16:47:07 +00:00

Updated — now carries both corrections

Pushed 999636e and 1d524a1, plus a merge of current main (base had moved). Supersedes my earlier "do not merge as-is" comment — the objections in it are now addressed in the files.

1. Issue 113's diagnosis retracted

The table headed "claims that could not be substantiated" denied a module.json, a digest pin at line 83, and an all-zeros runtime digest. All three exist. The original report was right and I rebutted it in the permanent record.

Withdrawn in full and replaced with what is true, distinguishing the two claims that remain genuinely unverified from those that were disproven — a readiness document that is simply not on the machine searched (a separate private repository), and a cached-image comparison that was never actually made against the digest this module pins.

Cause, stated in the record: one repository was searched and absence in it was written up as absence. The catalogue of the mesh being built is a separate repository, not checked out where the search ran — and all four claims were about it. Compounding it, the predecessor's object-store module and the one being cut over to were treated as one thing; they are different files in different repositories, one pinning a tag with no sidecar, the other a digest with two container resources.

Also corrected in 00-report.md: located-in: named the wrong repository; the "pins a tag, four and a half years old" passage described the predecessor; the section on why nothing broke is now scoped explicitly to the predecessor's machinery; and the open question "should images be pinned by digest?" is struck — this module already does, and it made no difference, because a deleted digest resolves to nothing either way.

2. The comparison rewritten

The ranking axis was wrong. Console SSO is not a requirement — a "user" is normally an application, so the requirement is per-application keys scoped to buckets, which the mesh already mints. And the login it ranked on never worked.
A fourth candidate was missing: the incumbent's own maintained fork. That turns this into two decisions — repoint, or migrate; and if migrating, to which. Repointing costs an image reference; migrating costs a data copy, two handler rewrites and a maintenance window. Repointing doesn't foreclose migrating, which is the argument for taking it first.
Garage now ranks first for the migration case, for the reasons in the doc. Its gap (no versioning, no SSE, partial lifecycle) is unmeasured against the ten buckets and is the one thing that could still disqualify it.
Numbers folded in: 230 GiB / 82,496 objects / 468 GiB raw at 2.03×, eight drive directories on one filesystem on one machine — which decides more than any feature, since the erasure coding isn't buying independent-drive redundancy.
The migration outline now includes the maintenance window for the opaque consumer, and a warning not to reuse a data directory the incumbent already holds.

Both errors are written up at the end of 01 rather than quietly fixed: a configured feature is not an observed one, and when a dependency dies, "who took it over" precedes "what replaces it" — searching for alternatives by construction returns things that are not the incumbent.

Verification

Done in an isolated worktree per playbook 07. Links: 0 broken across all four files. Disclosure scan clean — no node names, domains, paths, addresses, or bucket names. Effort stays status: active; nothing graduates before the two named measurements.

cycle:   265 documents checked, the chain holds
records: all checks passed
index:   current

## Updated — now carries both corrections Pushed `999636e` and `1d524a1`, plus a merge of current `main` (base had moved). **Supersedes my earlier "do not merge as-is" comment** — the objections in it are now addressed in the files. ### 1. Issue 113's diagnosis retracted The table headed *"claims that could not be substantiated"* denied a `module.json`, a digest pin at line 83, and an all-zeros runtime digest. **All three exist.** The original report was right and I rebutted it in the permanent record. Withdrawn in full and replaced with what is true, distinguishing the two claims that remain *genuinely unverified* from those that were *disproven* — a readiness document that is simply not on the machine searched (a separate private repository), and a cached-image comparison that was never actually made against the digest this module pins. **Cause, stated in the record:** one repository was searched and absence in it was written up as absence. The catalogue of the mesh being built is a separate repository, not checked out where the search ran — and all four claims were about it. Compounding it, the predecessor's object-store module and the one being cut over to were treated as one thing; they are different files in different repositories, one pinning a tag with no sidecar, the other a digest with two container resources. Also corrected in `00-report.md`: `located-in:` named the wrong repository; the "pins a tag, four and a half years old" passage described the predecessor; the section on why nothing broke is now scoped explicitly to the predecessor's machinery; and the open question *"should images be pinned by digest?"* is **struck** — this module already does, and it made no difference, because a deleted digest resolves to nothing either way. ### 2. The comparison rewritten - **The ranking axis was wrong.** Console SSO is not a requirement — a "user" is normally an application, so the requirement is per-application keys scoped to buckets, which the mesh already mints. And the login it ranked on **never worked**. - **A fourth candidate was missing:** the incumbent's own maintained fork. That turns this into *two* decisions — repoint, or migrate; and if migrating, to which. Repointing costs an image reference; migrating costs a data copy, two handler rewrites and a maintenance window. Repointing doesn't foreclose migrating, which is the argument for taking it first. - **Garage now ranks first for the migration case**, for the reasons in the doc. Its gap (no versioning, no SSE, partial lifecycle) is **unmeasured against the ten buckets and is the one thing that could still disqualify it**. - **Numbers folded in:** 230 GiB / 82,496 objects / 468 GiB raw at 2.03×, eight drive directories **on one filesystem on one machine** — which decides more than any feature, since the erasure coding isn't buying independent-drive redundancy. - The migration outline now includes the **maintenance window** for the opaque consumer, and a warning not to reuse a data directory the incumbent already holds. Both errors are written up at the end of `01` rather than quietly fixed: *a configured feature is not an observed one*, and *when a dependency dies, "who took it over" precedes "what replaces it"* — searching for alternatives by construction returns things that are not the incumbent. ### Verification Done in an isolated worktree per playbook 07. Links: 0 broken across all four files. Disclosure scan clean — no node names, domains, paths, addresses, or bucket names. Effort stays `status: active`; nothing graduates before the two named measurements. ``` cycle: 265 documents checked, the chain holds records: all checks passed index: current ```

jschoubben merged commit cdcd4da27e into main

2026-09-24 16:47:19 +00:00

jschoubben deleted branch storage/015-reopen-the-candidate-comparison

2026-09-24 16:47:19 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/hq#105