novox/hq
6
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Record the maintained MinIO fork as the object store, and choose Garage only after the consumers' S3 features are counted (research 015, issue 413)

Browse Source
This commit is contained in:
jochen
2026-10-10 21:52:00 +02:00
parent 16cfe6d8c3
commit b6c592cb9d
2 changed files with 167 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-RESEARCH/015-the-object-store-after-minio/00-overview.md
+3 -1
View File
@@ -1,5 +1,7 @@
---
status: active
status: graduated
became:
- 02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md
initiated: 2026-09-24
touches:
- 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md
02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md
+164
View File
@@ -0,0 +1,164 @@
---
topic: what runs on it
status: accepted
date: 2026-10-10
deciders: jochen
reconstructed: false
---
# 288. The object store stays on the maintained fork, and Garage is chosen only after the consumers' S3 features are counted
## Context
[Research 015](../01-RESEARCH/015-the-object-store-after-minio/00-overview.md) asked which S3 implementation
replaces MinIO as the mesh's object store, and how the data moves. It found two decisions, not one: whether to
repoint to the maintained fork of MinIO or to migrate, and, if migrating, to which product. It ranked the
migration candidates Garage, then SeaweedFS, and not yet RustFS, and it said nothing graduates before two things
are measured: which S3 features the consumers use, and whether the fork reads the incumbent's data.
**What happened since, measured on 2026-10-10.**
- [Issue 113](../04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md) (the object store's
images were withdrawn upstream) is resolved in the issue tracker. Its fix repinned the object-store module to
the maintained fork's server image by digest, built the module's client from source, and moved its data off the
predecessor's directory. No decision record chose that fork; the fix did. This record is that decision.
- The object store runs on the control-node from that pin: the tool runner lists the module's four tools there
(`mesh_search "minio"`), and the module's manifest on the catalogue's main branch names the fork's image.
- The manifest declares **one data directory on one node**, not the eight-directory erasure-coded shape research
015 measured. That shape was the predecessor's. The question the research left open (*is that shape still
warranted?*) is therefore answered for the running module: it was not carried over.
- The manifest's upgrade policy is `record`: a person takes each build of the module, after a backup
([ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)),
and its reason names what the store holds: **the operator's photos, which are irreplaceable**.
**The count of S3 features, from the catalogue only.** The count was taken from the consumers' manifests and
code in the catalogue repository, and nothing was read from the store itself, because the store holds the
photos and a count needs none of their bytes. What the catalogue shows:
| Who | Where it was read | S3 features it uses |
|---|---|---|
| The object store's own tools and provisioner | the module's client, tools and provisioner code | list, head, create and delete a bucket; list objects; head an object; a presigned GET or PUT; an admin service account per consumer with a policy scoped to its one bucket |
| A file-sync application, using the store as primary storage | its manifest | path-style addressing, a region, plain HTTP, one bucket, no server-side encryption key configured |
| A second application, keeping its records in the document store and its files in a bucket | its manifest | an endpoint, a port, one bucket and one access key, passed to its own code |
Two modules require the `s3-bucket` provision, and nothing else in the catalogue does. The provider's own code
uses no object versioning, no object locking, no server-side encryption and no lifecycle rules. **The two
consumers' own S3 calls are not in the catalogue**: the file-sync application's object-store code is its
upstream's, and the second application is built from a repository of its own. So the count cannot be finished
from the catalogue's code and manifests alone. Finishing it means reading those two codebases, which touches
no data. Those four features are the ones Garage lacks, which is why the count decides whether Garage can be
chosen.
**What nothing declares yet.** Neither consumer says `kept-by` for its bucket, so the store's data is declared
`valuable` under [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md),
while the manifest's own words say the photos in it are `irreplaceable`.
The record was searched first (`records_search` for "object store minio fork garage") and returned nothing.
## Considered Options
### What the object store runs now
1. **Stay on the maintained fork, as issue 113's fix left it.** Chosen. It costs nothing more: the fork is
running, a node that holds nothing can obtain it again, and patches resume. It does not end the dependence on
a codebase its original authors abandoned, and this record does not pretend it does.
2. *Migrate now, to the best-ranked candidate.* Rejected. It moves irreplaceable data on a ranking that one
unmeasured fact (the features the consumers call) can still overturn, and it needs a maintenance window for
the file-sync application, whose copy drifts while it runs.
3. *Migrate now, to RustFS, because its admin API is closest and the handlers port with least work.* Rejected,
for the reason research 015 gave: it reached general availability eight days before the comparison was
written, it is not an in-place drop-in, and it had an open defect in the credential path the mesh relies on.
### Which product a migration moves to
1. **Garage first, chosen only once the count shows no consumer needs what it lacks.** Chosen. Its permission
model (one access key, one bucket, read, write or owner) is the mesh's bucket provision stated directly, its
admin API matches how the mesh provisions, and the file-sync application documents it as primary storage.
2. *SeaweedFS first.* Rejected as first: its S3 surface translates onto its own file-system API, and the
file-sync application does not document it. It stays the second candidate if the count rules Garage out.
3. *Ceph RGW.* Rejected, as in research 015: a platform, where the object store is one ordinary module.
### Where the count comes from
1. **The consumers' code and manifests, never the store.** Chosen. Listing buckets or objects reads the
operator's data to answer a question the code answers, and a mistake in that reading is a mistake against
irreplaceable data.
2. *The store's access log or its bucket configuration.* Rejected for the same reason, and because it shows
only what was called in the window logged, not what the code can call.
## Decision
### 1. The object store is the maintained fork of MinIO, for now
The object-store module runs the maintained fork, pinned by digest, as issue 113's fix left it. This record is
the decision that fix lacked. The dependency stays on **S3 the protocol**, as
[to-be 07](../03-DESIGN/01-to-be/07-the-foundation.md) states it, and the product can change without a change of
design.
The fork is reconsidered at once, with an issue opened, when any of these holds:
- the fork publishes no release for **six months**;
- a security advisory against the pinned release stays unpatched for **30 days**;
- the image at the pinned digest can no longer be pulled by a node that holds nothing.
### 2. Garage is the first migration candidate, and it is chosen only after the count
The first step of any migration is the count: for each consumer of `s3-bucket`, the S3 operations and features
its code calls, read from its code and manifest, written into issue 413. **Tracked by issue 413**, whose worker
writes the count. The count reads no data from the store: not a bucket listing, not an object, not a log.
Garage is chosen only if no consumer needs object versioning, object locking, server-side encryption or a
lifecycle rule beyond plain expiry. If one does, SeaweedFS is measured next on the same count, and a new record
says which.
### 3. A migration moves only with a backup first, a check of every object after, and the operator's go
The object store holds the operator's photos, and they are irreplaceable. Any move of its data, to Garage or to
anything else, follows all three of these, in this order, and none is skipped:
1. **A backup taken first.** A restore point of the store's data directory, taken by the node's `node-backup`
holder before the first byte is copied, and restored beside the live data once to prove it reads. A move
that starts without a proven restore point is refused.
2. **A check of every object after.** For every bucket: the same object count on both sides, and for every
object the same size and the same checksum. A transfer tool's exit code is not a check. One object that
differs fails the move.
3. **The operator's go.** The operator says go before the copy starts and again before consumers are pointed at
the new store. The binding of a consumer to its data moves only by a person
([ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md)), so no agent and no walk does it
alone.
The incumbent stays running read-only beside the new store as the way back for at least **30 days** after the
consumers are pointed away, and its data is removed only after that, by a person, on its own go. The file-sync
application's final copy is taken with its writes stopped, in a window the operator agrees to.
### Small limits, set here
- The reconsideration triggers in §1: six months without a release, 30 days for an unpatched advisory.
- The 30 days the incumbent stays as the way back in §3.
- A check compares every object, not a sample: the measured store held 82,496 objects in research 015, and every
one is compared.
## Consequences
- Issue 113's fix now has its decision, and the mesh's object store is no longer an unrecorded choice.
- No to-be design is written by this record, because no design changes: to-be 07 already says the dependency is on
the protocol. The design for a migration is written when the count chooses its product, under issue 413.
- Research 015's open question on mirroring third-party images into the mesh's own registry is not answered
here. It remains the open question of issue 113, and it is a general one about every module's images.
- What got harder: nothing moves off the fork without the count, a proven restore point, a check of every object
and the operator's go twice. That is slower than a copy, on purpose.
- The gap between the store's declared class (`valuable`) and the photos it holds (`irreplaceable`) stands until a
consumer declares `kept-by`; issue 413 tracks it beside the count, because the backup a migration takes is
sized and kept by that class.
## References
- [Research 015 — the object store after MinIO](../01-RESEARCH/015-the-object-store-after-minio/00-overview.md) and
[its comparison](../01-RESEARCH/015-the-object-store-after-minio/01-candidate-comparison.md).
- [Issue 113](../04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md) and its fix in the
catalogue's object-store module.
- Issue 413 (the count of S3 features, and any migration), in the issue tracker.
- [ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md),
[ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md),
[ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md).
- [To-be 07 — the foundation](../03-DESIGN/01-to-be/07-the-foundation.md).