diff --git a/01-RESEARCH/015-the-object-store-after-minio/00-overview.md b/01-RESEARCH/015-the-object-store-after-minio/00-overview.md index 88707ee5..a4d3633d 100644 --- a/01-RESEARCH/015-the-object-store-after-minio/00-overview.md +++ b/01-RESEARCH/015-the-object-store-after-minio/00-overview.md @@ -1,5 +1,7 @@ --- -status: active +status: graduated +became: + - 02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md initiated: 2026-09-24 touches: - 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md diff --git a/02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md b/02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md new file mode 100644 index 00000000..15188fbe --- /dev/null +++ b/02-DECISIONS/0288-the-object-store-stays-on-the-maintained-fork-and-garage-is-chosen-only-after-the-consumers-s3-features-are-counted.md @@ -0,0 +1,171 @@ +--- +topic: what runs on it +status: accepted +date: 2026-10-10 +deciders: jochen +reconstructed: false +--- + +# 288. The object store stays on the maintained fork, and Garage is chosen only after the consumers' S3 features are counted + +## Context + +[Research 015](../01-RESEARCH/015-the-object-store-after-minio/00-overview.md) asked which S3 implementation +replaces MinIO as the mesh's object store, and how the data moves. It found two decisions, not one: whether to +repoint to the maintained fork of MinIO or to migrate, and, if migrating, to which product. It ranked the +migration candidates Garage, then SeaweedFS, and not yet RustFS, and it said nothing graduates before two things +are measured: which S3 features the consumers use, and whether the fork reads the incumbent's data. + +**What happened since, measured on 2026-10-10.** + +- [Issue 113](../04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md) (the object store's + images were withdrawn upstream) is resolved in the issue tracker. Its fix repinned the object-store module to + the maintained fork's server image by digest, built the module's client from source, and moved its data off the + predecessor's directory. No decision record chose that fork; the fix did. This record is that decision. +- The object store runs on the control-node from that pin: the tool runner lists the module's four tools there + (`mesh_search "minio"`), and the module's manifest on the catalogue's main branch names the fork's image. +- The manifest declares **one data directory on one node**, not the eight-directory erasure-coded shape research + 015 measured. That shape was the predecessor's. The question the research left open (*is that shape still + warranted?*) is therefore answered for the running module: it was not carried over. +- The manifest's upgrade policy is `record`: a person takes each build of the module, after a backup + ([ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)), + and its reason names what the store holds: **the operator's photos, which are irreplaceable**. + +**The count of S3 features, from the catalogue only.** The count was taken from the consumers' manifests and +code in the catalogue repository, and nothing was read from the store itself, because the store holds the +photos and a count needs none of their bytes. What the catalogue shows: + +| Who | Where it was read | S3 features it uses | +|---|---|---| +| The object store's own tools and provisioner | the module's client, tools and provisioner code | list, head, create and delete a bucket; list objects; head an object; a presigned GET or PUT; an admin service account per consumer with a policy scoped to its one bucket | +| A file-sync application, using the store as primary storage | its manifest | path-style addressing, a region, plain HTTP, one bucket, no server-side encryption key configured | +| A second application, keeping its records in the document store and its files in a bucket | its manifest | an endpoint, a port, one bucket and one access key, passed to its own code | + +Two modules require the `s3-bucket` provision, and nothing else in the catalogue does. The provider's own code +uses no object versioning, no object locking, no server-side encryption and no lifecycle rules. **The two +consumers' own S3 calls are not in the catalogue**: the file-sync application's object-store code is its +upstream's, and the second application is built from a repository of its own. So the count cannot be finished +from the catalogue's code and manifests alone. Finishing it means reading those two codebases, which touches +no data. Those four features are the ones Garage lacks, which is why the count decides whether Garage can be +chosen. + +**What nothing declares yet.** Neither consumer says `kept-by` for its bucket, so the store's data is declared +`valuable` under [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md), +while the manifest's own words say the photos in it are `irreplaceable`. + +The record was searched first (`records_search` for "object store minio fork garage") and returned nothing. + +## Considered Options + +### What the object store runs now + +1. **Stay on the maintained fork, as issue 113's fix left it.** Chosen. It costs nothing more: the fork is + running, a node that holds nothing can obtain it again, and patches resume. It does not end the dependence on + a codebase its original authors abandoned, and this record does not pretend it does. +2. *Migrate now, to the best-ranked candidate.* Rejected. It moves irreplaceable data on a ranking that one + unmeasured fact (the features the consumers call) can still overturn, and it needs a maintenance window for + the file-sync application, whose copy drifts while it runs. +3. *Migrate now, to RustFS, because its admin API is closest and the handlers port with least work.* Rejected, + for the reason research 015 gave: it reached general availability eight days before the comparison was + written, it is not an in-place drop-in, and it had an open defect in the credential path the mesh relies on. + +### Which product a migration moves to + +1. **Garage first, chosen only once the count shows no consumer needs what it lacks.** Chosen. Its permission + model (one access key, one bucket, read, write or owner) is the mesh's bucket provision stated directly, its + admin API matches how the mesh provisions, and the file-sync application documents it as primary storage. +2. *SeaweedFS first.* Rejected as first: its S3 surface translates onto its own file-system API, and the + file-sync application does not document it. It stays the second candidate if the count rules Garage out. +3. *Ceph RGW.* Rejected, as in research 015: a platform, where the object store is one ordinary module. + +### Where the count comes from + +1. **The consumers' code and manifests, never the store.** Chosen. Listing buckets or objects reads the + operator's data to answer a question the code answers, and a mistake in that reading is a mistake against + irreplaceable data. +2. *The store's access log or its bucket configuration.* Rejected for the same reason, and because it shows + only what was called in the window logged, not what the code can call. + +## Decision + +### 1. The object store is the maintained fork of MinIO, for now + +The object-store module runs the maintained fork, pinned by digest, as issue 113's fix left it. This record is +the decision that fix lacked. The dependency stays on **S3 the protocol**, as +[to-be 07](../03-DESIGN/01-to-be/07-the-foundation.md) states it, and the product can change without a change of +design. + +The fork is reconsidered at once, with an issue opened, when any of these holds: + +- the fork publishes no release for **six months**; +- a security advisory against the pinned release stays unpatched for **30 days**; +- the image at the pinned digest can no longer be pulled by a node that holds nothing. + +**How this is checked: nothing notices any of the three yet.** A check that pulls every pinned third-party image as +a node holding nothing would, and watches the object store's upstream for releases and advisories, is tracked by +issue 421, which is open with no worker. Until it is built, the triggers are read by hand when issue 413 is worked. + +### 2. Garage is the first migration candidate, and it is chosen only after the count + +The first step of any migration is the count: for each consumer of `s3-bucket`, the S3 operations and features +its code calls, read from its code and manifest, written into issue 413. **Tracked by issue 413**, whose worker +writes the count. The count reads no data from the store: not a bucket listing, not an object, not a log. + +Garage is chosen only if no consumer needs object versioning, object locking, server-side encryption or a +lifecycle rule beyond plain expiry. If one does, SeaweedFS is measured next on the same count, and a new record +says which. + +### 3. A migration moves only with a backup first, a check of every object after, and the operator's go + +The object store holds the operator's photos, and they are irreplaceable. Any move of its data, to Garage or to +anything else, follows all three of these, in this order, and none is skipped: + +1. **A backup taken first.** A restore point of the store's data directory, taken by the node's `node-backup` + holder before the first byte is copied, and restored beside the live data once to prove it reads. A move + that starts without a proven restore point is refused. +2. **A check of every object after.** For every bucket: the same object count on both sides, and for every + object the same size and the same checksum. A transfer tool's exit code is not a check. One object that + differs fails the move. +3. **The operator's go.** The operator says go before the copy starts and again before consumers are pointed at + the new store. The binding of a consumer to its data moves only by a person + ([ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md)), so no agent and no walk does it + alone. + +The incumbent stays running read-only beside the new store as the way back for at least **30 days** after the +consumers are pointed away, and its data is removed only after that, by a person, on its own go. The file-sync +application's final copy is taken with its writes stopped, in a window the operator agrees to. + +### Small limits, set here + +- The reconsideration triggers in §1: six months without a release, 30 days for an unpatched advisory. +- The 30 days the incumbent stays as the way back in §3. +- A check compares every object, not a sample: the measured store held 82,496 objects in research 015, and every + one is compared. + +## Consequences + +- Issue 113's fix now has its decision, and the mesh's object store is no longer an unrecorded choice. +- No to-be design is written by this record, because no design changes: to-be 07 already says the dependency is on + the protocol. The design for a migration is written when the count chooses its product, under issue 413. +- Research 015's open question on mirroring third-party images into the mesh's own registry is not answered + here. It is a general question about every module's images, and it needs a decision of its own; issue 113, + where it was first asked, is resolved, so it is tracked by issue 421 together with the check above. +- What got harder: nothing moves off the fork without the count, a proven restore point, a check of every object + and the operator's go twice. That is slower than a copy, on purpose. +- The gap between the store's declared class (`valuable`) and the photos it holds (`irreplaceable`) stands until a + consumer declares `kept-by`. It is tracked by issue 413 beside the count (the issue's second note), and the + consumer that keeps the photos declares `kept-by` before a migration starts, because the backup a migration takes is + sized and kept by that class. + +## References + +- [Research 015 — the object store after MinIO](../01-RESEARCH/015-the-object-store-after-minio/00-overview.md) and + [its comparison](../01-RESEARCH/015-the-object-store-after-minio/01-candidate-comparison.md). +- [Issue 113](../04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md) and its fix in the + catalogue's object-store module. +- Issue 413 (the count of S3 features, the missing `kept-by`, and any migration) and issue 421 (nothing notices a + withdrawn image or a silent upstream), in the issue tracker. +- [ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md), + [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md), + [ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md). +- [To-be 07 — the foundation](../03-DESIGN/01-to-be/07-the-foundation.md).