diff --git a/01-RESEARCH/015-the-object-store-after-minio/00-overview.md b/01-RESEARCH/015-the-object-store-after-minio/00-overview.md new file mode 100644 index 0000000..fee62e3 --- /dev/null +++ b/01-RESEARCH/015-the-object-store-after-minio/00-overview.md @@ -0,0 +1,119 @@ +--- +status: active +initiated: 2026-09-24 +touches: + - 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md + - 02-DECISIONS/0033-the-substrate-is-a-store-and-a-broker.md + - 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md + - 02-DECISIONS/0049-a-consumers-identity-fits-the-tightest-backend.md + - 02-DECISIONS/0078-the-store-and-broker-are-modules.md + - 02-DECISIONS/0084-which-provider-serves-a-consumer.md + - 03-DESIGN/00-as-is/03-provisioning.md + - 03-DESIGN/01-to-be/07-the-foundation.md + - 04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md +--- + +# 015 — The object store after MinIO: which S3 implementation, and how the data moves + +**The question.** The mesh's object store is MinIO. Its community edition is archived upstream, +its server and client images have been deleted from every public registry, and the pinned release +is four and a half years old and will never be patched +([issue 113](../../04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md)). +Which S3-compatible implementation replaces it, and what is the migration track for the data and +the provisioning model that sit on top of it? + +**Why now, and why not sooner.** Nothing is on fire: nodes that already hold the images keep +running, and issue 113 establishes that the deploy path tolerates an unfetchable-but-present +image by design. The forcing function is not an outage but a one-way door — **no node that does +not already hold the images can ever provision the module again**, so the mesh's ability to stand +a node up from its declarations is already broken for this module, and silently. + +**The direction is not a departure from the design; it is the design.** The foundation document +already states the commitment: + +> The dependency is on the **protocol**, not the product: AMQP for the bus, S3 for the object +> store, the OCI protocol for the registry. That is what keeps the naming safe rather than a +> commitment that cannot be revisited. + +The object store is also **not** a foundation service — ADR 0028 removed it, and it is an +ordinary module required through the module graph by whatever wants one. (The "exception that is +not a swap" in that passage is the relational store, whose provisioning model borrows PostgreSQL's +own meaning of databases, roles and schemas. The object store carries no such coupling: a bucket +is a bucket.) So this effort is an instantiation of an existing principle, not a redesign — which +is the cheapest kind of decision to make and the strongest kind to cite. + +## What the replacement has to carry, measured + +Taken from the module's manifest, its composition, its tool surface, and a search for its +consumers across the catalogue — not from assumption. + +| Requirement | Evidence in the module today | +|---|---| +| S3 API | The protocol every consumer speaks; already the design's stated dependency. | +| **OIDC login against the mesh's identity provider** | Six configuration variables are wired and populated in practice — discovery URL, client id, client secret, scopes, display name, redirect — plus a dedicated entrypoint script that blocks startup until the provider answers. This is live, not aspirational. | +| Erasure-coded multi-node topology | Four server nodes with two data directories each, behind a load balancer. | +| A single-node form | Declared as a flavour, for development and small nodes. | +| Buckets as a typed provision | The module declares a provision type of `bucket` on a named network; the mesh mints the credential and the provider creates it (ADRs 0048, 0084). | +| A tool surface | Bucket create/list/delete, object list/info/delete, presigned URL, and provisioning. | +| A console | Published on its own subdomain through the reverse proxy, with an unlimited request-body middleware for uploads. | + +**Consumers, counted:** one application module, one capture module that takes a private bucket per +node, one workflow module's tools, and the delivery/rescue internals of the shared library. The +surface is small — the cost is concentrated in the provisioning handler, the tool handlers and the +OIDC story, not spread across the catalogue. + +## Candidates + +Scoped to **SeaweedFS** as the primary, with the others recorded so the rejection is not +rediscovered. + +- **SeaweedFS** — Apache-2.0, Go, twelve-plus years of development, erasure coding, and OIDC + support in its S3/STS layer. Chosen to scope because it is the only candidate that plausibly + preserves the OIDC requirement above, which is the one requirement that is live and least + substitutable. +- **Garage** — the lightest to operate and the simplest model, but **no native identity-provider + integration**. Adopting it means losing OIDC console login or fronting it with a proxy. A real + functional regression against something currently in use. +- **RustFS** — markets itself as a binary-level drop-in retaining existing data, buckets and + configuration, which would make the data migration close to trivial. Young, and that claim is + exactly the kind that must be verified on a copy before it is believed. +- **Ceph RGW** — the most capable and the most operationally expensive; disproportionate to a mesh + where the object store is an ordinary module, not a platform. + +**The first thing to verify, because the choice turns on it:** how much of SeaweedFS's OIDC story +is in the freely licensed build, and whether its shape — IAM/STS token exchange — can actually +stand in for a console that redirects a human to an identity provider. If it cannot, the honest +finding may be that **no** candidate preserves the current feature set, and the decision becomes +which regression to accept. That question is worth answering before any migration work starts. + +## The migration track, in outline + +Data movement is the easy half, and deliberately reversible. + +1. **Stand the replacement up beside the incumbent**, on its own ports and its own provision type. + No downtime, nothing removed. +2. **Copy bucket by bucket with a neutral tool.** `rclone` rather than the incumbent's own client + — the client has been withdrawn upstream too, so building the migration on it would inherit + the same dependency this effort exists to remove. +3. **Verify per bucket** — object counts and checksums, not a transfer exit code. +4. **Repoint consumers through the connection the module already publishes.** Consumers read an + API URL from the module's declared connections rather than addressing the store directly, so + the cutover surface is that value plus the provisioning and tool handlers. +5. **Freeze writes, final incremental sync, flip**, and keep the incumbent read-only as the + rollback until confidence is earned. +6. **Retire**, and only then remove the module. + +The genuinely new work is not the copy. It is the **provisioning handler** and the **tool +handlers**, which are written against MinIO's admin API, and the OIDC wiring. + +## Open questions + +- How much of the OIDC requirement survives, and in which build? See above — this gates the + choice. +- Does the mesh's bucket provision translate to the candidate's identity model without weakening + what ADR 0049 says about a consumer's identity fitting the tightest backend? +- Should this effort also answer issue 113's general question — mirroring third-party images into + the mesh's own registry — or is that a separate decision? Replacing one withdrawn product with + another unmirrored upstream leaves the same one-way door in place, just further from the hinge. +- Is the four-node erasure-coded topology still warranted, or was it inherited? Worth re-asking + while the product is being chosen, rather than reproducing a shape by default. diff --git a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md new file mode 100644 index 0000000..7573c96 --- /dev/null +++ b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md @@ -0,0 +1,117 @@ +--- +status: located +opened: 2026-09-24 +located-in: [hal modules/minio] +fixed-by: +amended-design: +--- + +# 113 — The object store's images were withdrawn upstream, and only a node that already holds them can still run it + +## What was observed + +On 2026-09-24, during a service-by-service cutover, the object-store module could not be built on +a node that did not already hold its images. Both images the module needs answer an anonymous +pull with `401 UNAUTHORIZED`: + +``` +/minio/minio 401 /minio/operator 200 +/minio/mc 401 /minio/console 200 +``` + +The module pins a **tag**, not a digest, and the default is four and a half years old: + +``` +image: /minio/minio:${MINIO_VERSION:-RELEASE.2022-01-07T01-53-23Z} +``` + +The cause is upstream and outside the mesh: the vendor **deleted** the community server and client +repositories. It is not an access policy that a credential could answer, and nothing about the +mesh's own registry configuration, resolver or trust settings is involved. + +- The vendor removed both repositories from the main public registry on **2026-09-11**. Its API + answers `404` for the server repository while a sibling in the same namespace answers `200`. +- The secondary registry that the wider ecosystem repointed to as a stopgap **no longer lists + them either**. Sixty-eight repositories in that namespace are still public and pull normally; + the server and the client are simply absent, and the namespace is now dominated by the vendor's + commercially licensed line. +- The open-source repository was archived in **February 2026**, and the community edition has been + source-only since **October 2025**. No new images are published anywhere. + +## What did not happen, and why it is recorded + +The first reading of this was that the registry had *disabled anonymous pulls for the whole +vendor namespace*. That was wrong in a way worth keeping, because the evidence looked conclusive: + +- The anonymous token carries `"actions": []` for the affected repositories and `['pull']` for + working ones — a real signal, but it is **also exactly what a repository that does not exist + returns**. A deliberately invented repository name in the same namespace produced a + byte-identical response. The signal cannot distinguish *revoked* from *absent*. +- The token also carries `"$disabled"`, which was read as confirmation. It appears on **every** + repository on that registry, including the ones pulling successfully. It describes image + **signing**, not access. + +Sibling repositories in the same namespace pulling normally is what rules out a namespace-wide +policy, and the registries' own APIs — `404` against `200` — are what establish deletion. + +## The mesh was not blocked, which is the other half + +A node that already holds the images runs the module normally. The node carrying the cutover holds +the pinned server image, the client, and the load-balancer image the module composes with, all +pulled years ago. Its resolved version variable matches the cached tag exactly. + +This is by design and not by luck. The deploy stage pulls **best-effort** and then asserts only +that every image the composition declares **resolves locally**, precisely so that an image which +exists on the node but can no longer be fetched does not fail a deploy. The code comment naming +the precedent describes this case exactly — *"an old tag pulled years ago and since removed +upstream"* — and records that failing on the pull instead had previously made a module +undeployable while all of its images sat on the node. + +So the deploy logs a warning and succeeds. Dropping the module from the cutover queue was not +necessary. + +## Why it matters beyond this instance + +The instance is harmless; the standing condition is not. + +1. **No new node can ever provision this module.** Every node that does not already hold the + images is permanently unable to obtain them, and the same will be true of any module whose + upstream withdraws an image. + + This is the failure mode of a deliberate design choice, which is why it is worth recording + rather than patching. The foundation design chooses **references over payload** — *"the bundle + names images by digest and the host fetches them"* — on the stated grounds that + *"reproducibility comes from pinning the identity of a thing rather than carrying its bytes"* + ([to-be 07](../../03-DESIGN/01-to-be/07-the-foundation.md)). That reasoning is sound. It holds + only while a pinned identity stays **resolvable**, and nothing in the mesh's control guarantees + that for an image in somebody else's registry. The passage is about + the foundation bundle, and this module is not in it; but the pattern — pin the identity, fetch + the bytes on demand — is how every module gets its third-party images, so the exposure is + general even though the sentence is local. +2. **The pinned release is permanently unpatched.** It is four and a half years old, upstream is + archived, and no security fix will ever reach it. +3. **Nothing detects this class of failure.** The condition is invisible until a node without the + image tries to deploy. The very guard that correctly stops this from breaking existing nodes — + assert local resolvability, not the pull — also means a warning is the only trace, and a + warning is not a rule. A mesh cannot state that its modules are installable while the only + evidence is that they are already installed. + +The third point is the general one, and it is not specific to this vendor: an image pinned by tag +against a registry the mesh does not control is a dependency with no guarantee behind it, and the +mesh currently learns it has lost one only by trying to use it. + +## Open questions + +- Should the mesh **hold** the images it depends on — mirroring third-party images into its own + registry at adoption, so a module's installability does not depend on an upstream's continued + goodwill? That is the fix that generalises. It costs storage and a policy about what to mirror, + and it is a deliberate move **away** from references-over-payload for third-party images + specifically — so it should be decided as such, not smuggled in as a fix. +- Should a module's images be pinned **by digest** rather than by tag? It makes the artifact + exact and auditable, but does nothing about withdrawal — a deleted digest is just as gone. +- What **checks** that every module in the catalogue is still obtainable from a node that holds + nothing? Nothing does today. A periodic cold-pull of the catalogue would have caught this on + 2026-09-11 rather than thirteen days later, mid-cutover. +- For this module specifically: replace the product. The design already says the dependency is on + **S3 the protocol, not the product** — see + [research 015](../../01-RESEARCH/015-the-object-store-after-minio/00-overview.md). diff --git a/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md new file mode 100644 index 0000000..77f1cdf --- /dev/null +++ b/04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/01-diagnosis.md @@ -0,0 +1,110 @@ +# 113 — Diagnosis + +## 2026-09-24 — the trail + +The symptom arrived already carrying a diagnosis: *the registry has disabled anonymous pulls for +the entire vendor namespace.* Everything below was an attempt to confirm that, and it did not +survive. + +### Step 1 — the token is not the test + +The reported evidence was the anonymous pull token's contents: `"actions": []` and `"$disabled"`. +A token is an intermediate artifact. The test is whether a manifest can actually be fetched with +it, so the first step was to request one: + +``` +GET /v2/minio/minio/manifests/ -> 401 UNAUTHORIZED +``` + +That confirmed the failure but said nothing about its scope or cause. + +### Step 2 — a control ruled out the stated cause + +The same request flow, in the same minute, against other repositories: + +| repository | token actions | manifest | +|---|---|---| +| the vendor's server | `[]` | **401** | +| the vendor's client | `[]` | **401** | +| the vendor's operator | `['pull']` | 200 | +| the vendor's console | `['pull']` | 200 | +| the vendor's sidecar proxy | `['pull']` | 200 | +| an unrelated public project | `['pull']` | 200 | + +**A namespace-wide policy is ruled out.** Two repositories fail; their siblings in the same +namespace pull normally. + +### Step 3 — both pieces of the original evidence were red herrings + +- `"$disabled"` is present on **every** repository on that registry, including all of the + successful ones above. It belongs to the signing context, not to authorisation. It carries no + information about this failure at all. +- `"actions": []` with a `401` is **indistinguishable from a repository that does not exist.** A + deliberately invented repository name in the vendor's namespace returned a byte-identical + response — empty actions, `401`. The signal cannot separate *access revoked* from *not there*, + so it cannot support the conclusion it was used for. + +That second point turned the question from *who revoked access* to *is it still there*. + +### Step 4 — the registries' own APIs establish deletion + +Asked directly, rather than through the pull path: + +- **Primary registry:** its repository API answers **404** for the server repository, and **200** + for a sibling in the same namespace. The repository is gone, not private. +- **Secondary registry:** a listing of the vendor's namespace returns **68 public repositories**. + The server and the client are **absent from the list**. Present are the operator, console, + sidecar proxy, benchmarking and key-management images — and a large, newer set belonging to the + vendor's commercially licensed line. + +### Step 5 — upstream confirms, and dates it + +The vendor deleted the community server and client from the primary registry on **2026-09-11**. +This was the last step of a staged withdrawal: free image publishing stopped in **October 2025**, +the community console UI was removed mid-2025, and the open-source repository was archived in +**February 2026**. The secondary registry was where the ecosystem repointed as a stopgap; it has +since lost the two repositories as well. + +**Conclusion: the images were withdrawn, not restricted.** No credential can answer this, because +there is nothing left to authenticate against. A different image source is the only remedy. + +## Why the mesh kept working, checked rather than assumed + +The claim that the module "genuinely isn't ready for cutover" was tested and is false for the node +in question. + +- The node **holds** the pinned server image, the client, and the load-balancer image the module + composes with — all pulled years before the withdrawal. +- The module's resolved version variable on that node matches the cached tag **exactly**, so the + composition references an image that is present. +- The composition declares **no pull policy**, so a present tag is used as-is. +- The deploy stage pulls best-effort, then asserts only that every declared image resolves + locally. A pull error becomes a logged warning when all images are present. +- The start path restarts the unit and does not pull. The one `--pull always` in the tree sits + inside a generated guide describing the **superseded** approach, not in the code that writes + units. + +So a deploy of this module on that node succeeds today. + +## Claims in the original report that could not be substantiated + +Recorded because they were specific and load-bearing, and acting on them would have wasted time. + +| Claim | Finding | +|---|---| +| The module's manifest is a `module.json`, pinning the server image by digest at line 83 | There is **no `module.json` anywhere** in the monorepo. The manifest is YAML and pins a **tag**. No digest pin exists. | +| The module's own runtime artifact is a placeholder with an all-zero digest | **Zero occurrences** of that image name or of an all-zero digest anywhere in the tree. The module declares no runtime or sidecar artifact. | +| A dated readiness document records six modules with placeholder digests | **No such file exists.** | +| The server image is cached locally at an older release than the module pins | The cached tag is **exactly** the pinned one, not an older release. | + +None of these change the real finding, which stands: the images are gone upstream. + +## What is located, and what is not + +**Located:** the module in the monorepo's catalogue — it pins, by tag, an image that no longer +exists anywhere public. + +**Not located, and deliberately left open:** the general condition. The mesh has no mirror of the +third-party images its modules depend on and no check that a module is obtainable by a node +holding nothing. That is a design gap rather than a defect in this module, and it is stated as an +open question on the report rather than answered here.