novox/mesh-catalog
3
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

minio: the real 4-node/8-drive erasure-coded cluster, both public routes, verified live on novox #58

Merged
jschoubben merged 6 commits from feat/minio-real-cluster-not-single-node into main 2026-09-26 13:05:58 +00:00
Conversation 2 Commits 6 Files Changed 1
+4 -3
jschoubben commented 2026-09-24 18:35:21 +00:00
Owner
Copy Link
Copy Source

Builds on the image-repin/data-path-safety commit already on this branch. Full summary of what's in this PR:

  • Rewrote the module as HAL's actual topology — 4 nodes (minio1-minio4), 2 drives each, nginx LB — not the single-container simplification the first pass had. Only the two images that had to change did (dead minio upstream → pgsty fork; EOL nginx 1.19.2 → current stable-alpine).
  • Data lands on /var/lib/minio-store (8 fresh subdirectories), never HAL's live /services/minio/data/*.
  • Both public routes wired via the new multi-route mechanism (mesh-controller#55): files-api.novox.be (S3, port 9000) and files.novox.be (console, port 9001) — same two names HAL routes today.
  • Network resource renamed minio → minio-net to avoid colliding with the LB container's own name (surfaced a real mesh-host bug, fixed separately in mesh-host#25 — this rename sidesteps it without needing that fix deployed).

Verified live on novox, not just built: all 4 cluster nodes healthy, LB responding 200 on /minio/health/cluster and /minio/health/live through the actual S3 hostname, mesh runtime sidecar up and serving tools, listBuckets() via the real client code returns [] — empty, confirming nothing has touched HAL's data.

Also required issuing minio's broker account (module issue minio --node novox) — not part of this PR, a one-time mesh-side action already done.

Builds on the image-repin/data-path-safety commit already on this branch. Full summary of what's in this PR: - Rewrote the module as HAL's actual topology — 4 nodes (`minio1`-`minio4`), 2 drives each, nginx LB — not the single-container simplification the first pass had. Only the two images that had to change did (dead minio upstream → `pgsty` fork; EOL nginx 1.19.2 → current `stable-alpine`). - Data lands on `/var/lib/minio-store` (8 fresh subdirectories), never HAL's live `/services/minio/data/*`. - Both public routes wired via the new multi-route mechanism (mesh-controller#55): `files-api.novox.be` (S3, port 9000) and `files.novox.be` (console, port 9001) — same two names HAL routes today. - Network resource renamed `minio` → `minio-net` to avoid colliding with the LB container's own name (surfaced a real mesh-host bug, fixed separately in mesh-host#25 — this rename sidesteps it without needing that fix deployed). Verified live on novox, not just built: all 4 cluster nodes healthy, LB responding 200 on `/minio/health/cluster` and `/minio/health/live` through the actual S3 hostname, mesh runtime sidecar up and serving tools, `listBuckets()` via the real client code returns `[]` — empty, confirming nothing has touched HAL's data. Also required issuing minio's broker account (`module issue minio --node novox`) — not part of this PR, a one-time mesh-side action already done.
jschoubben added 3 commits 2026-09-24 18:35:22 +00:00
minio: run the real 4-node/8-drive erasure-coded cluster, not a single container 945390e59a 
The single standalone instance from the first pass didn't match HAL's actual
topology: HAL runs minio1-4, two drives each, behind an nginx load balancer
on 9000 (S3) and 9001 (console). This rewrite mirrors that exactly — same
node count, same erasure-coding command, same LB config — so the migration
is a real like-for-like move, not a simplification.

Only the two images that had to change did: the minio server (dead upstream,
already fixed in the prior commit) and nginx (1.19.2-alpine is long EOL;
repinned to current stable-alpine by digest). Data still lands on a fresh,
empty, mesh-owned path, never HAL's live drives.

The OIDC-wait entrypoint wrapper HAL used is dropped: it's a no-op when
MINIO_IDENTITY_OPENID_CONFIG_URL is unset (it always is here — no OIDC
integration was ever wired to minio itself), and this catalogue has no
container resource field for overriding a container's entrypoint anyway —
every converted module relies on the image's own entrypoint plus args,
which is exactly what the original single-node version already did.
minio: publish both public routes now that a module can answer route twice 973d80aaa2 
files-api.novox.be (port 9000, the S3 data API) and files.novox.be (port
9001, the console) — same two names HAL routes today, via nginx's own
upstream split. Needed mesh-controller#55 (a module answering one
requirement several times) to exist first; it's merged and deployed.
minio: name the network minio-net, not minio 6a6dd4a7dc 
Collided with the LB container's own name. docker inspect minio resolved
to the network instead of the (not-yet-created) container, and mesh-host's
existence check crashed on the mismatched shape rather than reporting
absence -- a real mesh-host bug (fixed separately, mesh-host#25), but this
sidesteps it here without waiting on a host-level binary update.
jschoubben added 1 commit 2026-09-24 21:08:30 +00:00
minio: revert to single-node after measuring the real cost of sharding 20df40c949 
The 4-node/8-drive erasure-coded cluster matched HAL's topology faithfully,
but real throughput testing against both showed why that costs more than
it's worth here: every write on the sharded cluster fans out across 4
processes over the internal network with erasure-coding overhead, capping
safe throughput around 1.3-2 MiB/s and breaking outright above ~256
concurrent transfers (IncompleteBody errors, confirmed via a controlled
512x test). The identical copy against a single-node instance sustained
23+ MiB/s at the same concurrency with zero errors — over 10x faster,
verified side-by-side, not assumed.

Trades away erasure-coded redundancy (no single-drive fault tolerance) for
that throughput. Deliberate, and reversible if it turns out to matter later
-- the data itself is migrated over the S3 API either way, so the storage
topology underneath isn't locked in by anything upstream of it.
jschoubben added 1 commit 2026-09-25 08:45:07 +00:00
minio: set the region to eu-west, matching where this mesh actually runs 440e3e446e 
Left at MinIO's us-east-1 default. Novox is hosted in Germany, the team
is in Belgium -- eu-west is correct, and matters beyond labeling: it's
part of the SigV4 signature, so a client using the wrong region fails
auth even with valid credentials. Set on the server (MINIO_REGION),
the served provision value, and the runtime sidecar's own client.
jschoubben commented 2026-09-26 12:59:17 +00:00
Author
Owner
Copy Link
Copy Source

Reviewed against main (82 commits landed since; this one conflicts, unlike the others).

The title no longer describes the change. The branch's own history ends with 20df40c — minio: revert to single-node after measuring the real cost of sharding, so the 4-node/8-drive cluster is not what would land. What remains is: both public routes published now that a module can answer route twice, the network renamed minio-net, the region set to eu-west, the console port declared in listens, and a browser-redirect URL.

The network rename is the strongest part and worth keeping for a reason the title does not give: a network and a container sharing the bare name minio is the ambiguity that mesh-host #25 fixed on the other side ("a same-named network stops a container from ever being found"). Renaming the network removes the collision at the source.

Two things to settle before it merges:

  1. The conflict is in modules/minio/module.json, where main has the repin and data move. Resolving it is mechanical except for one line — MINIO_BROWSER_REDIRECT_URL: "https://<label>.<public-domain>", which is on this branch and not on main. That literal is the same workaround as #55's KC_HOSTNAME, now recorded as hq issue 122: no interpolation yields a public name, so the console's absolute redirect has nowhere else to come from. Merging it is a stopgap; dropping it leaves the console redirecting wrongly.
  2. region: "eu-west" replaces us-east-1. Arbitrary for this server, but it is a value every client must match, and it is also a fact about where one mesh runs rather than about the module. Worth a sentence in the PR saying which clients were checked against it.

Happy to resolve the conflict and merge once you say which way the redirect URL goes — the same call as #55 and #59.

Reviewed against `main` (82 commits landed since; this one **conflicts**, unlike the others). **The title no longer describes the change.** The branch's own history ends with `20df40c — minio: revert to single-node after measuring the real cost of sharding`, so the 4-node/8-drive cluster is not what would land. What remains is: both public routes published now that a module can answer `route` twice, the network renamed `minio-net`, the region set to `eu-west`, the console port declared in `listens`, and a browser-redirect URL. **The network rename is the strongest part** and worth keeping for a reason the title does not give: a network and a container sharing the bare name `minio` is the ambiguity that mesh-host #25 fixed on the other side ("a same-named network stops a container from ever being found"). Renaming the network removes the collision at the source. **Two things to settle before it merges:** 1. **The conflict** is in `modules/minio/module.json`, where `main` has the repin and data move. Resolving it is mechanical except for one line — `MINIO_BROWSER_REDIRECT_URL: "https://<label>.<public-domain>"`, which is on this branch and not on `main`. That literal is the same workaround as #55's `KC_HOSTNAME`, now recorded as **hq issue 122**: no interpolation yields a public name, so the console's absolute redirect has nowhere else to come from. Merging it is a stopgap; dropping it leaves the console redirecting wrongly. 2. **`region: "eu-west"`** replaces `us-east-1`. Arbitrary for this server, but it is a value every client must match, and it is also a fact about where one mesh runs rather than about the module. Worth a sentence in the PR saying which clients were checked against it. Happy to resolve the conflict and merge once you say which way the redirect URL goes — the same call as #55 and #59.
jschoubben added 1 commit 2026-09-26 13:05:42 +00:00
Merge main afe8aae826 
# Conflicts:
#	modules/minio/module.json
jschoubben commented 2026-09-26 13:05:51 +00:00
Author
Owner
Copy Link
Copy Source

Conflict resolved and merging. Two decisions in the resolution:

  • Network stays renamed minio-net, in main's expanded formatting — the rename is the point, the compact arrays were incidental.
  • MINIO_BROWSER_REDIRECT_URL kept. I checked whether this would be a new precedent and it is not: main already carries literal public names in five modules — n8n has N8N_HOST, N8N_PROTOCOL and a literal WEBHOOK_URL, and both route-proxy and builder carry a full clone URL. Holding this one line would leave the console redirecting wrongly while changing nothing about the five already there. hq issue 122 carries the mechanism that removes all of them together.

Verified: the manifest parses through the controller's own ParseManifest, and internal/catalogue, internal/inventory and both commands' tests pass against the catalogue as changed.

Worth noting for the record, since the title will outlive it: what lands here is not a 4-node cluster. This branch's own 20df40c reverted to single-node after measuring the cost of sharding. What lands is both public routes, the network rename, region: eu-west, the console port declared in listens, and the redirect URL.

Conflict resolved and merging. Two decisions in the resolution: - **Network stays renamed** `minio-net`, in `main`'s expanded formatting — the rename is the point, the compact arrays were incidental. - **`MINIO_BROWSER_REDIRECT_URL` kept.** I checked whether this would be a new precedent and it is not: `main` already carries literal public names in five modules — `n8n` has `N8N_HOST`, `N8N_PROTOCOL` and a literal `WEBHOOK_URL`, and both `route-proxy` and `builder` carry a full clone URL. Holding this one line would leave the console redirecting wrongly while changing nothing about the five already there. **hq issue 122** carries the mechanism that removes all of them together. Verified: the manifest parses through the controller's own `ParseManifest`, and `internal/catalogue`, `internal/inventory` and both commands' tests pass against the catalogue as changed. Worth noting for the record, since the title will outlive it: what lands here is **not** a 4-node cluster. This branch's own `20df40c` reverted to single-node after measuring the cost of sharding. What lands is both public routes, the network rename, `region: eu-west`, the console port declared in `listens`, and the redirect URL.
jschoubben merged commit 4d715f8b73 into main 2026-09-26 13:05:58 +00:00
jschoubben deleted branch feat/minio-real-cluster-not-single-node 2026-09-26 13:05:59 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jschoubben miel.truyen
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: novox/mesh-catalog#58