Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 jschoubben
|
a97feeefe5 | ||
|
jschoubben
|
a18550e460 |
@@ -1,78 +0,0 @@
|
||||
---
|
||||
name: hq-defer
|
||||
description: Use when a thought is raised that should be remembered but NOT worked on now — an aside during other work, a "we should look at X someday", a known gap nobody is assigning yet. Triggers on "defer this", "park this", "register this thought", "note this for later", "don't work on it, just remember it". Records and returns to whatever was already in progress.
|
||||
---
|
||||
|
||||
# hq-defer
|
||||
|
||||
Parks a thought so it is not lost, **without moving the work off course.** The defer is the
|
||||
point: the thought is recorded and the previous task resumes.
|
||||
|
||||
This skill wraps no playbook, because deferring is not part of the development cycle — it is
|
||||
what happens *before* something enters it. A parked thought has no number, no owner and no
|
||||
status, and that is correct.
|
||||
|
||||
## Where it goes, and why not the repository
|
||||
|
||||
Record it as **one memory file** in Claude's persistent memory directory for this project (the
|
||||
path is given in the session's memory instructions), with `metadata.type: project`, plus a
|
||||
one-line pointer in `MEMORY.md`.
|
||||
|
||||
**Not** in `04-ISSUES`, `01-RESEARCH` or anywhere else in the repository:
|
||||
|
||||
- A parked thought is not an issue or a research effort. Giving it a number asserts it has been
|
||||
triaged, which is exactly what deferring says has not happened.
|
||||
- A shared "deferred" or "someday" document is a **central status file**, which
|
||||
[`AGENTS.md`](../../../AGENTS.md) forbids. Status lives in frontmatter on real records, and a
|
||||
parked thought has no real record yet.
|
||||
- A repository write means a branch, a commit and a pull request — drift, which is the one thing
|
||||
this skill exists to avoid.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Write the memory file. Slug is kebab-case and descriptive of the thought, not of the act of
|
||||
deferring.
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: <kebab-slug>
|
||||
description: Deferred note — <one line>
|
||||
metadata:
|
||||
type: project
|
||||
---
|
||||
|
||||
Raised and deliberately deferred on YYYY-MM-DD: **<the thought, in the user's own terms>**
|
||||
|
||||
**Why:** what was being worked on when it came up, and that deferring was intentional so
|
||||
that work was not pulled off course.
|
||||
|
||||
**How to apply:** treat as an open thread, not an assignment. Do not start on it
|
||||
unprompted. If it graduates it needs an HQ home first — an issue under playbook
|
||||
[03](../../../00-META/process/03-issues.md) if a stated behaviour does not happen, or
|
||||
research under playbook [01](../../../00-META/process/01-research.md) if it is still an
|
||||
idea. Say which is undetermined, if it is.
|
||||
```
|
||||
|
||||
2. Append one line to `MEMORY.md`: `- [<Title>](<kebab-slug>.md) — deferred YYYY-MM-DD; parked, no HQ
|
||||
record, do not start unprompted`.
|
||||
3. Convert relative dates to absolute before writing. "Last week" is worthless in six months.
|
||||
4. Check for an existing memory covering the same thought and update it instead of adding a
|
||||
duplicate.
|
||||
|
||||
## Then stop
|
||||
|
||||
Reply in **at most two lines** — what was recorded, and that it is parked — and **return to
|
||||
whatever was in progress before.** Do not summarise the parked thought back at length, do not
|
||||
propose a plan for it, do not ask which playbook it belongs to, and do not open anything.
|
||||
|
||||
If nothing was in progress, say only that it is recorded.
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not create an issue, a research effort, a decision record or a design document.
|
||||
- Do not create a branch, commit or pull request.
|
||||
- Do not start investigating the thought, however cheap the first check looks.
|
||||
- Do not name nodes, domains, addresses, absolute paths or usernames in the memory file — the
|
||||
thought may later be quoted into this repository, which is public.
|
||||
- Do not decide whether it is an issue or research when the evidence does not say. Recording
|
||||
"undetermined" is the honest outcome and costs nothing later.
|
||||
@@ -50,9 +50,7 @@ consumers across the catalogue — not from assumption.
|
||||
| Requirement | Evidence in the module today |
|
||||
|---|---|
|
||||
| S3 API | The protocol every consumer speaks; already the design's stated dependency. |
|
||||
| ~~OIDC login against the mesh's identity provider~~ | **Struck 2026-09-24. Not a requirement, and it never worked.** Six variables are wired and an entrypoint blocks on the provider, which reads as a live feature. The module's own hook comment records the end state as *"policy claim missing"* — a failing login. See [01](01-candidate-comparison.md). |
|
||||
| **Per-application access keys, each scoped to a bucket** | The real requirement. A "user" of the store is normally an application; the mesh already mints a credential per provisioned bucket. |
|
||||
| **One live consumer using it as opaque primary storage** | A file-sync application, since early 2023: objects named by internal id, metadata in its own database. Highest-risk consumer — a live copy drifts, and its bucket name must be preserved. |
|
||||
| **OIDC login against the mesh's identity provider** | Six configuration variables are wired and populated in practice — discovery URL, client id, client secret, scopes, display name, redirect — plus a dedicated entrypoint script that blocks startup until the provider answers. This is live, not aspirational. |
|
||||
| Erasure-coded multi-node topology | Four server nodes with two data directories each, behind a load balancer. |
|
||||
| A single-node form | Declared as a flavour, for development and small nodes. |
|
||||
| Buckets as a typed provision | The module declares a provision type of `bucket` on a named network; the mesh mints the credential and the provider creates it (ADRs 0048, 0084). |
|
||||
@@ -66,49 +64,34 @@ OIDC story, not spread across the catalogue.
|
||||
|
||||
## Candidates
|
||||
|
||||
**Four candidates, not three.** The comparison was briefly narrowed to SeaweedFS on the strength of
|
||||
console single sign-on; that axis turned out not to be a requirement, and the incumbent's own
|
||||
maintained fork had been omitted altogether. Both errors, and why they happened, are recorded in
|
||||
[01 — the candidates measured](01-candidate-comparison.md), which carries the evidence and the
|
||||
requirement-by-requirement detail.
|
||||
Scoped to **SeaweedFS** as the primary, with the others recorded so the rejection is not
|
||||
rediscovered.
|
||||
|
||||
In short, and only in short:
|
||||
- **SeaweedFS** — Apache-2.0, Go, twelve-plus years of development, erasure coding, and OIDC
|
||||
support in its S3/STS layer. Chosen to scope because it is the only candidate that plausibly
|
||||
preserves the OIDC requirement above, which is the one requirement that is live and least
|
||||
substitutable.
|
||||
- **Garage** — the lightest to operate and the simplest model, but **no native identity-provider
|
||||
integration**. Adopting it means losing OIDC console login or fronting it with a proxy. A real
|
||||
functional regression against something currently in use.
|
||||
- **RustFS** — markets itself as a binary-level drop-in retaining existing data, buckets and
|
||||
configuration, which would make the data migration close to trivial. Young, and that claim is
|
||||
exactly the kind that must be verified on a copy before it is believed.
|
||||
- **Ceph RGW** — the most capable and the most operationally expensive; disproportionate to a mesh
|
||||
where the object store is an ordinary module, not a platform.
|
||||
|
||||
- **The maintained fork of the incumbent** — the community edition was archived and its images
|
||||
deleted, but a fork publishes, tracks CVEs, and preserves the on-disk format, S3 API and
|
||||
environment surface. Costs **an image reference** where every other option costs a data
|
||||
migration, two rewrites and a maintenance window. Does not end the dependence on an abandoned
|
||||
codebase; buys time to choose deliberately.
|
||||
- **Garage** — its permission model *is* the requirement (per access key, per bucket), its admin
|
||||
API is the closest match to how the mesh provisions, and the highest-risk consumer is
|
||||
first-party documented against it. Remaining cost: no object versioning, no server-side
|
||||
encryption or object locking, partial lifecycle — **unmeasured against the ten buckets, and the
|
||||
one thing that could still disqualify it**.
|
||||
- **SeaweedFS** — longest field record and erasure coding. Its console sign-on is a paid feature,
|
||||
which is now beside the point. What weighs against it is narrower: its S3 surface is a gateway
|
||||
translating onto its own file-system API, with no first-party support for the opaque consumer.
|
||||
- **RustFS** — closest in shape to the incumbent, so the least porting. But it reached general
|
||||
availability eight days before this was written, and carries an open defect in the credential
|
||||
path. Two earlier claims about it are corrected in 01: it is **not** a drop-in that retains
|
||||
existing data.
|
||||
- **Ceph RGW** — remains rejected as disproportionate where the object store is an ordinary
|
||||
module rather than a platform.
|
||||
|
||||
**This is now two decisions, not one:** whether to repoint to the fork or migrate, and — if
|
||||
migrating — to which. Repointing does not foreclose migrating, which is the argument for taking it
|
||||
first. On the corrected requirement the migration ranking is Garage, then SeaweedFS, and not yet
|
||||
RustFS. Two measurements gate any graduation: **which S3 endpoints the consumers actually call**
|
||||
(Garage cannot be ranked fairly until counted), and **whether the fork can read the incumbent's
|
||||
on-disk format in place** — tested on a copy, because the migration between them is one-way. Both
|
||||
are in [01](01-candidate-comparison.md#what-is-still-unmeasured).
|
||||
**The first thing to verify, because the choice turns on it:** how much of SeaweedFS's OIDC story
|
||||
is in the freely licensed build, and whether its shape — IAM/STS token exchange — can actually
|
||||
stand in for a console that redirects a human to an identity provider. If it cannot, the honest
|
||||
finding may be that **no** candidate preserves the current feature set, and the decision becomes
|
||||
which regression to accept. That question is worth answering before any migration work starts.
|
||||
|
||||
## The migration track, in outline
|
||||
|
||||
Data movement is the easy half, and deliberately reversible.
|
||||
|
||||
1. **Stand the replacement up beside the incumbent**, on its own ports, its own provision type and
|
||||
**its own data directory**. Nothing removed. The data directory matters: reusing one the
|
||||
incumbent already holds would put a fresh single-drive store on top of a live erasure set.
|
||||
1. **Stand the replacement up beside the incumbent**, on its own ports and its own provision type.
|
||||
No downtime, nothing removed.
|
||||
2. **Copy bucket by bucket with a neutral tool.** `rclone` rather than the incumbent's own client
|
||||
— the client has been withdrawn upstream too, so building the migration on it would inherit
|
||||
the same dependency this effort exists to remove.
|
||||
@@ -117,22 +100,16 @@ Data movement is the easy half, and deliberately reversible.
|
||||
API URL from the module's declared connections rather than addressing the store directly, so
|
||||
the cutover surface is that value plus the provisioning and tool handlers.
|
||||
5. **Freeze writes, final incremental sync, flip**, and keep the incumbent read-only as the
|
||||
rollback until confidence is earned. For the opaque consumer this is **not optional and not
|
||||
instant**: it stores objects by internal id with metadata in its own database, so a copy taken
|
||||
while it runs will drift. It needs a maintenance window for the final sync, and the window is
|
||||
proportional to 82,496 objects rather than to 230 GiB.
|
||||
rollback until confidence is earned.
|
||||
6. **Retire**, and only then remove the module.
|
||||
|
||||
The genuinely new work is not the copy. It is the **provisioning handler** and the **tool
|
||||
handlers**, both written against the incumbent's admin API. *The OIDC wiring was previously listed
|
||||
here and is struck: it is not a requirement and it never worked.*
|
||||
handlers**, which are written against MinIO's admin API, and the OIDC wiring.
|
||||
|
||||
## Open questions
|
||||
|
||||
- ~~How much of the OIDC requirement survives, and in which build?~~ **Answered, and it was the
|
||||
wrong question.** The console requirement does not exist, and the login it referred to never
|
||||
worked. What replaced it: which S3 endpoints consumers actually call, and whether the fork reads
|
||||
the incumbent's format in place.
|
||||
- How much of the OIDC requirement survives, and in which build? See above — this gates the
|
||||
choice.
|
||||
- Does the mesh's bucket provision translate to the candidate's identity model without weakening
|
||||
what ADR 0049 says about a consumer's identity fitting the tightest backend?
|
||||
- Should this effort also answer issue 113's general question — mirroring third-party images into
|
||||
|
||||
@@ -1,176 +0,0 @@
|
||||
# 015 / 01 — The candidates measured
|
||||
|
||||
*Rewritten 2026-09-24. An earlier version of this document ranked the candidates on whether they
|
||||
preserved single-sign-on to the object store's **console**. That was the wrong axis — it is not a
|
||||
requirement — and a fourth candidate was missing entirely. Both errors are recorded at the end,
|
||||
because how a comparison came to be ranked on the wrong thing is worth more than the ranking was.*
|
||||
|
||||
## The requirement, corrected
|
||||
|
||||
Taken from the operator and from the running system, not from the module's shape.
|
||||
|
||||
**A "user" of the object store is normally an application.** The requirement is therefore
|
||||
**per-application access keys, each scoped to its own bucket** — not per-human single sign-on. The
|
||||
mesh already works this way: it mints a credential for every provisioned bucket, and the consumer
|
||||
reads an endpoint from the module's declared connection rather than addressing the store directly.
|
||||
|
||||
**The console is not a requirement.** It was the axis the previous version ranked on, and it should
|
||||
not have been.
|
||||
|
||||
**The identity-provider login never worked.** The predecessor's module wires six OIDC variables and
|
||||
blocks startup until the provider answers, which reads like a working feature. It is not: the
|
||||
module's own hook comment records the end state as *"policy claim missing"* — a **failing** login,
|
||||
written up as progress because it proved the provider had registered. The identity provider emits no
|
||||
such claim, nothing in the module creates the mapper, and the configured scope alone would not carry
|
||||
a custom one. Two days of logs show no genuine login attempts, only internet scanners failing on an
|
||||
STS API version. **Nothing should be carried forward on the assumption this works**, and no
|
||||
candidate should be credited or penalised for matching it.
|
||||
|
||||
**One consumer is live, opaque, and holds real user files.** A file-sync application has used the
|
||||
store as its **primary storage** since early 2023: objects named by an internal id, with all
|
||||
metadata in its own database. Three consequences — a copy taken while it runs will drift, its bucket
|
||||
name must be preserved or its database references break, and it is the highest-risk consumer of the
|
||||
lot.
|
||||
|
||||
## What is actually stored, measured
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Logical | **230 GiB, 82,496 objects, 10 buckets** |
|
||||
| Raw on disk | **468 GiB** — eight drive directories at 59 GiB each |
|
||||
| Implied scheme | 468 ÷ 230 = **2.03×**, confirming erasure coding at half parity |
|
||||
| Headroom | ~1.3 TiB free on the filesystem holding it |
|
||||
|
||||
**All eight "drives" are directories on one filesystem on one machine.** The erasure coding is
|
||||
therefore not buying independent-drive redundancy; the real failure domain is the array underneath,
|
||||
which has its own. This single fact decides more of the comparison than any product feature: a
|
||||
scheme's redundancy model is close to irrelevant here, and what remains is its storage overhead.
|
||||
|
||||
At 230 GiB with 1.3 TiB free, **storage overhead is not a deciding cost either.** Replication at
|
||||
three copies would run ~690 GiB against the present 468 GiB — about **+222 GiB**, comfortably
|
||||
absorbed. Erasure coding at a wider stripe would *save* roughly 146 GiB. Both are rounding errors
|
||||
against the headroom, and neither should decide this.
|
||||
|
||||
## The candidates
|
||||
|
||||
Four, not three. The previous version omitted the first.
|
||||
|
||||
### The maintained fork of the incumbent
|
||||
|
||||
The community edition was archived upstream and its images deleted
|
||||
([issue 113](../../04-ISSUES/113-the-object-stores-images-were-withdrawn-upstream/00-report.md)),
|
||||
but **a fork is maintained and publishing** — `pgsty/minio`, from the Pigsty project. It restores
|
||||
the console stripped from the community
|
||||
build, rebuilt image and package distribution, tracks CVEs, and states that it preserves the on-disk
|
||||
format, the S3 API and the environment-variable surface. Verified by pulling it: it reports a
|
||||
current release, permissive-to-copyleft licensing unchanged from upstream, and identifies itself as
|
||||
a community fork. Adoption is real — the server image has been pulled three quarters of a million
|
||||
times.
|
||||
|
||||
**Why it reorders the comparison.** Every other candidate costs a data migration, a provisioning
|
||||
handler rewritten against a different admin API, a tool surface ported, and a maintenance window for
|
||||
the opaque consumer. The fork costs **an image reference**. It also closes the issue's one-way door:
|
||||
a node holding nothing can provision the module again, and patches resume.
|
||||
|
||||
**What it does not do** is end the dependence on a codebase its original authors abandoned. It is
|
||||
maintenance mode, largely one project's effort, with no new features intended. It buys time to
|
||||
choose deliberately rather than under pressure — which is worth a great deal, and is not the same as
|
||||
a decision.
|
||||
|
||||
### Garage
|
||||
|
||||
**The best fit for how the mesh provisions.** Its permission model is *per access key, per bucket,
|
||||
read/write/owner* — which is the requirement above stated verbatim rather than approximated. Its
|
||||
admin API is a first-class REST surface with tokens scopeable to exactly the two operations a bucket
|
||||
provision performs. The opaque consumer is **first-party documented** against it, for primary
|
||||
storage, including client-side encryption support.
|
||||
|
||||
Its previously-recorded penalties mostly dissolve under the corrected requirement: it has no console
|
||||
and no identity-provider integration, neither of which is wanted; and it replaces AWS-style ACLs and
|
||||
bucket policies with its own per-key-per-bucket model, which is the thing being asked for.
|
||||
|
||||
**What genuinely remains.** It replicates rather than erasure-codes — immaterial at this volume and
|
||||
on a single array, as above. It does **not implement the full span of S3 endpoints**: object
|
||||
versioning is absent, object locking and server-side encryption endpoints are absent, and lifecycle
|
||||
is partial. **Whether any of the ten buckets depends on those is unmeasured, and it is the one thing
|
||||
that could still disqualify it.**
|
||||
|
||||
### SeaweedFS
|
||||
|
||||
Longest field record of the group, permissive licence, erasure coding, and identity-provider
|
||||
integration on the S3 API through token exchange. **Its console sign-on is a paid feature** — the
|
||||
admin UI itself is open, its identity integration is not. That finding is what falsified the
|
||||
previous version's narrowing, and it is now largely beside the point, since the console is not a
|
||||
requirement.
|
||||
|
||||
What weighs against it here is narrower and more specific: its S3 surface is a **gateway
|
||||
translating onto its own file-system API**, with acknowledged divergence from AWS behaviour at the
|
||||
edges, and there is no first-party documentation for the opaque consumer. For a store already
|
||||
holding real user files in an opaque layout, first-party support is worth more than a feature list.
|
||||
It also carries more moving parts than a single-machine deployment needs.
|
||||
|
||||
### RustFS
|
||||
|
||||
Closest in shape to the incumbent — a similar admin API and client compatibility, so the existing
|
||||
handlers would port with least effort — under a permissive licence, with erasure coding and a
|
||||
console that does integrate an identity provider.
|
||||
|
||||
Two things were recorded about it earlier that were **wrong, and are corrected here**: it is *not* a
|
||||
binary-level drop-in that retains existing data (API compatibility and on-disk compatibility are
|
||||
separate paths, and the on-disk one is preview-scoped with documented encryption limits), and it
|
||||
therefore offers no shortcut around the migration. It also carries **an open defect in the exact
|
||||
area the mesh depends on** — an access key created by an identity-provider user reported denied on
|
||||
all S3 operations.
|
||||
|
||||
Decisively for now: **it reached general availability eight days before this was written.** For a
|
||||
component holding 230 GiB of real user files, field record is a feature, and it does not have one.
|
||||
|
||||
### Ceph RGW
|
||||
|
||||
Remains rejected, for the reason already recorded: disproportionate where the object store is an
|
||||
ordinary module rather than a platform.
|
||||
|
||||
## Where this leaves it
|
||||
|
||||
**The decision is no longer "which product replaces the incumbent".** It is two decisions, and they
|
||||
can be taken in either order but should not be confused:
|
||||
|
||||
1. **Repoint to the maintained fork, or migrate now?** Repointing is an image reference and it
|
||||
closes the issue. Migrating now costs a data copy, two rewrites and a maintenance window, and
|
||||
buys independence from an abandoned codebase sooner.
|
||||
2. **If migrating, which?** On the corrected requirement the ranking is **Garage first** — its
|
||||
permission model *is* the requirement, its provisioning API is the closest match, and the
|
||||
highest-risk consumer is first-party supported. SeaweedFS second, on field record, with a
|
||||
translation-layer caveat that matters more here than its feature list. RustFS not yet, on age.
|
||||
|
||||
Taking (1) does not foreclose (2), and that asymmetry is the argument for taking (1) first.
|
||||
|
||||
## What is still unmeasured
|
||||
|
||||
1. **Whether any of the ten buckets needs object versioning, server-side encryption or lifecycle.**
|
||||
This gates Garage specifically and nothing else here answers it.
|
||||
2. **Whether the fork's release can actually read the incumbent's on-disk format in place.** The
|
||||
format is claimed compatible across a multi-year gap; the migration between them is one-way, so
|
||||
this is tested on a copy or not at all.
|
||||
3. **Whether the opaque consumer's maintenance window is acceptable**, and how long it actually is
|
||||
at 82,496 objects.
|
||||
4. **Whether the eight-drive erasure-coded shape is warranted at all.** The evidence above says it
|
||||
is not buying what it appears to: eight directories, one array, one machine. It looks inherited.
|
||||
|
||||
**Nothing graduates to a decision before 1 and 2.**
|
||||
|
||||
## Two errors in the previous version of this document
|
||||
|
||||
Recorded because the shape of both survives anonymisation and neither is unique to this effort.
|
||||
|
||||
**It ranked on a requirement that did not exist.** Console single sign-on was treated as the axis
|
||||
because the module's configuration showed it wired up, and a wired-up configuration was read as a
|
||||
used feature. It was neither used nor working. *A configured feature is not an observed one*, and
|
||||
the evidence needed was the operator's answer and the logs — both cheap, neither consulted before
|
||||
the ranking was written.
|
||||
|
||||
**It omitted the incumbent's own fork.** The whole effort began because an upstream withdrew its
|
||||
images; whether anyone had continued that upstream was the first question to ask and it was not
|
||||
asked. The candidate list was assembled from a search for *alternatives*, which by construction
|
||||
returns things that are not the incumbent. *When a dependency dies, "who took it over" precedes
|
||||
"what replaces it".*
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
status: proposed
|
||||
date: 2026-09-01
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-24
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 107. Persistent data is a directory bind, never a named volume
|
||||
|
||||
## Context
|
||||
|
||||
Measured 2026-09-24, mid-migration: four modules mount a named Docker volume for real state —
|
||||
`mesh-store` (every database the mesh holds), `mesh-broker` (its data and TLS material),
|
||||
`mesh-registry` (every image), and `searxng`'s cache. Every other module in the catalogue —
|
||||
more than forty — mounts a host directory, `/var/lib/<module>/...`.
|
||||
|
||||
The predecessor did not make this choice. HAL's own `postgres` bound `./db-data`, and its
|
||||
`lavinmq` bound `./data` — directories, both. The mesh's adoption of them
|
||||
([`a63ef3d`](https://git.novox.be/novox/mesh-catalog/commit/a63ef3d), "the postgres module
|
||||
adopts mesh-store instead of raising its own", 2026-09-16) introduced the named volume; three
|
||||
other modules followed the same shape since. Full account of what was found and fixed:
|
||||
[issue 115](../04-ISSUES/115-a-named-docker-volume-is-invisible-and-one-flag-from-gone/00-report.md).
|
||||
|
||||
## The two guarantees are not the same guarantee
|
||||
|
||||
A named volume and a host directory both survive **ordinary** container recreation — a rebuild,
|
||||
a `take`, the `docker rm -f` and push this migration already uses routinely. Neither loses data
|
||||
to that. That was never the question.
|
||||
|
||||
What they do not both survive:
|
||||
|
||||
- **`docker rm -fv`, `docker volume rm`, `docker system prune --volumes`** all target a named
|
||||
volume specifically. The first is one character from the command this migration's own rules
|
||||
already call for after every address change. A host directory has no equivalent command that
|
||||
destroys it by accident — removing it is always a deliberate `rm -rf` on a path someone typed.
|
||||
- **Visibility.** Every tool this migration has used all night to find and verify data —
|
||||
`ls`, `find`, `grep`, a backup job — reaches a host directory for free. A named volume requires
|
||||
knowing to ask Docker (`docker volume inspect`) before its bytes, at
|
||||
`/var/lib/docker/volumes/<name>/_data`, are reachable at all.
|
||||
|
||||
## Decision
|
||||
|
||||
**A container mount holding data that must survive is a host directory bind. A named volume is
|
||||
permitted only for data that is disposable if lost** — a cache, a scratch space, something the
|
||||
module rebuilds on next start without consequence. `searxng`'s `valkey` cache is close to this
|
||||
line and was converted anyway, for consistency and because it costs nothing to.
|
||||
|
||||
Ownership is the one thing a host directory does not get for free that a named volume does:
|
||||
Docker initialises a fresh named volume's ownership to what the container's first process needs;
|
||||
a host directory is whatever created it. A directory made for this purpose must be given the
|
||||
image's expected UID before the container using it starts — read from the running instance being
|
||||
replaced when one exists, rather than guessed.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The four modules were converted: `distribution`, `lavinmq`, `postgres`, `searxng`. Data copied
|
||||
and verified byte-for-byte before each manifest changed; `mesh-store` stopped cleanly first, so
|
||||
its copy is crash-consistent rather than a live read of a running postgres.
|
||||
- **The ownership gap above was not theoretical — it is what happened.** The new directories
|
||||
were created by the operator's tooling running as root; `mesh-store` crash-looped on
|
||||
`mkdir: ... Permission denied` until its directory's ownership was set to match what the
|
||||
original volume already had. Worth a check at `module add` time — nothing catches this today
|
||||
beyond the container failing to start.
|
||||
- Old named volumes were not deleted. They remain the rollback path until confidence in the new
|
||||
mounts is established over time, not one clean start.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 115](../04-ISSUES/115-a-named-docker-volume-is-invisible-and-one-flag-from-gone/00-report.md)
|
||||
- `mesh-catalog` PR #54
|
||||
@@ -1,102 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-25
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0007-connectivity.md
|
||||
---
|
||||
|
||||
# 108. A route carries the policy applied to a request, and names a secret rather than holding one
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0007](0007-connectivity.md) settled that **a route is a grant**: a module contributes the name
|
||||
it wants and the port it listens on, and the proxy hands back the public name. That governs the
|
||||
**grant**. It says nothing about what a request arriving at the name is permitted to do, and the
|
||||
mesh's proxy currently permits everything: its request path is a host lookup and a forward, with no
|
||||
authentication, no source check, no redirect handling and no middleware anywhere in it.
|
||||
|
||||
The standing requirement is that the mesh does **at minimum** what the system it replaces already
|
||||
does. Measured against the predecessor's live configuration and its module catalogue
|
||||
([issue 116](../04-ISSUES/116-route-proxy-has-no-auth-or-ip-restriction/00-report.md)), four
|
||||
capabilities are relied on and absent:
|
||||
|
||||
| Capability | Dependents, counted |
|
||||
|---|---|
|
||||
| Authentication | **three** modules, each gating a credential-less admin surface — a key-value browser UI, a **database web UI**, and the replaced ingress's own dashboard |
|
||||
| Refusal scoped to a path | **one**, and it is a live **incident mitigation** closing a write primitive that was abused |
|
||||
| Path-scoped routing with priority | **two** — the refusal above, and a certificate-challenge path on a host that otherwise routes elsewhere |
|
||||
| Redirect | **two** live routes canonicalising a `www` name onto its apex |
|
||||
|
||||
Counting needed two sources and neither alone is complete: the catalogue cannot see what was
|
||||
hand-written on a node, and a node's configuration directory cannot see what modules declare as
|
||||
container labels. An earlier count read one source and undercounted authentication by two.
|
||||
|
||||
**The third row is a prerequisite, not a sibling.** The proxy's table maps a host to exactly one
|
||||
target, so a host cannot be routed two ways. Adding authentication and a source filter would not
|
||||
make the refusal rule expressible.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave policy out of the mesh; keep the affected routes on the adopted ingress.** *Rejected.*
|
||||
The mesh would run two reverse proxies indefinitely with no principled division between them, and
|
||||
one of the routes held back is a live mitigation — leaving it on a component being decommissioned
|
||||
means its removal date is whenever somebody forgets.
|
||||
2. **A separate, proxy-side settings layer keyed by route name.** *Rejected.* It keeps the grant
|
||||
literally clean, but answering *"what protects this route"* then requires reading two files that
|
||||
nothing keeps in step. A route's protection is part of what a route is.
|
||||
3. **The grant carries a reference; the detail lives in a second layer.** *Rejected.* Both costs of
|
||||
option 2, plus a naming indirection to maintain.
|
||||
4. **Carry the password hash in the declaration.** *Rejected.* It would be the first credential
|
||||
value in a declaration, and a precedent is easier to set than to withdraw. A hash is not a
|
||||
plaintext password, but it is sufficient to pass the gate it protects.
|
||||
5. **An open middleware surface the proxy applies.** *Rejected.* It recreates the thing being
|
||||
replaced, makes the proxy's behaviour unbounded, and an open surface is far harder to narrow later
|
||||
than a closed one is to widen.
|
||||
|
||||
## Decision
|
||||
|
||||
**A route contribution carries the policy applied to requests arriving at its name**, alongside the
|
||||
name, the port and the location it already carries.
|
||||
|
||||
**The set is closed, and it is these four:** authentication; refusal scoped to a path; path-scoped
|
||||
routing with priority; redirect. A fifth is an amendment to this record, deliberately — each
|
||||
addition should be earned by a dependent that exists.
|
||||
|
||||
**Where policy needs a credential, the declaration names a secret the mesh mints and holds. It never
|
||||
carries the value.** This keeps the existing secret machinery as the only thing that holds
|
||||
credentials, and keeps hashes out of anything regenerated, synced or committed.
|
||||
|
||||
**Consequently the routing table is keyed by host and path, with priority** — not by host alone.
|
||||
This follows from the decision rather than being a separate one: two of the four capabilities need a
|
||||
single host routed more than one way.
|
||||
|
||||
## Consequences
|
||||
|
||||
**What this makes possible.** The affected routes can leave the adopted ingress, and "at minimum"
|
||||
becomes a satisfiable claim rather than a standing exception. The incident mitigation gets a durable
|
||||
home in the mesh, which its own note already asked for.
|
||||
|
||||
**What got harder.** The contribution shape grows, and every provider of `route` must understand
|
||||
more of it. The table is no longer a flat map, and priority introduces ordering that has to be
|
||||
deterministic rather than incidental — equal priorities must resolve the same way every time or the
|
||||
proxy becomes non-reproducible. A closed set means a new need is a decision, not a patch.
|
||||
|
||||
**What does not change.** The proxy remains a reference implementation: the contract is the file the
|
||||
mesh writes, not the program that reads it. Another proxy may implement the same file.
|
||||
|
||||
**How this is checked.** A rule states how it is verified, so this one does. A lab bed must show,
|
||||
against a mesh that declared them: a route with authentication refusing an unauthenticated request
|
||||
and admitting an authenticated one; a path-scoped refusal shadowing an ordinary route on the same
|
||||
host while that ordinary route still serves every other path; a redirect answering with the
|
||||
redirect; and — the negative case, which is the one that rots quietly — **a declaration carrying a
|
||||
credential value rather than a reference being refused**, so option 4 cannot return by accident.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 116](../04-ISSUES/116-route-proxy-has-no-auth-or-ip-restriction/00-report.md) — the count,
|
||||
the evidence, and the structural finding about the table.
|
||||
- [ADR 0007](0007-connectivity.md) — a route is a grant. This record extends it to the request.
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — the provision vocabulary a contribution belongs to.
|
||||
- [To-be 08 §3](../03-DESIGN/01-to-be/08-connectivity.md) — where exposure is specified.
|
||||
@@ -123,7 +123,6 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0094** — [A module may hold several secrets from one provider, each a pair of its own](0094-a-module-may-hold-several-secrets-from-one-provider.md)
|
||||
- **0095** — [The control plane is the way to ask a module](0095-the-control-plane-is-the-way-to-ask-a-module.md)
|
||||
- **0098** — [A fact a provider makes at first start is fetched from it, not carried in its manifest](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)
|
||||
- **0108** — [A route carries the policy applied to a request, and names a secret rather than holding one](0108-a-route-carries-the-policy-applied-to-a-request.md)
|
||||
|
||||
### What runs on them, and how it gets there
|
||||
|
||||
@@ -133,7 +132,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0026** — [The mesh has a session of its own, and it is the node session's mechanism](0026-the-mesh-has-a-session-of-its-own.md)
|
||||
- **0027** — [A provision names what the consumer is coupled to, not the role it plays](0027-a-provision-names-what-the-consumer-is-coupled-to.md)
|
||||
- **0035** — [One implementation, several surfaces, and what that costs](0035-one-implementation-several-surfaces.md)
|
||||
- **0038** — [The mesh assigns the port, and a module does not care](0038-the-mesh-assigns-the-port.md)
|
||||
- **0038** — [The mesh assigns the port, and a module does not care](0038-the-mesh-assigns-the-port.md) *(proposed)*
|
||||
- **0040** — [What a module is](0040-what-a-module-is.md)
|
||||
- **0041** — [Events are a relationship, the lighter sibling of provisioning](0041-events-are-a-relationship.md)
|
||||
- **0042** — [The shape of an event on the wire](0042-the-shape-of-an-event-on-the-wire.md)
|
||||
@@ -173,7 +172,6 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0086** — [A secret reaches a process as a file, and an exception is declared](0086-a-secret-reaches-a-process-as-a-file.md)
|
||||
- **0096** — [An upstream image is copied between registries, never through a machine's image store](0096-an-upstream-image-is-copied-between-registries.md)
|
||||
- **0097** — [A vendor image is a declared build input, and a recipe fetches nothing undeclared](0097-a-vendor-image-is-a-declared-build-input.md)
|
||||
- **0107** — [Persistent data is a directory bind, never a named volume](0107-persistent-data-is-a-directory-bind-never-a-named-volume.md)
|
||||
|
||||
### How it is checked
|
||||
|
||||
|
||||
@@ -7,12 +7,11 @@ code:
|
||||
- mesh-controller internal/identity/authority.go
|
||||
- mesh-host internal/identity/serving.go
|
||||
- mesh-host internal/apply (the service that reflects a rule set)
|
||||
updated: 2026-09-25
|
||||
updated: 2026-09-23
|
||||
decisions:
|
||||
- 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md
|
||||
- 02-DECISIONS/0106-the-bus-is-nats.md
|
||||
- 02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md
|
||||
- 02-DECISIONS/0108-a-route-carries-the-policy-applied-to-a-request.md
|
||||
- 02-DECISIONS/0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md
|
||||
- 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
- 02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md
|
||||
@@ -443,39 +442,6 @@ exactly the per-module cost it is meant to remove. The design is the composition
|
||||
stopgap until the manifest layer can carry a label and a domain separately
|
||||
([ADR 0066](../../02-DECISIONS/0066-public-routing-is-name-agnostic.md)).
|
||||
|
||||
### A route also carries what a request arriving at it may do
|
||||
|
||||
*2026-09-25, from comparing the mesh's proxy against the ingress it would replace
|
||||
([issue 116](../../04-ISSUES/116-route-proxy-has-no-auth-or-ip-restriction/00-report.md)),
|
||||
decided in [ADR 0108](../../02-DECISIONS/0108-a-route-carries-the-policy-applied-to-a-request.md).*
|
||||
|
||||
A grant hands back a name. It did not say what the name admits, and the proxy admitted everything —
|
||||
its request path was a host lookup and a forward. Measured against what the replaced ingress
|
||||
actually relies on, four things were missing: **authentication**, **refusal scoped to a path**,
|
||||
**path-scoped routing with priority**, and **redirect**. Three modules depend on the first, each to
|
||||
gate an admin surface that has no login of its own; one dependent of the second is a live incident
|
||||
mitigation.
|
||||
|
||||
**Policy belongs to the route, not beside it.** A contribution carries it along with the name, the
|
||||
port and the location. The alternative — a proxy-side settings layer keyed by route name — keeps the
|
||||
grant literally clean but makes *"what protects this route"* a question answered from two files that
|
||||
nothing keeps in step. A route's protection is part of what a route is.
|
||||
|
||||
**The set is closed at those four.** A fifth is an amendment, so each addition is earned by a
|
||||
dependent that exists rather than added because a middleware surface was open. An open surface would
|
||||
recreate the thing being replaced, and is far harder to narrow later than a closed one is to widen.
|
||||
|
||||
**Where policy needs a credential, the declaration names a secret; it never carries one.** The mesh
|
||||
already mints and holds credentials, and that machinery stays the only thing that does — so a hash
|
||||
never reaches anything regenerated, synced or committed.
|
||||
|
||||
**This re-keys the table.** Two of the four need one host routed more than one way, so the proxy
|
||||
matches on host **and path**, with priority, rather than mapping a host to a single target. Equal
|
||||
priorities must resolve identically every time, or the proxy stops being reproducible.
|
||||
|
||||
The proxy remains a reference implementation: the contract is the file the mesh writes, not the
|
||||
program that reads it, and another proxy may implement the same file.
|
||||
|
||||
## 4 — Filtering
|
||||
|
||||
**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
status: resolved
|
||||
status: open
|
||||
opened: 2026-09-22
|
||||
located-in: [mesh-catalog]
|
||||
fixed-by: mesh-catalog — the 11 not-defensible modules (de-spiegel, gitea, hello-web, mailu, mssql, n8n, novox.be, only-office, photos, photos-eef, photos-filip) had their hardcoded machine-side port mapping stripped to the bare software port, letting ADR 0038's existing assignment machinery (internal/inventory/ports.go's PortFor, declaration.go's publishedOn) assign it as designed. postgres, lavinmq, distribution (foundation, genesis-rewritten) and unifi (protocol-fixed) were left as-is — the two defensible kinds this report names.
|
||||
located-in: []
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-24
|
||||
located-in: [mesh-catalog modules/minio]
|
||||
located-in: [hal modules/minio]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
@@ -19,14 +19,11 @@ pull with `401 UNAUTHORIZED`:
|
||||
<registry>/minio/mc 401 <registry>/minio/console 200
|
||||
```
|
||||
|
||||
The module pins the server image **by digest**, in its `module.json`:
|
||||
The module pins a **tag**, not a digest, and the default is four and a half years old:
|
||||
|
||||
"image": "<registry>/minio/minio@sha256:14cea493…"
|
||||
|
||||
*Corrected 2026-09-24. This first said the module pinned a **tag** whose default was four and a
|
||||
half years old. That described the **predecessor** mesh's object-store module — a different file
|
||||
in a different repository — not the module being cut over to. The conflation, and what it cost,
|
||||
is retracted in full in [the diagnosis](01-diagnosis.md).*
|
||||
```
|
||||
image: <registry>/minio/minio:${MINIO_VERSION:-RELEASE.2022-01-07T01-53-23Z}
|
||||
```
|
||||
|
||||
The cause is upstream and outside the mesh: the vendor **deleted** the community server and client
|
||||
repositories. It is not an access policy that a credential could answer, and nothing about the
|
||||
@@ -57,12 +54,7 @@ vendor namespace*. That was wrong in a way worth keeping, because the evidence l
|
||||
Sibling repositories in the same namespace pulling normally is what rules out a namespace-wide
|
||||
policy, and the registries' own APIs — `404` against `200` — are what establish deletion.
|
||||
|
||||
## The predecessor mesh was not blocked, which is the other half
|
||||
|
||||
*Scope, corrected 2026-09-24: everything in this section describes the **predecessor's** delivery
|
||||
machinery and its object-store module. It is what made the instance harmless, and it is why
|
||||
dropping the module from the queue was unnecessary. It says nothing about how the mesh being built
|
||||
resolves images, which is a different mechanism.*
|
||||
## The mesh was not blocked, which is the other half
|
||||
|
||||
A node that already holds the images runs the module normally. The node carrying the cutover holds
|
||||
the pinned server image, the client, and the load-balancer image the module composes with, all
|
||||
@@ -104,10 +96,9 @@ The instance is harmless; the standing condition is not.
|
||||
warning is not a rule. A mesh cannot state that its modules are installable while the only
|
||||
evidence is that they are already installed.
|
||||
|
||||
The third point is the general one, and it is not specific to this vendor: an image pinned against
|
||||
a registry the mesh does not control — **by tag or by digest, it makes no difference** — is a
|
||||
dependency with no guarantee behind it, and the mesh currently learns it has lost one only by
|
||||
trying to use it.
|
||||
The third point is the general one, and it is not specific to this vendor: an image pinned by tag
|
||||
against a registry the mesh does not control is a dependency with no guarantee behind it, and the
|
||||
mesh currently learns it has lost one only by trying to use it.
|
||||
|
||||
## Open questions
|
||||
|
||||
@@ -116,11 +107,8 @@ trying to use it.
|
||||
goodwill? That is the fix that generalises. It costs storage and a policy about what to mirror,
|
||||
and it is a deliberate move **away** from references-over-payload for third-party images
|
||||
specifically — so it should be decided as such, not smuggled in as a fix.
|
||||
- ~~Should a module's images be pinned **by digest** rather than by tag?~~ **Answered, and the
|
||||
premise was wrong.** This module already pins by digest, and it made no difference: the
|
||||
repository was deleted, so the digest resolves to nothing. A digest buys an exact, auditable
|
||||
artifact; it buys no protection whatever against withdrawal. Struck rather than deleted, because
|
||||
the question was asked from a mistaken reading of the manifest and that is worth seeing.
|
||||
- Should a module's images be pinned **by digest** rather than by tag? It makes the artifact
|
||||
exact and auditable, but does nothing about withdrawal — a deleted digest is just as gone.
|
||||
- What **checks** that every module in the catalogue is still obtainable from a node that holds
|
||||
nothing? Nothing does today. A periodic cold-pull of the catalogue would have caught this on
|
||||
2026-09-11 rather than thirteen days later, mid-cutover.
|
||||
|
||||
@@ -86,49 +86,23 @@ in question.
|
||||
|
||||
So a deploy of this module on that node succeeds today.
|
||||
|
||||
## RETRACTED — the four claims this diagnosis called unsubstantiated
|
||||
## Claims in the original report that could not be substantiated
|
||||
|
||||
*Added 2026-09-24, the same day, after the error was pointed out.*
|
||||
Recorded because they were specific and load-bearing, and acting on them would have wasted time.
|
||||
|
||||
This diagnosis originally carried a table headed *"Claims in the original report that could not be
|
||||
substantiated"*, asserting that a `module.json` did not exist, that no digest pin existed, that an
|
||||
all-zeros runtime digest appeared nowhere, and that a readiness document was not on disk. **The
|
||||
table was wrong and it is withdrawn in full.** The original report was accurate.
|
||||
|
||||
| Claim, as reported | Actual finding |
|
||||
| Claim | Finding |
|
||||
|---|---|
|
||||
| The manifest is a `module.json` pinning the server image by digest at line 83 | **True, and exactly.** `modules/minio/module.json`, digest pin, line 83. |
|
||||
| The module's own runtime artifact carries an all-zeros placeholder digest | **True.** A second container resource pins a runtime sidecar at an all-zeros digest, meaning nothing was ever published for it. |
|
||||
| A dated readiness document records several modules with placeholder digests | **Unverified, not disproven.** It is not on the machine searched. The migration record is a separate private repository that is not checked out there, so its absence locally is not evidence. |
|
||||
| The cached server image is an older release than the module pins | **Unverified.** What was checked was the *predecessor's* tag pin against the cache, which did match. Whether the cached image is the digest this module pins was never checked. |
|
||||
| The module's manifest is a `module.json`, pinning the server image by digest at line 83 | There is **no `module.json` anywhere** in the monorepo. The manifest is YAML and pins a **tag**. No digest pin exists. |
|
||||
| The module's own runtime artifact is a placeholder with an all-zero digest | **Zero occurrences** of that image name or of an all-zero digest anywhere in the tree. The module declares no runtime or sidecar artifact. |
|
||||
| A dated readiness document records six modules with placeholder digests | **No such file exists.** |
|
||||
| The server image is cached locally at an older release than the module pins | The cached tag is **exactly** the pinned one, not an older release. |
|
||||
|
||||
### Why it went wrong, stated plainly
|
||||
|
||||
**One repository was searched, and absence in it was reported as absence.** The catalogue of the
|
||||
mesh being built is a **separate repository**, not checked out on the machine where the search ran.
|
||||
Every one of the four claims was about that repository. The searches were real and their output was
|
||||
reported honestly; the inference drawn from them was not warranted.
|
||||
|
||||
Compounding it, the predecessor's object-store module and the one being cut over to were treated as
|
||||
the same thing. They are different files, in different repositories, with different shapes: the
|
||||
predecessor's is a compose file pinning a **tag** with a version variable, and it declares no
|
||||
sidecar; the one being cut over to is a JSON manifest pinning a **digest**, and it declares two
|
||||
container resources. Findings about the first were written up as findings about the second.
|
||||
|
||||
**The lesson worth keeping, because it is not specific to this issue:** *"zero occurrences anywhere
|
||||
in the tree"* is only ever as strong as the tree that was searched, and a diagnosis must name which
|
||||
tree that was. This one did not, which is what let a one-repository search read as a mesh-wide fact.
|
||||
A confident rebuttal of a correct report is worse than no diagnosis, because it sends the next
|
||||
person looking in the wrong place with the authority of a written record behind them.
|
||||
|
||||
What none of this changes: the images are gone upstream, and that finding stands on the registries'
|
||||
own APIs.
|
||||
None of these change the real finding, which stands: the images are gone upstream.
|
||||
|
||||
## What is located, and what is not
|
||||
|
||||
**Located:** the object-store module in the catalogue of the mesh being built — it pins, by digest,
|
||||
a server image that no longer exists anywhere public, and a runtime sidecar that was never
|
||||
published.
|
||||
**Located:** the module in the monorepo's catalogue — it pins, by tag, an image that no longer
|
||||
exists anywhere public.
|
||||
|
||||
**Not located, and deliberately left open:** the general condition. The mesh has no mirror of the
|
||||
third-party images its modules depend on and no check that a module is obtainable by a node
|
||||
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-24
|
||||
located-in: [mesh-controller module.json, mesh-host internal/apply]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 114 — Should the controller run as a container, or as a process the host supervises directly?
|
||||
|
||||
## What was observed
|
||||
|
||||
On the control-node, 2026-09-24, over a long session of operating the mesh through
|
||||
`mesh-controller`'s CLI (build, push, plan, status, module moved). Every mutating step reached the
|
||||
binary the same way: `docker exec mesh-controller /mesh-controller <command>` — because
|
||||
`mesh-controller`'s own manifest declares its one resource as:
|
||||
|
||||
```json
|
||||
{ "id": "server", "type": "container", "name": "mesh-controller", "network": "host", "args": ["serve"] }
|
||||
```
|
||||
|
||||
Two things about that declaration are worth naming together, because neither is a problem on its
|
||||
own and the combination is what raises the question:
|
||||
|
||||
- **`network: host`.** The controller does not use container network isolation, which is the
|
||||
property a `container` resource type usually buys over a `process` one. It runs with the node's
|
||||
own network namespace either way.
|
||||
- **It is the mesh's single point of coordination.** [`03-DESIGN/01-to-be/06-the-controller.md`](../../03-DESIGN/01-to-be/06-the-controller.md)
|
||||
is explicit: "one node runs it, and nothing takes over" — no election, no quorum, no failover;
|
||||
recovery is restore, not failover.
|
||||
|
||||
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md) gives the host — the one thing tier 0 requires
|
||||
to be a real system daemon — exactly this reasoning for refusing to run in a container: *"installing
|
||||
the container runtime is a step of the bootstrap, so a host inside a container would need the thing
|
||||
it exists to install."* The controller is one tier up and does not install the runtime, but it
|
||||
shares the profile that argument turns on: something the rest of the mesh's operation depends on,
|
||||
sharing fate with a runtime that is not itself.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
Practically, tonight: every controller interaction was raw shell into a container (`docker exec`),
|
||||
not a first-class surface — no logs command beyond `docker logs`, no `systemctl status`, and a
|
||||
session permission classifier that (correctly) treats arbitrary shell into a container as needing
|
||||
sign-off every time, unlike an ordinary supervised process. That friction is a symptom, not the
|
||||
issue itself.
|
||||
|
||||
The actual question is whether `type: container` is buying the controller anything here besides
|
||||
image-based delivery and a restart policy — both of which [ADR 0005](../../02-DECISIONS/0005-the-node-host.md)'s
|
||||
launcher pattern already describes as buildable directly into the host's own supervision (restart on
|
||||
exit, count consecutive failures, roll back after too many, halt after that), for the host's own
|
||||
unit. If the controller were declared `type: process` instead — still built and versioned through
|
||||
the same delivery pipeline, just executed on the node and supervised by the host the way the host
|
||||
supervises itself — it would stop sharing fate with the container runtime's health (restarts,
|
||||
upgrades, disk pressure evicting containers) for the one piece of software whose absence the rest of
|
||||
the mesh is designed to tolerate but nothing is designed to *want*.
|
||||
|
||||
This is squarely a question, not a claim that today's shape is wrong: [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)
|
||||
already tolerates the controller being down by construction (nodes reconcile from their own
|
||||
last-applied state), which may make the container-runtime coupling moot in practice. Nobody has
|
||||
checked.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Does `mesh-host`'s `process` resource type already support the restart/failure-counting semantics
|
||||
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md) describes for the host's own launcher well
|
||||
enough for something this central — or would this need host-side work first?
|
||||
- With `network: host` already in use, what does `type: container` provide the controller today that
|
||||
`type: process` would not?
|
||||
- Is there a real circularity risk — the controller's own health depending on the container runtime
|
||||
it (indirectly, via the host) manages — or does "one node runs it, nothing takes over" already make
|
||||
a controller outage tolerable regardless of which resource type it is?
|
||||
- If the answer is "keep it a container," what does that answer, precisely, that this issue asked —
|
||||
so the next person who notices the same asymmetry finds it answered rather than open again?
|
||||
@@ -1,75 +0,0 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-24
|
||||
located-in: [mesh-catalog]
|
||||
fixed-by: mesh-catalog PR #54 — distribution, lavinmq, postgres and searxng converted to host directory binds; data copied and verified (mesh-store stopped cleanly first for a crash-consistent copy), old named volumes kept as the rollback path
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 115 — A named Docker volume is invisible to the operator, and one flag from gone
|
||||
|
||||
## What was observed
|
||||
|
||||
On the control-node, 2026-09-24, mid-migration, checking every module in the catalogue for how it
|
||||
mounts its data. Five container mounts across four modules use a **named Docker volume** rather
|
||||
than a host directory:
|
||||
|
||||
```
|
||||
distribution mesh-registry → mesh-registry-data:/var/lib/registry
|
||||
lavinmq mesh-broker → mesh-broker-data:/var/lib/lavinmq (+ mesh-broker-tls)
|
||||
postgres mesh-store → mesh-store-data:/var/lib/postgresql/data
|
||||
searxng valkey → searxng-valkey-data:/data
|
||||
```
|
||||
|
||||
Every other module in the catalogue — more than forty of them — mounts a host directory,
|
||||
`/var/lib/<module>/...`, matching what `DATA-CUTOVER.md` and every rehearsed recipe tonight
|
||||
assumes. These four are the exception, not a second convention.
|
||||
|
||||
`mesh-store` is the one that matters most: it holds every database migrated tonight, including a
|
||||
live `keycloak` restore verified minutes before this was written.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
[`03-DESIGN/01-to-be/22-the-work-ahead.md`](../../03-DESIGN/01-to-be/22-the-work-ahead.md) shows
|
||||
this was a deliberate choice, not an oversight — *"Data survives on the named volumes"* — but that
|
||||
sentence answers a narrower question than the one this issue raises. It says a named volume
|
||||
survives **ordinary container recreation** (a rebuild, a `take`, a routine `docker rm -f` and
|
||||
push), which is true and which every module already gets from either a named volume or a host
|
||||
directory equally.
|
||||
|
||||
What it does not address: a named volume is **one flag away from deleted**, in a way a host
|
||||
directory structurally cannot be.
|
||||
|
||||
- `docker rm -f` alone does not remove a named volume — it persists, unreferenced, until something
|
||||
targets it by name.
|
||||
- `docker rm -fv`, `docker volume rm`, and `docker system prune --volumes` all do target it, and
|
||||
the difference from the command this migration already uses routinely (`docker rm -f` — see
|
||||
`HANDOFF.md`'s own "a restart does not re-read anything" rule) is one character.
|
||||
- A named volume is invisible to an operator working the way this migration has worked all
|
||||
night: `ls`, `find`, `grep` across `/services/*` and `/var/lib/*`. Finding it requires knowing to
|
||||
ask Docker (`docker volume inspect`), and its actual bytes sit under
|
||||
`/var/lib/docker/volumes/<name>/_data`, a path nothing points at.
|
||||
- Nothing external can back it up, snapshot it, or notice it growing without going through
|
||||
Docker's own volume machinery — a host directory is a directory; a filesystem-level backup job
|
||||
already reaches it for free.
|
||||
|
||||
`mesh-store` carries the sharpest version of this: every module's database, the mesh's own
|
||||
inventory, identity and licence stores — the single foundation piece the rest of the mesh depends
|
||||
on — sits somewhere the operator's ordinary tools do not look.
|
||||
|
||||
## Decided
|
||||
|
||||
**Persistent data is a host directory bind, never a named volume. A named volume may hold only
|
||||
data that is disposable if lost.** All four hold real state and all four are in scope — including
|
||||
`mesh-registry` and `searxng`'s cache, not only `mesh-store`. Decided 2026-09-24; the record is
|
||||
[ADR 0107](../../02-DECISIONS/0107-persistent-data-is-a-directory-bind-never-a-named-volume.md).
|
||||
|
||||
## Open questions
|
||||
|
||||
- The safe migration path for each, in order of stakes — `mesh-store` live and holding every
|
||||
database migrated tonight, `mesh-broker`, `mesh-registry`, then `searxng`'s cache, which is
|
||||
genuinely disposable and may not need migrating at all if it is rebuilt rather than moved.
|
||||
- Should the catalogue refuse a module declaring a named volume for anything but disposable data
|
||||
at `module add`, the way [issue 091](../091-a-module-definition-carries-a-machine-port/00-report.md)
|
||||
asks the same of a hardcoded machine port — so a convention violation is caught at registration
|
||||
rather than found by reading the whole catalogue?
|
||||
@@ -1,152 +0,0 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-09-25
|
||||
located-in: [mesh-controller examples/route-proxy]
|
||||
fixed-by: mesh-controller PR #58 — the four capabilities implemented in the reference proxy; the table re-keyed by host and path with a total ordering; a declaration carrying a credential refused rather than served, and an unreadable secret failing closed
|
||||
amended-design: 03-DESIGN/01-to-be/08-connectivity.md
|
||||
---
|
||||
|
||||
# 116 — The mesh's proxy applies no policy to a request: no authentication, no source restriction, no path scoping, no redirects
|
||||
|
||||
## What was observed
|
||||
|
||||
On 2026-09-25, deciding whether the mesh's own reverse proxy
|
||||
([to-be 08](../../03-DESIGN/01-to-be/08-connectivity.md)) can replace the ingress the mesh adopted
|
||||
from the predecessor, as the permanent public entry point. The standing requirement is that the mesh
|
||||
does **at minimum** what the system it replaces already does, so the comparison is against the
|
||||
predecessor's real, live configuration and its module catalogue — not against a feature list.
|
||||
|
||||
**The proxy's entire request path is a host lookup and a forward.** Its handler takes the request's
|
||||
host, finds a target in a table, and either answers a named 404 or hands the request to the reverse
|
||||
proxy. There is no authentication check, no source-address check, no redirect handling and no
|
||||
middleware chain anywhere in the program.
|
||||
|
||||
The table is the reason this is structural rather than a missing feature: it maps **host → one
|
||||
target**, and the lookup strips the port and lowercases the host. **A host cannot be routed two ways.**
|
||||
|
||||
The design is deliberate as far as it goes — *"a route hands back a name, not a credential"* — but
|
||||
that governs the **route grant**. It says nothing about what a request arriving at that name is
|
||||
allowed to do, which is what the live configuration relies on.
|
||||
|
||||
## What the predecessor actually relies on, counted
|
||||
|
||||
Two sources, because neither alone is complete: the **module catalogue**, and the **ingress's own
|
||||
dynamic configuration** on the node. The catalogue misses what was hand-written on the node; the
|
||||
node's directory misses what modules declare as container labels. An earlier version of this report
|
||||
read only the latter and undercounted as a result.
|
||||
|
||||
### 1. Basic authentication — three dependents in the catalogue
|
||||
|
||||
| Module | What it is the only gate on |
|
||||
|---|---|
|
||||
| the key-value store | its browser UI, which has no login of its own |
|
||||
| the relational store | a **database web UI** |
|
||||
| the ingress itself | its own dashboard — twice, counting a desktop flavour |
|
||||
|
||||
Every one is a credential-less admin surface whose sole protection is a middleware the mesh's proxy
|
||||
does not have. The database web UI is the worst of the three, and the ingress dashboard means the
|
||||
ingress is currently protecting itself with a mechanism its replacement lacks.
|
||||
|
||||
### 2. Outright refusal on a path — an incident mitigation
|
||||
|
||||
One hand-written rule on the node blocks external access to an internal API path on the forge. Its
|
||||
own header records it as **incident response to a compromise**, closing the write primitive that was
|
||||
abused. It is expressed as an allow-list containing a single documentation-range address — that is,
|
||||
a deny-everyone — and the middleware is named accordingly.
|
||||
|
||||
It is **not** an address-scoped allow-list in any useful sense, and reading it as one points at the
|
||||
wrong fix. What it needs is the ability to refuse a request outright, scoped to a path.
|
||||
|
||||
The same header already records where this belongs: *"not mesh-managed. Durable home is the
|
||||
route-proxy module; re-home when convenient."*
|
||||
|
||||
### 3. Path-scoped routing with priority — and this one gates the others
|
||||
|
||||
That rule matches a **path prefix** on a host that is **already routed elsewhere**, and carries an
|
||||
explicit high priority so it shadows the ordinary route. The mail module needs the same shape for a
|
||||
different reason: it routes a certificate-challenge path on a host that otherwise goes to the mail
|
||||
front end.
|
||||
|
||||
Because the table maps a host to exactly one target, **neither is expressible today, and adding
|
||||
authentication and a source filter would not make them so.** Path scoping with priority is a
|
||||
prerequisite for the refusal rule, not a feature beside it.
|
||||
|
||||
### 4. Redirect rules — live, and they fail quietly
|
||||
|
||||
Two routes canonicalise a `www` name onto its apex with a rewriting redirect. They appear in the
|
||||
node's configuration and **not** in the catalogue, so a catalogue-only survey misses them. They are
|
||||
the easiest of the four to lose, because losing them produces no error — just two public names that
|
||||
quietly stop redirecting.
|
||||
|
||||
## What compared cleanly, and is not part of this issue
|
||||
|
||||
- **Large and streaming request bodies.** Four modules raise or remove the body cap — the object
|
||||
store, the file-sync application, the image registry. The standard library's reverse proxy streams
|
||||
with no default cap, so this needs nothing added.
|
||||
- **Connection upgrades**, used by at least one console route: native to the standard library's
|
||||
reverse proxy. Not yet verified live against this proxy, but not absent by design the way the four
|
||||
gaps above are.
|
||||
- **Certificate issuance.** The proxy refuses to certify any name the mesh did not route, and
|
||||
defaults to a staging issuer until a node opts in to production
|
||||
([issue 004](../004-certificate-issuance-targets-production/00-report.md)) — stricter than the
|
||||
hand-maintained configuration it would replace.
|
||||
- **Several public names for one module.** Already solved by the `contributes` many-shape; needs
|
||||
nothing from the proxy.
|
||||
|
||||
## Why it matters
|
||||
|
||||
Under the "at minimum" rule, **nothing can be called a replacement for the predecessor's ingress
|
||||
while any of the four is missing** — and one of them is a live mitigation for an exploited
|
||||
vulnerability. The two outcomes if it is left unfixed are both bad:
|
||||
|
||||
- the affected routes stay on the adopted ingress indefinitely, leaving the mesh running two
|
||||
reverse proxies side by side with no principled division between them; or
|
||||
- they are migrated anyway, and three credential-less admin surfaces become reachable by anyone who
|
||||
can resolve a name, while a known-exploited path loses the block that was put in front of it
|
||||
during an incident.
|
||||
|
||||
## Open questions — answered
|
||||
|
||||
These were design questions, not implementation details, and the fix was not written before they
|
||||
were answered. All three were settled in
|
||||
[ADR 0108](../../02-DECISIONS/0108-a-route-carries-the-policy-applied-to-a-request.md): policy goes
|
||||
**on the route**, the set is **closed at the four**, and a declaration **names a secret and never
|
||||
carries one**. Kept as asked, because what was rejected and why is the half worth having.
|
||||
|
||||
- **Where does request-level policy come from?** Today a contribution carries a name and a port.
|
||||
Extending it to carry policy keeps the mesh as the source of truth, consistent with everything
|
||||
else a route already does. A separate proxy-side settings layer keyed by route name decouples
|
||||
policy from the grant but adds a second place to look. The module's own README notes *"the contract
|
||||
is the file, not this program"*, so this is a contract decision and not a property of one
|
||||
reference implementation.
|
||||
- **May a declaration carry a credential?** A password hash in a contribution puts a secret in a
|
||||
declaration. That cuts across how the mesh mints and holds secrets, and it should be settled
|
||||
deliberately rather than as a side effect of whichever option is less code.
|
||||
- **Is the right general shape "these four", or something narrower?** Authentication, refusal, path
|
||||
scoping and redirects are what the predecessor uses *today*. Whether route-level policy should be
|
||||
an open middleware surface, or exactly these four and no more, is worth deciding before any of it
|
||||
is written — an open surface is far harder to withdraw than to add.
|
||||
|
||||
## What the fix covers, and what it does not yet allow
|
||||
|
||||
*2026-09-25, on resolution.* The gap this issue reports is closed: the proxy applies policy, the
|
||||
four capabilities exist, the table is keyed by host and path with a total ordering, and the two
|
||||
failure modes that would rot quietly are held by tests — a declaration carrying a credential is
|
||||
refused rather than served, and an unreadable secret makes the route refuse rather than open.
|
||||
|
||||
**It does not yet let an operator move the affected routes.** That needs the mesh side: a manifest
|
||||
able to declare these values, and the controller minting the secret that `auth` names. Until both
|
||||
exist the capability is reachable only by writing the routes file by hand, so the routes held back
|
||||
on the adopted ingress stay there.
|
||||
|
||||
Resolved rather than left open because the issue reports a gap **in the proxy**, and that gap is
|
||||
gone. The remaining work is not this fault persisting; it is the ordinary build-out of a contract
|
||||
this record's decision created, and it belongs to
|
||||
[to-be 08](../../03-DESIGN/01-to-be/08-connectivity.md) rather than here.
|
||||
|
||||
**One thing found while fixing it, worth keeping.** Priority was first read with the reader for
|
||||
ports, which caps at 65535 — and the one real rule this has to reproduce is declared at 100000. It
|
||||
parsed to zero, so refusal and path scoping would both have shipped looking complete, passing their
|
||||
own tests, and doing nothing on the only case that motivated them. *A validator borrowed from a
|
||||
neighbouring field is a silent default*, and the test that now guards it goes through the proxy,
|
||||
because at the parser the value looked fine.
|
||||
@@ -1,47 +0,0 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-25
|
||||
located-in: [mesh-catalog modules/umami]
|
||||
---
|
||||
|
||||
# 118 — umami's store answers the dial and times out the query
|
||||
|
||||
## What was observed
|
||||
|
||||
`umami.novox.be` has answered `502` through the whole of 2026-09-25's migration session
|
||||
(first noted mid-afternoon, still true at night). The container restart-loops on a
|
||||
timescale of about a minute. Its own log, every cycle:
|
||||
|
||||
```
|
||||
✓ DATABASE_URL is defined.
|
||||
✓ Database connection successful.
|
||||
Invalid `prisma.$queryRaw()` invocation:
|
||||
Raw query failed. Code: `N/A`. Message: `Operation has timed out`
|
||||
```
|
||||
|
||||
The connection is established — the dial succeeds — and the first raw query then times
|
||||
out. This is not a credentials fault and not an unreachable store.
|
||||
|
||||
## What it is not
|
||||
|
||||
- Not the routing layer: the `502` is Traefik faithfully reporting a backend that is
|
||||
restart-looping. The stale duplicate Traefik router for this name (a HAL-era
|
||||
hand-authored file beside the mesh-written one) was removed the same night and changed
|
||||
nothing, as expected.
|
||||
- Not the mesh's grant machinery: the binding and sealed secret compose, and the store
|
||||
accepts the login — a wrong credential refuses the dial, and this dial succeeds.
|
||||
|
||||
## Where to look
|
||||
|
||||
A dial that succeeds and a query that times out, from a container on one network to a
|
||||
store on another, has the shape of a path-MTU or conntrack fault (large response packets
|
||||
dropped after the small handshake ones pass), or of the store accepting the TCP
|
||||
connection while the backend it proxies for is wedged. Neither is proven. What is known
|
||||
to differ for umami against every working consumer of the same store tonight is nothing
|
||||
yet — that comparison is the first move.
|
||||
|
||||
## Why it is filed rather than chased
|
||||
|
||||
The 2026-09-25 session's scope was routing and the build chain; this fault predates the
|
||||
night's changes, survived them unchanged, and needs its own sitting with the store's own
|
||||
logs beside the consumer's.
|
||||
@@ -1,59 +0,0 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-26
|
||||
located-in: [mesh-host internal/apply, mesh-controller]
|
||||
---
|
||||
|
||||
# 119 — a hold is not a line in the apply report, and an operator flew blind into an outage
|
||||
|
||||
## What was observed
|
||||
|
||||
During the route-proxy edge cutover on novox (2026-09-26): the module was assigned, the
|
||||
push reported success, `status` said the node was doing everything it was told — and the
|
||||
module's three containers did not exist. The operator stopped the predecessor's proxy on
|
||||
the strength of those reports, and every public name on the node went dark until rollback.
|
||||
|
||||
The cause was correct behaviour, invisibly reported. The first (rolled-back) route-proxy
|
||||
attempt had left `/var/lib/route-proxy/*` on disk; on re-assign, the adopted node *found*
|
||||
those directories, held them (ADR 0100, exactly as designed), and held every container
|
||||
that mounts them — `"would mount /var/lib/route-proxy/ca, found on this adopted node;
|
||||
not run until route-proxy is taken"`. All of that lived only in `state.json`. What the
|
||||
operator saw:
|
||||
|
||||
- the push: `sent novox 346 resource(s)` — the controller's count of what it sent;
|
||||
- the node's journal: `applied 330 resource(s)` — sixteen fewer, with no line saying
|
||||
which sixteen or why;
|
||||
- `status`: green — a held resource is not "wrong", so nothing was flagged;
|
||||
- `node show novox`: the holds list did NOT include route-proxy's (it showed only holds
|
||||
the *controller* knew about from take-time listings, not what the node decided at
|
||||
apply-time).
|
||||
|
||||
Four surfaces, none carrying the one sentence that mattered: *route-proxy is assigned
|
||||
but not taken, and its containers will not run until it is.*
|
||||
|
||||
## Why this is a real fault and not operator error alone
|
||||
|
||||
The operator error (an edge-flip runbook that omitted `take`) was only possible because
|
||||
every surface reported success. A system whose correct refusals are indistinguishable
|
||||
from completed work will keep converting small procedural gaps into outages. The
|
||||
`sent 346 / applied 330` discrepancy was the single visible symptom, and interpreting it
|
||||
required reading `state.json` by hand.
|
||||
|
||||
## What would have prevented it
|
||||
|
||||
Any one of:
|
||||
|
||||
1. **The apply report says what it held.** `applied 330 resource(s), 16 held for
|
||||
untaken modules (route-proxy: 13, …)` — one line in the journal.
|
||||
2. **`status` counts holds against untaken-but-assigned modules.** A module assigned,
|
||||
pushed, and running zero of its containers is at minimum worth a "waiting on take"
|
||||
line — it is never converged in any useful sense.
|
||||
3. **`node show <node>` shows the node's own held list**, not only what take-time
|
||||
computed — the node already records it in `state.json` with reasons.
|
||||
|
||||
## Precedent
|
||||
|
||||
The photos cutover hit the same semantics benignly the same week (assign → held
|
||||
containers in `Created` state → take), and the mailu cutover documented "take is the
|
||||
verb, and ADR 0100 meant it". The semantics are consistent and right; the reporting is
|
||||
what let them be forgotten at the worst moment.
|
||||
@@ -1,47 +0,0 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-09-26
|
||||
located-in: [mesh-host internal/apply]
|
||||
---
|
||||
|
||||
# 121 — a volume path is not in the spec comparison, and a roll-out raced a data move
|
||||
|
||||
## What was observed
|
||||
|
||||
Landing the "module data lives in /var/lib" change on novox (mesh-catalog #97), two
|
||||
distinct faults surfaced in one hour:
|
||||
|
||||
1. **Building a module with a roll-out upgrade policy IS deploying it.** gitea's policy
|
||||
was roll-out; the `build` that registered its repathed manifest sent it to the node
|
||||
immediately, which recreated the container mounting the *not-yet-renamed* (empty)
|
||||
`/var/lib/gitea/data`. The forge came back as its own install page, fresh host keys
|
||||
and all, and every subsequent pipeline build died on `repository not found` — which
|
||||
also blocked the fix, since re-registering the other modules needed the forge. The
|
||||
operator narrative "build, then move data, then push" is only safe under the record
|
||||
policy; nothing warned that one module in the batch would skip the pause.
|
||||
|
||||
2. **Changing a container's volume paths does not recreate the container.** After the
|
||||
final push, five of the six repathed modules kept their old containers running
|
||||
("Up 13–26 hours") — the new declaration's volume paths differ from the running
|
||||
containers' mounts, and the apply judged them current. Same class as mesh-host #27
|
||||
(`dns`/`ip` absent from the comparison): a field the comparison does not read is a
|
||||
field that can never change a running container. Benign here only because a rename
|
||||
on one filesystem preserves the mounted inode — the running containers keep serving
|
||||
the same bytes the new path names, and the next natural recreation converges. A
|
||||
cross-filesystem move, or a path change to *different* data, would have silently
|
||||
split the module between two worlds.
|
||||
|
||||
## What would have prevented it
|
||||
|
||||
- `build` printing the module's upgrade policy when that policy will act on the result
|
||||
("gitea rolls out on build — the node will receive this immediately"), or a
|
||||
`--register-only` flag for exactly this choreography.
|
||||
- Volumes (and every other container field) in the spec comparison, or the honest
|
||||
refusal: "this field changed and I cannot apply it without recreation."
|
||||
|
||||
## Recovery that worked
|
||||
|
||||
Instant renames both ways broke the circular dependency (forge needed for builds,
|
||||
builds needed for the push, push needed for the forge): data back to the old path,
|
||||
old-spec forge started, artifacts rebuilt, data renamed forward, push. Nothing lost;
|
||||
the install-page junk was discarded twice.
|
||||
Reference in New Issue
Block a user