163 lines
10 KiB
Markdown
163 lines
10 KiB
Markdown
---
|
|
status: resolved
|
|
opened: 2026-09-25
|
|
located-in: [mesh-catalog modules, mesh-controller internal/catalogue]
|
|
fixed-by: mesh-catalog PR 193 (the conversion), mesh-controller PR 170 (TestPlacedDirectoriesKeepTheirPaths, which proves it moved nothing); the placed-directory mechanism itself predates this in mesh-controller internal/catalogue/dir_into.go
|
|
amended-design: 03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md
|
|
---
|
|
|
|
# 119 — A module definition decides where its files live on the machine
|
|
|
|
## What was observed
|
|
|
|
A review of where module code reads its files turned up a cross-cutting pattern. Every module
|
|
definition in the catalogue chooses, in its own manifest, where on the machine its files live.
|
|
Counted on the catalogue's `main`, 2026-09-25:
|
|
|
|
| where in the definition | host-path strings |
|
|
|---|---|
|
|
| directory and file resources | 257 |
|
|
| container mounts, host side | 230 |
|
|
| own secrets | 78 |
|
|
| bindings | 53 |
|
|
| env-files | 50 |
|
|
| secrets | 35 |
|
|
| container environment | 28 |
|
|
| accesses | 21 |
|
|
| receives, grants | 24 |
|
|
| everything else | 13 |
|
|
|
|
**789 host-path strings in 70 of the 71 definitions.** Mounts are checked: a container may not
|
|
mount a path its module never declared ([ADR 0091](../../02-DECISIONS/0091-a-mount-is-declared-three-ways.md)).
|
|
Nothing checks the same path where it is retyped as a value: an environment variable, an env-file
|
|
line, a literal in module code.
|
|
|
|
## 2026-09-26 — counted by what the path *is*, and what a node's layout would replace
|
|
|
|
The count above treats every host path the same. They are not the same, and only one of the
|
|
categories is this issue's subject. Recounted by role across the 71 definitions:
|
|
|
|
| What the path is | Count | Does a definition naming it name *this* machine? |
|
|
|---|---|---|
|
|
| A module's own data, `/var/lib/<module>/…` | 289 | **Yes** — this issue |
|
|
| What the mesh writes for a module, `/var/lib/mesh/<module>/…` | 225 | **Yes** — this issue |
|
|
| The operator's shared data, `/services/…` | 166 | Separately answered: an `access`, never created or owned by the mesh ([ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md)); *where* it is belongs on the assignment |
|
|
| A system file the mesh owns, `/etc/…` and a few under `/var/` | 18 | **No.** The path is the fact — that file is at that path on every machine of the kind. Naming it says nothing about this installation |
|
|
| A path inside a container | 236 | **No.** The software's own contract, true in any mesh that runs the image |
|
|
|
|
So the category a node's default layout would replace is **514**, not 789 and not 698: a module's own
|
|
data, and what the mesh writes for that module. The rest is either already answered or was never the
|
|
problem.
|
|
|
|
## What the design already says, and what it does not
|
|
|
|
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md) (proposed)
|
|
makes a directory a host provision whose contract is the owner and mode the module needs, and says
|
|
where it lands on the machine is the assignment's.
|
|
[To-be 27](../../03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md) (proposed) resolves
|
|
that two ways: **a node's default layout — a root per node with one directory per assignment beneath
|
|
it** — used when the assignment says nothing, and **a placement** when the assignment puts one
|
|
directory elsewhere, such as where an adopted machine's data already is.
|
|
|
|
That is the reservation model: assigning a module to a node gives it a directory, and the definition
|
|
never names one. Three things are not written anywhere:
|
|
|
|
1. **Where the root comes from.** Nothing says it is a node setting, fixed when the node is installed,
|
|
and nothing says what a node that states none falls back to.
|
|
2. **What sits beneath it.** To-be 27's own open list ends at exactly this: the layout beneath the
|
|
root, beyond one directory per assignment. The catalogue currently keeps two subtrees per module —
|
|
its data, and what the mesh writes for it — so this decides whether an assignment's directory holds
|
|
both, or the mesh keeps its own.
|
|
3. **The order the existing 514 are retired in**, which 0112 also leaves open.
|
|
|
|
## Why this is not a manifest edit
|
|
|
|
Every one of those 514 paths names a directory that holds data a service is using. A definition that
|
|
changes where it looks, without the data moving with it, does not fail: the mesh creates the new
|
|
directory with the right owner and mode, the container starts, and the service comes up **empty**.
|
|
The loudest case in this mesh is an object store consumer whose bucket holds 174.9 GiB; a mail spool
|
|
and the mesh's own store are the same shape.
|
|
|
|
So the retirement of these paths is a **data migration with a verification step**, module by module,
|
|
not a change to a definition. That is the reason this section documents and stops: the count and the
|
|
categories are what someone needs to plan it, and the planning belongs with whoever can see the
|
|
machine.
|
|
|
|
### Where that has already gone wrong
|
|
|
|
- **A provider that would provision nobody, silently.** One DNS provider mounts its grants
|
|
directory at a short path inside its container, then tells its provisioner to read the
|
|
contributions file at the host path, which does not exist in there. Nothing requires the
|
|
provision today, so it has not failed yet. When a consumer arrives, it will get no record, and
|
|
nobody will be told.
|
|
- **The mesh's own wire carries host paths into containers.** Each contribution names its
|
|
consumer's credential as "the file on this machine holding that consumer's credential", a host
|
|
path computed from the provider's grants directory. So every provider has to mount that directory
|
|
at the *identical* path, or it cannot read what it was given. Ten of the eleven providers with a
|
|
grants directory do. It is a convention nothing states or checks, and the eleventh is the
|
|
provider above.
|
|
- **The warning that would have caught it is lost in the SDK.** The controller always writes the
|
|
contributions file, even when empty, so a provider can tell "nothing asked" from "never written".
|
|
The SDK's reconcile loop treats an unreadable file as empty, and logs nothing.
|
|
- **Code carries copies with nothing checking them.** Several modules default a path in code when an
|
|
environment variable is unset. Five of those defaults disagree with the value their own manifest
|
|
sets. One of them is a host path used inside a container that does not mount it.
|
|
|
|
### And a module cannot be assigned to one node twice
|
|
|
|
Everything that identifies a running module is keyed by the module's name: its directories, its
|
|
container names, the login it presents to a provider, its broker account. Two assignments of one
|
|
module to one node would share every one of them. Assigning the same application twice is an
|
|
ordinary need: production beside staging, one site per customer, two instances of one service
|
|
configured differently, two stores of one engine.
|
|
|
|
[ADR 0112](../../02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md), which answers
|
|
this report, declines that need rather than meeting it. A module is assigned at most once to a node,
|
|
because every identity in the mesh is already a module on a node. The cases above become different
|
|
modules, or the same module on different machines.
|
|
|
|
## Why it matters beyond this instance
|
|
|
|
A definition that names machine paths is not portable between nodes. It cannot follow data onto a
|
|
second disk, or onto a machine being adopted with its data already in place, without editing the
|
|
module. It cannot run twice on one node. It keeps every path in two or three places with nothing
|
|
checking that they agree. The defects above are what that allows, and each was found by reading,
|
|
not by any check.
|
|
|
|
## Open questions
|
|
|
|
- Should a definition name any host path at all, or should every location come from the
|
|
assignment and the mesh?
|
|
- If a directory is something a module *requires* rather than *declares*, what is its contract:
|
|
ownership, mode, whether it is kept when the module goes?
|
|
- What identifies an assignment, if a module may be assigned to one node more than once?
|
|
- What would the contributions file carry instead of host paths, so a provider needs no
|
|
identical-path mount?
|
|
|
|
## Resolved, 2026-09-30 — the module's half; the mesh's half is issue 174
|
|
|
|
**A definition no longer decides where its own data lives.** Twenty-eight definitions that named their
|
|
data directories now place them: the module's root as `place: "."`, a sub-directory by its id, and every
|
|
host-side reference — bindings, secrets, own secrets, grants, receives, file paths, mounts, env-files —
|
|
as `${dir:<id>}`. Twenty-eight others had already been written that way. Five directories whose id is
|
|
not their last segment keep their path as a placement, which is the exception the design allows and
|
|
the reason nothing else has to move for them.
|
|
|
|
**Nothing moved, and a test says so.** The controller's `TestPlacedDirectoriesKeepTheirPaths` takes the
|
|
catalogue before and after, resolves every converted definition on the default root with the
|
|
controller's own rule, and compares it whole with the definition before it: identical for all
|
|
twenty-eight. So the retirement this record said was a data migration turned out not to be one, on
|
|
one condition — a node's default root is where the data already is, and every node's is — and the
|
|
machines see no change. A node that sets another root is the case this does not cover, and it does
|
|
not exist.
|
|
|
|
**What remains is not this record's.** The 232 host paths still in the catalogue are where the mesh
|
|
writes what it makes for a module, under `/var/lib/mesh/<module>`; design 27 says the mesh places
|
|
those itself, and it does not yet. That is [issue 174](../174-the-meshs-own-files-for-a-module-are-placed-by-the-definition/00-report.md).
|
|
The defects this record listed under *where that has already gone wrong* are unchanged by this and
|
|
stay in 174's scope where they concern the mesh's files; the operator's shared data stays an access
|
|
([ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md)).
|
|
|
|
The manifest change lands with the catalogue's next merge; the rollout is a rebuild that changes no
|
|
machine, checked by comparing each machine's plan before and after.
|