issue 121: a volume path is not in the spec comparison, and a roll-out raced a data move

02-DECISIONS/0112-a-module-definition-names-no-node-mesh-or-path.md

@@ -1,161 +0,0 @@
----
-topic: what runs on it
-status: proposed
-date: 2026-09-25
-deciders: jochen
-reconstructed: false
-extends: 0046-a-module-configuration-is-its-assignments-not-its-manifest.md
----
-
-# 112. A module definition names no node, no mesh and no path: everything it needs is resolved at assignment
-
-## Context
-
-[Issue 118](../04-ISSUES/118-a-module-definition-decides-where-its-files-live/00-report.md) found
-**789 host-path strings in 70 of the catalogue's 71 module definitions.** Each definition chooses
-where on the machine its directories, mounts, bindings, secrets, env-files and received files live,
-and often repeats that path in an environment variable or in code. Mounts are checked against what
-the definition declares ([ADR 0091](0091-a-mount-is-declared-three-ways.md)); nothing checks the
-copies. The issue records what that has already allowed:
-
-- a provider that would provision nobody without a word;
-- a contributions file that carries host paths into containers, so every provider must mount its
-  grants directory at the identical path;
-- defaults in code that disagree with their own manifests;
-- no way to assign one module to one node twice, because every identity is keyed by the module's name.
-
-**The mesh has already decided this once, for ports.** [ADR 0038](0038-the-mesh-assigns-the-port.md):
-the mesh assigns the machine-side port and the module says only what it needs, and three copies
-became one fact. [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)
-made the manifest identity and defaults, and the assignment's settings the configuration.
-[ADR 0084](0084-which-provider-serves-a-consumer.md) made which provider serves a consumer part of
-the assignment. Paths are the largest thing still left in the definition.
-
-## Considered Options
-
-**1. Keep host paths in definitions, and check that every copy agrees.** Rejected. It checks the
-agreement of something that should not be there. A definition still could not follow its data to
-another disk, be adopted onto a machine whose data is already somewhere, or run twice on one node.
-
-**2. Keep host paths in definitions, and have the mesh rewrite them per assignment.** Rejected. It is
-string surgery on paths: deciding which strings are machine paths by their shape, which is the
-inference this repository has refused elsewhere. And the definition would still read as though it
-decided where things live.
-
-**3. A definition names variables, and the assignment resolves them.** Chosen.
-
-## Decision
-
-**A module definition is node-agnostic and mesh-agnostic.** It names no node, no mesh and no host
-path. Everything that makes a running instance *this* instance is a variable.
-
-**Installing a module on a node resolves every variable, or refuses.** A refusal names each
-unresolved variable and what could answer it. Variables are answered from three sources:
-
-1. **The assignment's own configuration.** Values chosen for this module on this node, carried as
-   settings ([ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)). An
-   endpoint binding a public name to a port is one.
-2. **Provisions, resolved by the mesh against a contract.** What other modules provide: a database,
-   a bucket, a vhost, a secret. Each comes with the contract the mesh and the module's
-   specification define. Where a provision has several providers, which one answers is part of the
-   assignment ([ADR 0084](0084-which-provider-serves-a-consumer.md)), so a module may take its
-   database from another node. Some have one provider per mesh by decision: a module's own secret
-   is a `secret` provision the controller mints and the vault records
-   ([ADR 0085](0085-a-secret-is-a-provision.md), as amended). Whether the vault should instead
-   generate a secret against its contract is an open question, raised while reviewing this, and not
-   decided here.
-3. **What the mesh generates or knows.** The credential the controller mints for each provision a
-   module takes ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)), the ports it
-   assigns ([ADR 0038](0038-the-mesh-assigns-the-port.md)), facts about the machine.
-
-**A directory is a provision, provided by the node's host.** A module requires one by name, such as
-its configuration or its data. Its contract is the owner and mode it needs, including the owner its
-image expects ([ADR 0107](0107-persistent-data-is-a-directory-bind-never-a-named-volume.md)). It
-carries no persistence flag. A directory is kept while it holds anything
-([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), which refused a `keep` flag for good
-reason), and data that is disposable is not a directory at all but a named volume (0107).
-
-*Where* a directory is on the machine is the assignment's. A node has a default layout, and an
-assignment may place one directory elsewhere: on a second disk, or where an adopted machine's data
-already is ([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)). A directory is
-always provided on the module's own node, because a host path means nothing on any other.
-
-**An operator's shared data stays an `access`** ([ADR 0051](0051-shared-data-is-the-operators.md)),
-not a directory provision. 0051 rejected giving a directory an operator owner, because the mesh
-must never create, chown or remove such data, and that stands. What changes is only where its
-location is written: the module says it needs read or read-write access, and the assignment says
-where the data is.
-
-**Inside a container, a module sees its own paths.** The definition says where the image expects
-each directory. The mesh mounts the assignment's location there. No host path is ever a value a
-process reads.
-
-**The mesh's own files carry no host path.** Bindings, secrets and contributions are named relative
-to where the module receives them, so a provider reads what it was given without mounting anything
-at a machine-identical path.
-
-**An assignment has an identity of its own: an instance name**, defaulting to the module's name.
-Everything keyed by the module's name today is keyed by the instance instead: directories,
-container names, the login a consumer presents, broker accounts, a claim's holder, the settings
-an assignment carries, and a provider's identity. So **one module may be assigned to one node more
-than once.** What must stay singular stays so by a claim, or by the assignment's own configuration
-colliding: a public name already taken is refused like any other singular thing.
-
-## What this changes in earlier records
-
-On acceptance, each of these is amended by a record of its own, not edited:
-
-- [ADR 0051](0051-shared-data-is-the-operators.md): an access keeps its shape and its semantics; its
-  path moves from the definition to the assignment.
-- [ADR 0091](0091-a-mount-is-declared-three-ways.md): a mount's host side is a resolved variable,
-  checked as resolved rather than as a path the definition declares.
-- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md): settings are
-  addressed to an instance, not to a module on a node.
-- [ADR 0084](0084-which-provider-serves-a-consumer.md): a provider is a (node, instance) pair, not a
-  (node, module) pair, so a consumer can name one of two instances on one node.
-- The [glossary](../00-META/glossary.md): *provision* widens from "a service one module provides" to
-  include a directory the node's host provides, and *instance* is added. Neither lands while this
-  record is only proposed, because the glossary is the authority on the words in use, not on words
-  under review.
-
-## Consequences
-
-- **Every definition changes.** 70 of 71 name host paths today. The change is mechanical for most.
-  The design has to say how existing modules migrate without their data moving: an adopted or
-  already-running assignment is placed where its data already is.
-- The controller resolves variables at assignment and refuses unresolved ones. The host provides
-  directories. The contributions file's format changes, and so does the SDK's reconcile loop that
-  reads it.
-- Identity moves from the module to the instance, which touches logins, broker accounts, settings,
-  provider selection and every resource name.
-- **What got harder:** a definition no longer says where a module's data is on a machine. The
-  assignment does, and `plan` shows it. That is the point, and it is also a real loss of
-  at-a-glance legibility, which the overview has to give back.
-- **Not decided here:** the variable syntax; a node's default layout; whether a second instance of a
-  module is supported from the first step or after the definitions have moved; whether the vault
-  generates secrets.
-
-## How it is checked
-
-| Rule | Checked by |
-|---|---|
-| A definition names no host path | A catalogue test: the host side of every mount, and every resource location, binding, secret, receives and grants entry, is a variable rather than an absolute path. A declared list of exceptions shrinks to empty as definitions move. |
-| No host path is a value a process reads | A catalogue test: every absolute path in a container's environment or env-files lies on the container side of one of its mounts, or is declared the image's own. A second test finds literal paths in module code used as fallbacks for an environment variable. |
-| A definition names no node and no mesh | The parser has no field that names a node; a node is named only in an assignment. A catalogue test finds no domain name in any definition value. |
-| Installation resolves every variable | A resolution test with one variable unanswered: refused, naming the variable and its possible sources. |
-| A directory is provided on its module's own node | A resolution test: an assignment placing a directory on another node is refused. |
-| A provider reads what it was given without an identical mount | A provisioner test reading a contributions file whose credentials are named relative to where it is mounted. |
-| A module can run twice on one node | A resolution test assigning one module twice under two instance names: two directories, two logins, two containers, separate settings, no collision. |
-| A public name already taken is refused | A resolution test: a second instance asking for a public name the first holds is refused, naming the first. |
-| An adopted assignment is placed where its data is | An adoption test: the directory resolves to the data's existing location, and nothing is moved. |
-
-## References
-
-- [Issue 118](../04-ISSUES/118-a-module-definition-decides-where-its-files-live/00-report.md): the evidence
-- [ADR 0038](0038-the-mesh-assigns-the-port.md): the same decision, for ports
-- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md): configuration is the assignment's
-- [ADR 0084](0084-which-provider-serves-a-consumer.md): which provider answers is the assignment's
-- [ADR 0085](0085-a-secret-is-a-provision.md), [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md):
-  secrets as a provision, and who mints what
-- [ADR 0051](0051-shared-data-is-the-operators.md), [ADR 0107](0107-persistent-data-is-a-directory-bind-never-a-named-volume.md),
-  [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md): what a directory's contract carries, and what it must not

02-DECISIONS/README.md

@@ -155,7 +155,6 @@ python3 00-META/checks/index.py            fail if stale
 - **0087** — [A seeded file is created once, and what grows in it is not the mesh's](0087-a-seeded-file-is-created-once.md)
 - **0091** — [A mount is declared, and there are three things it can be](0091-a-mount-is-declared-three-ways.md)
 - **0099** — [A step that runs once names what it reads, and runs again when it changed](0099-a-step-that-runs-once-names-what-it-reads.md)
-- **0112** — [A module definition names no node, no mesh and no path: everything it needs is resolved at assignment](0112-a-module-definition-names-no-node-mesh-or-path.md)  *(proposed)*
 
 ### How it is built
 

04-ISSUES/118-a-module-definition-decides-where-its-files-live/00-report.md

@@ -1,79 +0,0 @@
----
-status: open
-opened: 2026-09-25
-located-in: []
-fixed-by:
-amended-design:
----
-
-# 118 — A module definition decides where its files live on the machine
-
-## What was observed
-
-A review of where module code reads its files turned up a cross-cutting pattern. Every module
-definition in the catalogue chooses, in its own manifest, where on the machine its files live.
-Counted on the catalogue's `main`, 2026-09-25:
-
-| where in the definition | host-path strings |
-|---|---|
-| directory and file resources | 257 |
-| container mounts, host side | 230 |
-| own secrets | 78 |
-| bindings | 53 |
-| env-files | 50 |
-| secrets | 35 |
-| container environment | 28 |
-| accesses | 21 |
-| receives, grants | 24 |
-| everything else | 13 |
-
-**789 host-path strings in 70 of the 71 definitions.** Mounts are checked: a container may not
-mount a path its module never declared ([ADR 0091](../../02-DECISIONS/0091-a-mount-is-declared-three-ways.md)).
-Nothing checks the same path where it is retyped as a value: an environment variable, an env-file
-line, a literal in module code.
-
-### Where that has already gone wrong
-
-- **A provider that would provision nobody, silently.** One DNS provider mounts its grants
-  directory at a short path inside its container, then tells its provisioner to read the
-  contributions file at the host path, which does not exist in there. Nothing requires the
-  provision today, so it has not failed yet. When a consumer arrives, it will get no record, and
-  nobody will be told.
-- **The mesh's own wire carries host paths into containers.** Each contribution names its
-  consumer's credential as "the file on this machine holding that consumer's credential", a host
-  path computed from the provider's grants directory. So every provider has to mount that directory
-  at the *identical* path, or it cannot read what it was given. Ten of the eleven providers with a
-  grants directory do. It is a convention nothing states or checks, and the eleventh is the
-  provider above.
-- **The warning that would have caught it is lost in the SDK.** The controller always writes the
-  contributions file, even when empty, so a provider can tell "nothing asked" from "never written".
-  The SDK's reconcile loop treats an unreadable file as empty, and logs nothing.
-- **Code carries copies with nothing checking them.** Several modules default a path in code when an
-  environment variable is unset. Five of those defaults disagree with the value their own manifest
-  sets. One of them is a host path used inside a container that does not mount it.
-
-### And a module cannot be assigned to one node twice
-
-Everything that identifies a running module is keyed by the module's name: its directories, its
-container names, the login it presents to a provider, its broker account. Two assignments of one
-module to one node would share every one of them. Assigning the same application twice is an
-ordinary need: production beside staging, one site per customer, two instances of one service
-configured differently, two stores of one engine.
-
-## Why it matters beyond this instance
-
-A definition that names machine paths is not portable between nodes. It cannot follow data onto a
-second disk, or onto a machine being adopted with its data already in place, without editing the
-module. It cannot run twice on one node. It keeps every path in two or three places with nothing
-checking that they agree. The defects above are what that allows, and each was found by reading,
-not by any check.
-
-## Open questions
-
-- Should a definition name any host path at all, or should every location come from the
-  assignment and the mesh?
-- If a directory is something a module *requires* rather than *declares*, what is its contract:
-  ownership, mode, whether it is kept when the module goes?
-- What identifies an assignment, if a module may be assigned to one node more than once?
-- What would the contributions file carry instead of host paths, so a provider needs no
-  identical-path mount?

04-ISSUES/118-umamis-store-answers-the-dial-and-times-out-the-query/00-report.md

@@ -0,0 +1,47 @@
+---
+status: located
+opened: 2026-09-25
+located-in: [mesh-catalog modules/umami]
+---
+
+# 118 — umami's store answers the dial and times out the query
+
+## What was observed
+
+`umami.novox.be` has answered `502` through the whole of 2026-09-25's migration session
+(first noted mid-afternoon, still true at night). The container restart-loops on a
+timescale of about a minute. Its own log, every cycle:
+
+```
+✓ DATABASE_URL is defined.
+✓ Database connection successful.
+Invalid `prisma.$queryRaw()` invocation:
+Raw query failed. Code: `N/A`. Message: `Operation has timed out`
+```
+
+The connection is established — the dial succeeds — and the first raw query then times
+out. This is not a credentials fault and not an unreachable store.
+
+## What it is not
+
+- Not the routing layer: the `502` is Traefik faithfully reporting a backend that is
+  restart-looping. The stale duplicate Traefik router for this name (a HAL-era
+  hand-authored file beside the mesh-written one) was removed the same night and changed
+  nothing, as expected.
+- Not the mesh's grant machinery: the binding and sealed secret compose, and the store
+  accepts the login — a wrong credential refuses the dial, and this dial succeeds.
+
+## Where to look
+
+A dial that succeeds and a query that times out, from a container on one network to a
+store on another, has the shape of a path-MTU or conntrack fault (large response packets
+dropped after the small handshake ones pass), or of the store accepting the TCP
+connection while the backend it proxies for is wedged. Neither is proven. What is known
+to differ for umami against every working consumer of the same store tonight is nothing
+yet — that comparison is the first move.
+
+## Why it is filed rather than chased
+
+The 2026-09-25 session's scope was routing and the build chain; this fault predates the
+night's changes, survived them unchanged, and needs its own sitting with the store's own
+logs beside the consumer's.

04-ISSUES/119-a-hold-is-not-a-line-in-the-apply-report/00-report.md

@@ -0,0 +1,59 @@
+---
+status: open
+opened: 2026-09-26
+located-in: [mesh-host internal/apply, mesh-controller]
+---
+
+# 119 — a hold is not a line in the apply report, and an operator flew blind into an outage
+
+## What was observed
+
+During the route-proxy edge cutover on novox (2026-09-26): the module was assigned, the
+push reported success, `status` said the node was doing everything it was told — and the
+module's three containers did not exist. The operator stopped the predecessor's proxy on
+the strength of those reports, and every public name on the node went dark until rollback.
+
+The cause was correct behaviour, invisibly reported. The first (rolled-back) route-proxy
+attempt had left `/var/lib/route-proxy/*` on disk; on re-assign, the adopted node *found*
+those directories, held them (ADR 0100, exactly as designed), and held every container
+that mounts them — `"would mount /var/lib/route-proxy/ca, found on this adopted node;
+not run until route-proxy is taken"`. All of that lived only in `state.json`. What the
+operator saw:
+
+- the push: `sent novox 346 resource(s)` — the controller's count of what it sent;
+- the node's journal: `applied 330 resource(s)` — sixteen fewer, with no line saying
+  which sixteen or why;
+- `status`: green — a held resource is not "wrong", so nothing was flagged;
+- `node show novox`: the holds list did NOT include route-proxy's (it showed only holds
+  the *controller* knew about from take-time listings, not what the node decided at
+  apply-time).
+
+Four surfaces, none carrying the one sentence that mattered: *route-proxy is assigned
+but not taken, and its containers will not run until it is.*
+
+## Why this is a real fault and not operator error alone
+
+The operator error (an edge-flip runbook that omitted `take`) was only possible because
+every surface reported success. A system whose correct refusals are indistinguishable
+from completed work will keep converting small procedural gaps into outages. The
+`sent 346 / applied 330` discrepancy was the single visible symptom, and interpreting it
+required reading `state.json` by hand.
+
+## What would have prevented it
+
+Any one of:
+
+1. **The apply report says what it held.** `applied 330 resource(s), 16 held for
+   untaken modules (route-proxy: 13, …)` — one line in the journal.
+2. **`status` counts holds against untaken-but-assigned modules.** A module assigned,
+   pushed, and running zero of its containers is at minimum worth a "waiting on take"
+   line — it is never converged in any useful sense.
+3. **`node show <node>` shows the node's own held list**, not only what take-time
+   computed — the node already records it in `state.json` with reasons.
+
+## Precedent
+
+The photos cutover hit the same semantics benignly the same week (assign → held
+containers in `Created` state → take), and the mailu cutover documented "take is the
+verb, and ADR 0100 meant it". The semantics are consistent and right; the reporting is
+what let them be forgotten at the worst moment.

04-ISSUES/121-a-volume-path-is-not-in-the-spec-comparison/00-report.md

@@ -0,0 +1,47 @@
+---
+status: open
+opened: 2026-09-26
+located-in: [mesh-host internal/apply]
+---
+
+# 121 — a volume path is not in the spec comparison, and a roll-out raced a data move
+
+## What was observed
+
+Landing the "module data lives in /var/lib" change on novox (mesh-catalog #97), two
+distinct faults surfaced in one hour:
+
+1. **Building a module with a roll-out upgrade policy IS deploying it.** gitea's policy
+   was roll-out; the `build` that registered its repathed manifest sent it to the node
+   immediately, which recreated the container mounting the *not-yet-renamed* (empty)
+   `/var/lib/gitea/data`. The forge came back as its own install page, fresh host keys
+   and all, and every subsequent pipeline build died on `repository not found` — which
+   also blocked the fix, since re-registering the other modules needed the forge. The
+   operator narrative "build, then move data, then push" is only safe under the record
+   policy; nothing warned that one module in the batch would skip the pause.
+
+2. **Changing a container's volume paths does not recreate the container.** After the
+   final push, five of the six repathed modules kept their old containers running
+   ("Up 13–26 hours") — the new declaration's volume paths differ from the running
+   containers' mounts, and the apply judged them current. Same class as mesh-host #27
+   (`dns`/`ip` absent from the comparison): a field the comparison does not read is a
+   field that can never change a running container. Benign here only because a rename
+   on one filesystem preserves the mounted inode — the running containers keep serving
+   the same bytes the new path names, and the next natural recreation converges. A
+   cross-filesystem move, or a path change to *different* data, would have silently
+   split the module between two worlds.
+
+## What would have prevented it
+
+- `build` printing the module's upgrade policy when that policy will act on the result
+  ("gitea rolls out on build — the node will receive this immediately"), or a
+  `--register-only` flag for exactly this choreography.
+- Volumes (and every other container field) in the spec comparison, or the honest
+  refusal: "this field changed and I cannot apply it without recreation."
+
+## Recovery that worked
+
+Instant renames both ways broke the circular dependency (forge needed for builds,
+builds needed for the push, push needed for the forge): data back to the old path,
+old-spec forge started, artifacts rebuilt, data renamed forward, push. Nothing lost;
+the install-page junk was discarded twice.