2026-09-21 09:59:22 +00:00
9 changed files with 180 additions and 17 deletions
@@ -0,0 +1,75 @@
+---
+topic: building it
+status: accepted
+date: 2026-09-21
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
+---
+
+# 86. A secret reaches a process as a file, and an exception is declared
+
+## Context
+
+The mesh seals a secret to the machine that uses it and discards the plaintext; the host unseals it
+into a file at 0600 ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)). Then
+the module hands it to its container through an env-file, and the runtime puts it where anything
+on the machine that can talk to the runtime can read it: `docker inspect` prints it, and
+`/proc/<pid>/environ` holds it for the life of the process
+([issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md)).
+
+Two facts made this a decision rather than a fix. **It is a supported shape**: placing
+`${secret:…}` in a file a container reads as its environment is what playbook 06 shows, and 36
+containers in 25 catalogue modules do it, the controller's own among them. And **nothing says which
+secrets are exposed**: a reader cannot tell from a manifest whether a credential is protected from
+`inspect` or not, because the manifest looks the same either way. The vault
+([ADR 0085](0085-a-secret-is-a-provision.md)) rests on the seal this undoes.
+
+## Considered Options
+
+1. **Leave it: the environment is where configuration goes.** Rejected — it gives the sealed value
+   back to a routine operation, and the care taken to seal it is then theatre.
+2. **Refuse every secret in an environment.** Rejected — some software reads its configuration
+   from the environment and nothing else, and a rule the catalogue cannot obey is a rule that gets
+   switched off.
+3. **A secret reaches a process as a file; an environment exception is declared, with a reason,
+   and refused otherwise.** Adopted.
+
+## Decision
+
+**A secret reaches a process as a file.** A module mounts the file the host wrote and points the
+program at it; the mesh's own programs accept a `_FILE` twin for every variable that carries a
+credential, the way the controller's store connections already did.
+
+**A container that reads a secret from its environment says so.** The manifest key
+`secrets-in-environment` on the container carries the reason. It is catalogue-level — the host
+never sees it — and it exists so a reader can tell from the manifest which secrets are exposed
+that way and why.
+
+**Everything else is refused.** A `${secret:…}` placeholder inside a container's `env` is never
+filled and is refused outright. A file carrying a secret that a container names in `env-file` is
+refused unless the container declares the exception.
+
+## Consequences
+
+The property *a sealed credential is readable only where it is used* becomes a manifest-level
+fact: true where no exception is declared, stated where one is. The controller reads all six of
+its credentials from files; the 35 other containers are marked with their reason and convert one
+by one where their software accepts a path, which is per-module work under
+[issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md).
+
+What got harder: a module author meets one more refusal, and the reason they write is only as
+honest as they are. What is not decided: the reach of the exposure on a node — which identities can
+talk to the runtime — which decides whether a declared exception is a hardening item or something
+sharper.
+
+## How it is checked
+
+The catalogue engine refuses at composition, before anything reaches a machine, and its tests
+refuse both shapes and prove the reason is stripped from what is sent.
+
+## References
+
+- [issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md)
+- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md), [ADR 0085](0085-a-secret-is-a-provision.md)
+- [`03-DESIGN/01-to-be/13-credentials-and-their-rotation.md`](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)
@@ -153,6 +153,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0069** — [A module is a repository and a path within it](0069-a-module-is-a-repository-and-a-path.md)
 - **0076** — [The SDK is a published package, and the toolchain resolves it by version](0076-the-sdk-is-a-published-package.md)
 - **0082** — [The registry is reached by name, and the overlay is its security](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)
+- **0086** — [A secret reaches a process as a file, and an exception is declared](0086-a-secret-reaches-a-process-as-a-file.md)

 ### How it is checked

@@ -1,11 +1,12 @@
 ---
 layer: as-is
 status: implemented
-code: [hal]
-updated: 2026-08-23
+code: [hal, mesh-controller, mesh-catalog, mesh-host]
+updated: 2026-09-21
 decisions:
  - 02-DECISIONS/0011-managed-files-are-generated-never-edited.md
  - 02-DECISIONS/0009-modules-and-the-graph.md
+  - 02-DECISIONS/0085-a-secret-is-a-provision.md
 ---

 # Configuration and secrets
@@ -73,9 +74,31 @@ when the file is created, so regeneration left the previous mode in place. The c
 worth remembering beyond the instance: a permission set at creation is not a permission
 maintained.

-**Rotation is not a mesh operation.** Secrets can be generated and granted; there is no
-mechanism that rotates one and informs everything holding it. Where a rotation has been done,
-it has been done by hand, and doing it wrong has taken services down.
+**Rotation was not a mesh operation** in the mesh being replaced, and doing it by hand took
+services down. On the mesh that exists now it is: `rotate <provision>` discards a pair credential
+and delivers both ends in one send, and a module's own secret is such a pair credential when the
+module takes it from the vault — which, at the time of writing, one module does (redis).
+
+## The vault, as it runs
+
+Since 2026-09-21 the mesh runs `mesh-vault`, a foundation module installed at genesis beside the
+adopted store and broker. It provides `secret`: a module that requires one receives a pair
+credential the controller minted, and the vault's ledger records the holder and the value's
+fingerprint, notices a rotation, and answers over the mesh by fingerprint only. It holds no value.
+
+Genesis makes the store's superuser and the broker's administrator rather than copying the
+template's, keeps them at the paths the store and broker modules declare as their own secrets, and
+makes an **operator sealing key** before the first secret is accepted: its private half is a file
+beside the produced bundle, which the operator carries off the machine, and its public half is
+what the mesh records. Every secret a module holds for itself and every pair credential is sealed
+to that key as well as to its node. The export of those copies is written beside the key at the
+end of genesis and kept by the vault on its own disk; the operator recovers any secret from it, off
+the mesh, with `secret recover`. Secrets made before the key existed, or sealed to a replaced key,
+are listed as such rather than passed off as recoverable. The produced bundle and the installer's
+transcript carry no credential in the clear.
+
+Two things this does not yet do: a module with several own secrets cannot take them all from the
+vault (issue 069), and an operator cannot hand a value into a pair credential (issue 070).

 ## Node-level and mesh-level values

@@ -5,6 +5,9 @@ code:
  - mesh-host examples/foundation-first-node.lock
  - mesh-host internal/apply
  - mesh-host internal/bootstrap/phase3.go
+  - mesh-host internal/bootstrap/rootsecrets.go
+  - mesh-host internal/bootstrap/operator.go
+  - mesh-catalog modules/mesh-vault
  - mesh-catalog modules/postgres
  - mesh-catalog modules/lavinmq
  - mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)
@@ -5,11 +5,12 @@ code:
  - mesh-controller internal/inventory/secrets.go
  - mesh-controller cmd/mesh-controller/rotate.go
  - mesh-controller examples/postgres-provisioner
-updated: 2026-09-20
+updated: 2026-09-21
 decisions:
  - 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
  - 02-DECISIONS/0009-modules-and-the-graph.md
  - 02-DECISIONS/0085-a-secret-is-a-provision.md
+  - 02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md
 ---

 # 13 — Credentials, and moving them
@@ -88,6 +89,19 @@ reads exactly like a credential that was never delivered. That has now happened
 environment needs a person in the middle of the one path that exists so there is not one, and puts
 a superuser password where `docker inspect` prints it.

+## A secret reaches a process as a file
+
+The care above ends at the container's door if the module then hands the value to the runtime as
+an environment variable: `docker inspect` prints it, and the process's `/proc` entry holds it for
+anything on the machine that can talk to the runtime. So a secret reaches a process **as a file**
+([ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md)): the module mounts
+the file the host wrote and points the program at it, and the mesh's own programs take a `_FILE`
+twin for every variable that carries a credential. Software that reads only its environment is
+not forbidden; it is **declared**, on the container, with a reason, so the manifest says which
+secrets are exposed that way. **Checked** at composition: the catalogue engine refuses a
+`${secret:…}` in a container's `env` outright, and a secret-carrying file named in `env-file`
+unless the container carries the declaration.
+
 ## How it is checked

 Not by comparing two files. **Two ends holding a matching string proves they agree, not that either
@@ -1,8 +1,8 @@
 ---
 layer: to-be
-status: in-progress
-code: [mesh-catalog, mesh-controller]
-updated: 2026-09-20
+status: implemented
+code: [mesh-catalog, mesh-controller, mesh-host]
+updated: 2026-09-21
 decisions:
  - 02-DECISIONS/0085-a-secret-is-a-provision.md
  - 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
@@ -106,9 +106,9 @@ The vault's second job is to hold these, and it does so without holding a value:
  answered by it.

 **Genesis** makes the operator key first and mints real root secrets in place of the fixed ones the
-foundation is raised with, so the mesh is handed over with nothing well-known in it. That step is
-the installer's and is not yet built; until it is, the fixed credentials are the as-is and are
-said so in [21](21-the-installation-in-full.md).
+foundation is raised with, so the mesh is handed over with nothing well-known in it. The installer
+does this ([21](21-the-installation-in-full.md)); what it runs is described in the as-is
+([`00-as-is/06`](../00-as-is/06-configuration-and-secrets.md)).

 A module's vault-provided secret — the pair credential of
 [13](13-credentials-and-their-rotation.md) — is sealed to the operator the same way, as is every
@@ -1,9 +1,9 @@
 ---
-status: open
+status: resolved
 opened: 2026-09-10
-located-in: []
-fixed-by:
-amended-design:
+located-in: [mesh-controller internal/catalogue, mesh-catalog modules/mesh-controller]
+fixed-by: ADR 0086; mesh-controller feat/secret-not-in-environment (envfile twins for the broker settings, catalogue refusal of secrets in env / undeclared env-file); mesh-catalog (the controller reads its six credentials from files; 35 containers declare their exception); mesh-host (the installer delivers any MESH_…_FILE own secret)
+amended-design: 03-DESIGN/01-to-be/13-credentials-and-their-rotation.md
 ---

 # 041 — A credential the mesh took care to seal ends up in the process environment
@@ -65,3 +65,11 @@ environment; 28 catalogue manifests use env-file for a secret, and nothing in th
 engine refuses a `${secret:…}` placeholder in one. The vault work of
 [ADR 0085](../../02-DECISIONS/0085-a-secret-is-a-provision.md) rests on the seal this weakens.

+## Resolved 2026-09-21
+
+By [ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md): a secret reaches
+a process as a file, an environment exception is declared with a reason, and the catalogue engine
+refuses the undeclared shape. The controller, the instance this was opened on, reads all six of
+its credentials from files. The 35 other containers carry a declared reason; converting each where
+its software accepts a path remains per-module work and is not this issue's.
+
@@ -2,7 +2,7 @@
 status: resolved
 opened: 2026-09-20
 located-in: [mesh-host internal/bootstrap, mesh-host examples/foundation-first-node.lock]
-fixed-by: mesh-host feat/secrets-vault (ee0c8b8, genesis root credentials + operator key + vault); mesh-controller feat/secrets-vault (e140ed5, 565f144); mesh-catalog feat/secrets-vault; proven by the one-node genesis bed step V5
+fixed-by: mesh-host PR 14 (e30a6b0), mesh-controller PR 34 (6b695c8), mesh-catalog PR 30 (d03520f), mesh-lab PR 39 (7e2e97f); proven by the one-node genesis bed step V5, 22/22
 amended-design: 03-DESIGN/01-to-be/24-the-secrets-vault.md
 ---

@@ -0,0 +1,39 @@
+---
+status: located
+opened: 2026-09-21
+located-in: [mesh-controller module.json, mesh-catalog modules/mesh-controller/module.json, mesh-host internal/bootstrap]
+fixed-by:
+amended-design:
+---
+
+# The controller's manifest exists twice, and nothing keeps the copies equal
+
+## Symptom, as observed
+
+The control plane's manifest lives at the root of its own repository, where the mesh reads it
+whenever it builds the controller from source, and again in the catalogue under
+`modules/mesh-controller`, where the installer reads it at genesis. The two differ only in how the
+image is named — a build section in one, a placeholder digest in the other — and are otherwise
+meant to be the same document.
+
+Changing the catalogue copy alone (to make the controller read its credentials from files,
+[ADR 0086](../../02-DECISIONS/0086-a-secret-reaches-a-process-as-a-file.md)) produced a mesh that
+installed correctly and then could not be pushed to: the first rebuild of the controller from
+source replaced the mesh's record with the repository's copy, which still carried the old shape,
+and every later push of the node was refused on the controller's behalf. `status` reported the
+machine as doing what it was told throughout, because nothing had been sent.
+
+## Why it matters beyond this instance
+
+- **One module, two sources of truth.** [ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md)
+  says a module is a repository and a path; the controller has two paths, and which one the mesh
+  believes depends on which command last touched it.
+- **The failure is silent at the wrong layer.** The divergence surfaces as a resolution refusal
+  on an unrelated push, not as a warning that the copies differ.
+- **It will happen again** to whoever next edits the controller's manifest for any reason.
+
+## What would close it
+
+Either the catalogue copy goes and genesis reads the controller's manifest from the controller's
+repository (the installer already has the checkout, since it builds from it), or a check refuses
+two copies that differ outside the image lines. The first is the honest one.