From 03266fd4a218d1408e69d4e41fa53dcf89ed8dad Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 1 Sep 2026 17:19:43 +0200 Subject: [PATCH] =?UTF-8?q?027=20=E2=80=94=20a=20container=20cannot=20foll?= =?UTF-8?q?ow=20a=20file,=20and=20a=20rotated=20credential=20is=20the=20ca?= =?UTF-8?q?se?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by the forge failing to start. I had put `restart-on` on nine containers so they would pick up a rotated credential; it belongs to a service, and the host refused the whole declaration. Removing it fixes the modules and leaves the reason I reached for it. The mechanism is written against exactly this, in the host's own words: a running service does not re-read its configuration, so replace the file, find it already running, do nothing, and the machine keeps behaving as before while every check passes. Every word of that applies to a container, and nearly everything the mesh runs is one. The cost is concrete. Rotation replaces the file and tells the provider to accept the new credential. A provider reconciles, so it takes it. A consumer is usually a container, so it does not — and the two ends hold different passwords, which is the fault ADR 0001 records costing two days. The test that proves rotation works uses a consumer that reads the file on each attempt, so it does not meet this. Two things a fix has to keep: it stays declared state rather than a command, because the link may not carry an action; and where an env-file changed, the honest verb is recreate rather than restart, because a container's environment is fixed at creation. --- .../00-report.md | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 04-ISSUES/027-a-container-cannot-follow-a-file/00-report.md diff --git a/04-ISSUES/027-a-container-cannot-follow-a-file/00-report.md b/04-ISSUES/027-a-container-cannot-follow-a-file/00-report.md new file mode 100644 index 0000000..c88562d --- /dev/null +++ b/04-ISSUES/027-a-container-cannot-follow-a-file/00-report.md @@ -0,0 +1,65 @@ +--- +status: located +opened: 2026-09-01 +located-in: [mesh-host, mesh-control] +fixed-by: +amended-design: +--- + +# 027 — A container cannot follow a file, and a rotated credential is the case + +## Symptom + +A service can say `restart-on`: *these files changed, so I must be restarted*. A container cannot. +It is not in the shape, and the host refuses a declaration that tries. + +So a container reading its password from a file keeps the password it started with, for ever. +Nothing reports anything: the file is right, the container is up, every check passes. + +## Why this is the same fault the mechanism exists for + +`restart-on` is written against exactly this, in the host's own words: + +> a running service does not re-read its configuration. Replace the file, find the service already +> running, do nothing, and the machine keeps behaving the way it did before — while every check +> passes, because the file is right and the service is up. + +Every word applies to a container, and more so. **Nearly everything the mesh runs is a container** +— a database, a forge, a mail system — and a credential arrives as a file it reads at start. + +## What it costs, concretely + +**Rotation does not reach a container.** Rotating a credential replaces the file on the machine and +tells the provider to accept the new one. The provider is a program that reconciles, so it takes +the change. The consumer is usually a container, so it does not. The two ends then hold different +passwords, which is the fault the whole design is arranged to prevent +([ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) records it costing +two days). + +The end-to-end test that proves rotation works uses a consumer that reads the file on each +attempt, so it does not meet this. + +## Why it was not noticed + +The gap is invisible from the control plane. A manifest carrying `restart-on` on a container +composes into a declaration without complaint and is refused on the machine, so the only way to +learn is to run one — which is how it was found, after nine of them had shipped across seven +modules. + +## What a fix has to keep + +- **Declared state, not a command.** `restart-on` is deliberately not *restart this*: it says the + running thing must reflect these files, and the host works out that it does not. Whatever + containers get must keep that shape, because the link may not carry an action + ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). +- **Recreate, not restart, where that is the honest verb.** A container's environment is fixed at + creation. If what changed is an `env-file`, restarting the container is not enough — it has to be + made again. That is a different act from a service reload and should not be described as one. +- **It must not fire on every reconcile.** A container that is recreated whenever the host looks at + it is worse than one that never follows the file. + +## The near alternative, and why it is not enough + +A module can avoid the problem by having its program read the file on each use rather than at +start. That works for something written for this mesh, and is not available for a database, a +forge or a mail system — which is the whole population this is about.