jschoubben
03266fd4a2
Found by the forge failing to start. I had put `restart-on` on nine containers so they would pick up a rotated credential; it belongs to a service, and the host refused the whole declaration. Removing it fixes the modules and leaves the reason I reached for it. The mechanism is written against exactly this, in the host's own words: a running service does not re-read its configuration, so replace the file, find it already running, do nothing, and the machine keeps behaving as before while every check passes. Every word of that applies to a container, and nearly everything the mesh runs is one. The cost is concrete. Rotation replaces the file and tells the provider to accept the new credential. A provider reconciles, so it takes it. A consumer is usually a container, so it does not — and the two ends hold different passwords, which is the fault ADR 0001 records costing two days. The test that proves rotation works uses a consumer that reads the file on each attempt, so it does not meet this. Two things a fix has to keep: it stays declared state rather than a command, because the link may not carry an action; and where an env-file changed, the honest verb is recreate rather than restart, because a container's environment is fixed at creation.
3.1 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | ||
|---|---|---|---|---|---|---|
| located | 2026-09-01 |
|
027 — A container cannot follow a file, and a rotated credential is the case
Symptom
A service can say restart-on: these files changed, so I must be restarted. A container cannot.
It is not in the shape, and the host refuses a declaration that tries.
So a container reading its password from a file keeps the password it started with, for ever. Nothing reports anything: the file is right, the container is up, every check passes.
Why this is the same fault the mechanism exists for
restart-on is written against exactly this, in the host's own words:
a running service does not re-read its configuration. Replace the file, find the service already running, do nothing, and the machine keeps behaving the way it did before — while every check passes, because the file is right and the service is up.
Every word applies to a container, and more so. Nearly everything the mesh runs is a container — a database, a forge, a mail system — and a credential arrives as a file it reads at start.
What it costs, concretely
Rotation does not reach a container. Rotating a credential replaces the file on the machine and tells the provider to accept the new one. The provider is a program that reconciles, so it takes the change. The consumer is usually a container, so it does not. The two ends then hold different passwords, which is the fault the whole design is arranged to prevent (ADR 0001 records it costing two days).
The end-to-end test that proves rotation works uses a consumer that reads the file on each attempt, so it does not meet this.
Why it was not noticed
The gap is invisible from the control plane. A manifest carrying restart-on on a container
composes into a declaration without complaint and is refused on the machine, so the only way to
learn is to run one — which is how it was found, after nine of them had shipped across seven
modules.
What a fix has to keep
- Declared state, not a command.
restart-onis deliberately not restart this: it says the running thing must reflect these files, and the host works out that it does not. Whatever containers get must keep that shape, because the link may not carry an action (ADR 0005). - Recreate, not restart, where that is the honest verb. A container's environment is fixed at
creation. If what changed is an
env-file, restarting the container is not enough — it has to be made again. That is a different act from a service reload and should not be described as one. - It must not fire on every reconcile. A container that is recreated whenever the host looks at it is worse than one that never follows the file.
The near alternative, and why it is not enough
A module can avoid the problem by having its program read the file on each use rather than at start. That works for something written for this mesh, and is not available for a database, a forge or a mail system — which is the whole population this is about.