jschoubben
cb2117f1c4
Two birth-address outages and a registry that would have been the third; a container that keeps a stale environment after its file changes; a host command that applied a converged declaration to an adopted node; the hub and the vault without seats. And the decision the operator made under it all: the hub adopts the predecessor's tunnel in place, key and peers and range and port.
46 lines
2.3 KiB
Markdown
46 lines
2.3 KiB
Markdown
---
|
|
status: located
|
|
opened: 2026-09-23
|
|
located-in: [mesh-host internal/apply]
|
|
fixed-by:
|
|
amended-design:
|
|
---
|
|
|
|
# 103 — A container is not recreated when a file it reads changes
|
|
|
|
## What was observed
|
|
|
|
The control-node, 2026-09-23. The store was given the predecessor's port. The host rewrote the
|
|
forge's and the analytics service's environment files with the new port — correctly — and left
|
|
both containers running with the old one in their environment. Both lost their database. Both
|
|
stayed "up" and healthy-looking for the twenty minutes it took to notice, then answered 502.
|
|
|
|
A restart did not help: a container reads its `env-file` when it is **created**, not when it
|
|
starts, so `docker restart` handed both containers the same stale environment. Only removing them
|
|
and letting the host recreate them fixed it.
|
|
|
|
The host decides whether a container needs recreating by comparing a hash of its declared spec.
|
|
The spec names the env file's *path*; the file's *content* is not part of it. So a change that
|
|
alters everything the process will see alters nothing the host compares.
|
|
|
|
## Why it matters beyond this instance
|
|
|
|
Every module with an `env-file` — which is most of them — has a configuration the host writes and a
|
|
container that reads it once. Any change to that configuration that the host applies without
|
|
recreating the container is applied to the disk and not to the service. The mesh then reports the
|
|
node as running what it was told, because the file is right; only the process is wrong.
|
|
|
|
The ports move is the obvious trigger, and it is the migration's whole method. But a rotated
|
|
credential, a re-provisioned database, a changed binding — anything the host substitutes into a
|
|
file a container reads — has the same shape.
|
|
|
|
## Open questions
|
|
|
|
- Should the spec hash cover the **content** of every file the container mounts or reads, so a
|
|
changed file recreates it — accepting that every such change is a restart of the service?
|
|
- Or should the host recreate on content change only for `env-file` and mounted secrets, and leave
|
|
bind-mounted data alone — a file the service reads at start versus a directory it reads while
|
|
running?
|
|
- How does the host report the difference between "the file is right" and "the process has read
|
|
it"? Today it cannot, and that is what made this silent.
|