jschoubben
bc64c5c187
The derived-value record is renumbered a second time: the key-value-buckets record took 0201 while this waited to merge, as the bundles record took 0188 before it. Both times free when chosen, taken by the time it landed. cycle.py caught it; three repositories cite this record, so the number matters. 225 — a provisioner has not been able to read its grant secrets since 01:30, when a module's own code left its container and the files stayed root's. Four thousand refusals, each worded as patience, and two consumers unserved. Not from ADR 0202 or 0189, which landed hours later; dates in the report. 226 — the store's sweep stops at the first reference recorded with an address and collects nothing. A guard that cannot tell 'I will not ask about this' from 'it would not answer' stops the wrong amount of work. 227 — the photo app's admin client asks for the port the proxy holds. A module pinned months behind carries everything its branch gained, the first time anything makes it move.
76 lines
3.5 KiB
Markdown
76 lines
3.5 KiB
Markdown
---
|
|
status: open
|
|
opened: 2026-10-04
|
|
located-in: [mesh-host internal/apply, mesh-controller internal/catalogue]
|
|
fixed-by:
|
|
amended-design:
|
|
---
|
|
|
|
# 225 — A provisioner cannot read the grant secrets since its code left the container, and every consumer of it is unserved
|
|
|
|
## What was observed
|
|
|
|
On the control machine, 2026-10-04, found while looking at why one app was restarting:
|
|
|
|
```
|
|
[mongodb] [provisioner:mongodb-database] mesh_novox_photos: secret not readable yet
|
|
(/var/lib/mongodb/grants/novox.photos.secret):
|
|
Error: EACCES: permission denied, open '/var/lib/mongodb/grants/novox.photos.secret'
|
|
```
|
|
|
|
**4330 times, every five seconds, since 01:30:20.** The consequence is not a log line: the
|
|
provisioner never reads the password, so it never creates the user, so the consumer never
|
|
connects —
|
|
|
|
```
|
|
UserNotFound: Could not find user "mesh_novox_photos" for db "admin"
|
|
```
|
|
|
|
— and the app crash-loops. Two consumers on this machine are in that state.
|
|
|
|
## Why
|
|
|
|
The grant secrets are what the mesh seals for each consumer and the host unseals beside the
|
|
provider's contributions file. They are written `-rw------- root root`, which was right while a
|
|
module's own code ran in a container as root.
|
|
|
|
[ADR 0198](../../02-DECISIONS/0198-a-modules-long-running-code-is-launched-by-the-node-runtime-and-reaches-the-bus-through-it.md)
|
|
moved a module's long-running code out of its container and under the node's runtime, which runs
|
|
as the operator's account. The provisioner is now that account; the secret is still root's. The
|
|
timestamps say it exactly: the files are dated 2026-09-26, the first refusal is 01:30:20 on the
|
|
day the runtime rolled.
|
|
|
|
**Nothing reports it.** The machine applies cleanly and reads as current; the provisioner says
|
|
`secret not readable yet`, whose wording is for a real and ordinary race on the first pass — the
|
|
host has not written the file yet — and which is indistinguishable, in the log, from a permanent
|
|
refusal. Four thousand occurrences of a message that means "wait a moment" is the shape to
|
|
recognise.
|
|
|
|
## Why it matters beyond this instance
|
|
|
|
This is every provider that provisions. The grant secret is the one file the sdk's harness reads
|
|
for every consumer, so a provider that cannot read it serves nobody — and says so only in a line
|
|
that reads like patience.
|
|
|
|
It is also the general question the runtime move leaves: **what the mesh seals for a module is
|
|
owned for the shape that module's code used to have.** Each module whose code moved is a module
|
|
whose files may now be unreadable to it, and ownership is the mesh's to state, not the module's
|
|
to work around.
|
|
|
|
## What this is not
|
|
|
|
Not caused by [ADR 0202](../../02-DECISIONS/0202-a-provider-declares-what-it-derives-for-each-consumer.md)
|
|
or [ADR 0189](../../02-DECISIONS/0189-the-store-keeps-what-the-records-name.md), which landed two
|
|
to three hours after the first refusal. Those rebuilt the two affected consumers, which recreated
|
|
their containers and made a silent fault visible as a restarting one. The dates are above.
|
|
|
|
## Open questions
|
|
|
|
- Who owns a grant secret now — the module's account, as `secrets-owner` already says for a
|
|
module's own secrets? Then the host writes it so, and this is a one-line statement in the
|
|
declaration rather than a convention.
|
|
- Should `secret not readable yet` stop saying "yet" after the first few passes? A message that
|
|
is right once and wrong four thousand times is a message that hides its own meaning.
|
|
- Which other modules' files did the runtime move leave behind? The sweep is the same question
|
|
for every path the mesh writes for a module: directories, bundles, received files.
|