jschoubben
ab1bd5598e
225: a grant secret is composed with the account that reads it — the node's account for a bundle, the declared secrets-owner for a container. Zero EACCES since 12:30:10 where there had been 4330, both users created, mongodb logging Authentication succeeded for each. The harness also stops calling a permanent refusal a race, which is the half that cost three hours. 226: normalising moved to the records, where the provenance is known, and a reference the sweep will not address is skipped rather than ending the sweep. The first build after the roll-out collected 200 and said 1126 remain — the backlog falls with every build instead of standing at 1681 for ever. 227: the three photo modules publish the endpoint they declare, and a catalogue-wide test makes it a rule: a container publishes only a port its module declares, or the mesh has nothing to assign and the number escapes. 232 came out from under 225: photos asked for a database its user does not live in, invisible while no user existed at all.
5.0 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | ||
|---|---|---|---|---|---|---|
| resolved | 2026-10-04 |
|
mesh-controller#263 |
225 — A provisioner cannot read the grant secrets since its code left the container, and every consumer of it is unserved
What was observed
On the control machine, 2026-10-04, found while looking at why one app was restarting:
[mongodb] [provisioner:mongodb-database] mesh_novox_photos: secret not readable yet
(/var/lib/mongodb/grants/novox.photos.secret):
Error: EACCES: permission denied, open '/var/lib/mongodb/grants/novox.photos.secret'
4330 times, every five seconds, since 01:30:20. The consequence is not a log line: the provisioner never reads the password, so it never creates the user, so the consumer never connects —
UserNotFound: Could not find user "mesh_novox_photos" for db "admin"
— and the app crash-loops. Two consumers on this machine are in that state.
Why
The grant secrets are what the mesh seals for each consumer and the host unseals beside the
provider's contributions file. They are written -rw------- root root, which was right while a
module's own code ran in a container as root.
ADR 0198 moved a module's long-running code out of its container and under the node's runtime, which runs as the operator's account. The provisioner is now that account; the secret is still root's. The timestamps say it exactly: the files are dated 2026-09-26, the first refusal is 01:30:20 on the day the runtime rolled.
Nothing reports it. The machine applies cleanly and reads as current; the provisioner says
secret not readable yet, whose wording is for a real and ordinary race on the first pass — the
host has not written the file yet — and which is indistinguishable, in the log, from a permanent
refusal. Four thousand occurrences of a message that means "wait a moment" is the shape to
recognise.
Why it matters beyond this instance
This is every provider that provisions. The grant secret is the one file the sdk's harness reads for every consumer, so a provider that cannot read it serves nobody — and says so only in a line that reads like patience.
It is also the general question the runtime move leaves: what the mesh seals for a module is owned for the shape that module's code used to have. Each module whose code moved is a module whose files may now be unreadable to it, and ownership is the mesh's to state, not the module's to work around.
What this is not
Not caused by ADR 0202 or ADR 0189, which landed two to three hours after the first refusal. Those rebuilt the two affected consumers, which recreated their containers and made a silent fault visible as a restarting one. The dates are above.
Open questions
- Who owns a grant secret now — the module's account, as
secrets-owneralready says for a module's own secrets? Then the host writes it so, and this is a one-line statement in the declaration rather than a convention. - Should
secret not readable yetstop saying "yet" after the first few passes? A message that is right once and wrong four thousand times is a message that hides its own meaning. - Which other modules' files did the runtime move leave behind? The sweep is the same question for every path the mesh writes for a module: directories, bundles, received files.
Resolved, 2026-10-04
A grant secret is composed with the account that will read it: the node's account where the
module's code is a bundle the runtime runs, its declared secrets-owner where it is still a
container, root where it says neither. The same rule the module's own secrets already followed,
reaching the other kind of secret the mesh writes for a module (mesh-controller#263).
givenTo could not have reached these: it claims the files a bundle's words name, and the
harness composes a grant secret's path from the contributions file, which no word names.
And the harness stops calling a permanent refusal a race (mesh-sdk#22, 0.1.9). After about a minute it says so plainly, rarely rather than every five seconds, and names what to look at — who owns the file and who the process runs as. That is the half that cost three hours.
How it was checked: on the control machine, after the push — the grant secrets belong to the
operator's account, zero EACCES since 12:30:10 where there had been 4330, both users were
created, and mongodb-server logs Authentication succeeded for each.
It also uncovered issue 232, which this fault had been hiding: with no user anywhere, "not found in admin" was a complete account of this issue and said nothing about the consumer asking the wrong database.