jschoubben
14be8576f8
088, 089, 120, 128 and 130 each name a commit that is on main and cites them — the forge's address following a moved port, a route naming its endpoint, a provisioner asking the backend what is there, the hosts file written into a marked block, and undeclaring giving a unit back the state it was found in. Each says it was closed by reading commits rather than by a run, so nobody reads a green that was never measured. 129 stays located on purpose: ca-trust is merged and no machine holds it, so the symptom it opened on is still true everywhere.
69 lines
3.9 KiB
Markdown
69 lines
3.9 KiB
Markdown
---
|
|
status: resolved
|
|
opened: 2026-09-26
|
|
located-in: [mesh-sdk src/provisioner, mesh-catalog modules/redis]
|
|
fixed-by: mesh-catalog bbda88c, merged in #84 — every credential provider says whether it still holds a consumer
|
|
amended-design:
|
|
---
|
|
|
|
# 120 — A provisioner remembers what it did, not what is there
|
|
|
|
## What was observed
|
|
|
|
The provisioner harness every provider is built on keeps, in memory, a hash of what it last applied
|
|
for each consumer: the login, the password and the values. On each pass it skips a consumer whose
|
|
hash has not changed. It never asks the backend whether what it made is still there.
|
|
|
|
The cache module shows what that allows. Its server is configured with a password and a data
|
|
directory, and **no ACL file**. So the per-consumer ACL users its provisioner creates exist only in
|
|
the server's memory. The server and the provisioner run in separate containers:
|
|
|
|
1. the provisioner creates an ACL user for each consumer, and records it as applied;
|
|
2. the server restarts, for an upgrade or a crash, and comes back with no consumer users;
|
|
3. the provisioner, still running, sees nothing changed in what it receives, and does nothing;
|
|
4. every consumer of the cache fails to authenticate, and **nothing reports it**. The provisioner's
|
|
log is quiet, and the mesh's status is green.
|
|
|
|
The consumers recover only when the provisioner itself restarts, because its memory is then empty.
|
|
Rotating the cache's administrative password happens to cover it, because that file is mounted into
|
|
the provisioner too and recreates it. Nothing else does.
|
|
|
|
Evidence, from the catalogue's and the SDK's main branches: the harness's reconcile loop (`applied`,
|
|
keyed by login, compared by hash before `create`), and the cache module's rendered configuration,
|
|
which names no ACL file. Found during research 016, how a credential can be rotated, proposed
|
|
alongside to-be 27.
|
|
|
|
## Why it matters beyond this instance
|
|
|
|
The cache is the case where the backend forgets on its own. The same gap opens whenever a backend
|
|
loses what was provisioned while the provisioner keeps running: a store restored from a backup taken
|
|
before a consumer was added, a login removed by hand, a server recreated on an empty data directory.
|
|
In each of them the provisioner reports that everything is applied, because it compares against its
|
|
own memory and not against the backend.
|
|
|
|
The harness's other half has the same shape. A consumer's contribution that disappears while the
|
|
provisioner is down is never removed, because only logins the running process applied are candidates
|
|
for removal. What the mesh wants and what the backend holds can drift in both directions, and the
|
|
harness sees neither.
|
|
|
|
This is the design permitting a silent failure. *A provider makes what its consumers require true*
|
|
is stated in [to-be 13](../../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md), and nothing
|
|
checks it after the first pass.
|
|
|
|
## Open questions
|
|
|
|
- Should the harness check each consumer's credential against the backend on every pass, or
|
|
periodically, instead of trusting its memory? Most adapters' `create` is already idempotent, so the
|
|
cheapest fix may be to drop the hash short-cut and apply every pass. What does that cost for a
|
|
provider that recreates an access key on every create, as the object store does?
|
|
- Should the cache keep its users in an ACL file, so a restart does not lose them? That fixes this
|
|
instance and leaves the gap for the others.
|
|
- Where does the record of what was applied live, if not in memory? ADR 0114, still
|
|
proposed, puts rotation state with the vault. The same place may answer this.
|
|
|
|
## Closed
|
|
|
|
*2026-09-29, in a grooming pass rather than by whoever fixed it.* A provisioner asks the backend what is there rather than trusting what it remembers doing. Found by
|
|
reading what the code repositories' commits cite: the fix names this issue and is on `main`. It was
|
|
not re-verified on a machine, and this record says so rather than implying a run that did not happen.
|