hq/04-ISSUES/120-a-provisioner-remembers-what-it-did-not-what-is/00-report.md

status, opened, located-in, fixed-by, amended-design

status	opened	located-in	fixed-by	amended-design
located	2026-09-26	mesh-sdk src/provisioner mesh-catalog modules/redis

120 — A provisioner remembers what it did, not what is there

What was observed

The provisioner harness every provider is built on keeps, in memory, a hash of what it last applied for each consumer: the login, the password and the values. On each pass it skips a consumer whose hash has not changed. It never asks the backend whether what it made is still there.

The cache module shows what that allows. Its server is configured with a password and a data directory, and no ACL file. So the per-consumer ACL users its provisioner creates exist only in the server's memory. The server and the provisioner run in separate containers:

1. the provisioner creates an ACL user for each consumer, and records it as applied;
2. the server restarts, for an upgrade or a crash, and comes back with no consumer users;
3. the provisioner, still running, sees nothing changed in what it receives, and does nothing;
4. every consumer of the cache fails to authenticate, and nothing reports it. The provisioner's log is quiet, and the mesh's status is green.

The consumers recover only when the provisioner itself restarts, because its memory is then empty. Rotating the cache's administrative password happens to cover it, because that file is mounted into the provisioner too and recreates it. Nothing else does.

Evidence, from the catalogue's and the SDK's main branches: the harness's reconcile loop (applied, keyed by login, compared by hash before create), and the cache module's rendered configuration, which names no ACL file. Found during research 016, how a credential can be rotated, proposed alongside to-be 27.

Why it matters beyond this instance

The cache is the case where the backend forgets on its own. The same gap opens whenever a backend loses what was provisioned while the provisioner keeps running: a store restored from a backup taken before a consumer was added, a login removed by hand, a server recreated on an empty data directory. In each of them the provisioner reports that everything is applied, because it compares against its own memory and not against the backend.

The harness's other half has the same shape. A consumer's contribution that disappears while the provisioner is down is never removed, because only logins the running process applied are candidates for removal. What the mesh wants and what the backend holds can drift in both directions, and the harness sees neither.

This is the design permitting a silent failure. A provider makes what its consumers require true is stated in to-be 13, and nothing checks it after the first pass.

Open questions

• Should the harness check each consumer's credential against the backend on every pass, or periodically, instead of trusting its memory? Most adapters' create is already idempotent, so the cheapest fix may be to drop the hash short-cut and apply every pass. What does that cost for a provider that recreates an access key on every create, as the object store does?
• Should the cache keep its users in an ACL file, so a restart does not lose them? That fixes this instance and leaves the gap for the others.
• Where does the record of what was applied live, if not in memory? ADR 0114, still proposed, puts rotation state with the vault. The same place may answer this.