2026-09-26 12:14:18 +00:00
1 changed files with 62 additions and 0 deletions
@@ -0,0 +1,62 @@
 ---
 status: located
 opened: 2026-09-26
 located-in: [mesh-sdk src/provisioner, mesh-catalog modules/redis]
 fixed-by:
 amended-design:
 ---
 # 120 — A provisioner remembers what it did, not what is there
 ## What was observed
 The provisioner harness every provider is built on keeps, in memory, a hash of what it last applied
 for each consumer: the login, the password and the values. On each pass it skips a consumer whose
 hash has not changed. It never asks the backend whether what it made is still there.
 The cache module shows what that allows. Its server is configured with a password and a data
 directory, and **no ACL file**. So the per-consumer ACL users its provisioner creates exist only in
 the server's memory. The server and the provisioner run in separate containers:
 1. the provisioner creates an ACL user for each consumer, and records it as applied;
 2. the server restarts, for an upgrade or a crash, and comes back with no consumer users;
 3. the provisioner, still running, sees nothing changed in what it receives, and does nothing;
 4. every consumer of the cache fails to authenticate, and **nothing reports it**. The provisioner's
   log is quiet, and the mesh's status is green.
 The consumers recover only when the provisioner itself restarts, because its memory is then empty.
 Rotating the cache's administrative password happens to cover it, because that file is mounted into
 the provisioner too and recreates it. Nothing else does.
 Evidence, from the catalogue's and the SDK's main branches: the harness's reconcile loop (`applied`,
 keyed by login, compared by hash before `create`), and the cache module's rendered configuration,
 which names no ACL file. Found during research 016, how a credential can be rotated, proposed
 alongside to-be 27.
 ## Why it matters beyond this instance
 The cache is the case where the backend forgets on its own. The same gap opens whenever a backend
 loses what was provisioned while the provisioner keeps running: a store restored from a backup taken
 before a consumer was added, a login removed by hand, a server recreated on an empty data directory.
 In each of them the provisioner reports that everything is applied, because it compares against its
 own memory and not against the backend.
 The harness's other half has the same shape. A consumer's contribution that disappears while the
 provisioner is down is never removed, because only logins the running process applied are candidates
 for removal. What the mesh wants and what the backend holds can drift in both directions, and the
 harness sees neither.
 This is the design permitting a silent failure. *A provider makes what its consumers require true*
 is stated in [to-be 13](../../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md), and nothing
 checks it after the first pass.
 ## Open questions
 - Should the harness check each consumer's credential against the backend on every pass, or
  periodically, instead of trusting its memory? Most adapters' `create` is already idempotent, so the
  cheapest fix may be to drop the hash short-cut and apply every pass. What does that cost for a
  provider that recreates an access key on every create, as the object store does?
 - Should the cache keep its users in an ACL file, so a restart does not lose them? That fixes this
  instance and leaves the gap for the others.
 - Where does the record of what was applied live, if not in memory? ADR 0114, still
  proposed, puts rotation state with the vault. The same place may answer this.