Issue 120: a provisioner remembers what it did, not what is there

The harness compares against its own memory, so a backend that loses what was provisioned (the cache's ACL users on a server restart) is never provisioned again, silently.
2026-09-26 00:44:10 +02:00
parent 56669ee23b
commit b3bd50c588
1 changed files with 62 additions and 0 deletions
@@ -0,0 +1,62 @@
+---
+status: located
+opened: 2026-09-26
+located-in: [mesh-sdk src/provisioner, mesh-catalog modules/redis]
+fixed-by:
+amended-design:
+---
+
+# 120 — A provisioner remembers what it did, not what is there
+
+## What was observed
+
+The provisioner harness every provider is built on keeps, in memory, a hash of what it last applied
+for each consumer: the login, the password and the values. On each pass it skips a consumer whose
+hash has not changed. It never asks the backend whether what it made is still there.
+
+The cache module shows what that allows. Its server is configured with a password and a data
+directory, and **no ACL file**. So the per-consumer ACL users its provisioner creates exist only in
+the server's memory. The server and the provisioner run in separate containers:
+
+1. the provisioner creates an ACL user for each consumer, and records it as applied;
+2. the server restarts, for an upgrade or a crash, and comes back with no consumer users;
+3. the provisioner, still running, sees nothing changed in what it receives, and does nothing;
+4. every consumer of the cache fails to authenticate, and **nothing reports it**. The provisioner's
+   log is quiet, and the mesh's status is green.
+
+The consumers recover only when the provisioner itself restarts, because its memory is then empty.
+Rotating the cache's administrative password happens to cover it, because that file is mounted into
+the provisioner too and recreates it. Nothing else does.
+
+Evidence, from the catalogue's and the SDK's main branches: the harness's reconcile loop (`applied`,
+keyed by login, compared by hash before `create`), and the cache module's rendered configuration,
+which names no ACL file. Found during research 016, how a credential can be rotated, proposed
+alongside to-be 27.
+
+## Why it matters beyond this instance
+
+The cache is the case where the backend forgets on its own. The same gap opens whenever a backend
+loses what was provisioned while the provisioner keeps running: a store restored from a backup taken
+before a consumer was added, a login removed by hand, a server recreated on an empty data directory.
+In each of them the provisioner reports that everything is applied, because it compares against its
+own memory and not against the backend.
+
+The harness's other half has the same shape. A consumer's contribution that disappears while the
+provisioner is down is never removed, because only logins the running process applied are candidates
+for removal. What the mesh wants and what the backend holds can drift in both directions, and the
+harness sees neither.
+
+This is the design permitting a silent failure. *A provider makes what its consumers require true*
+is stated in [to-be 13](../../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md), and nothing
+checks it after the first pass.
+
+## Open questions
+
+- Should the harness check each consumer's credential against the backend on every pass, or
+  periodically, instead of trusting its memory? Most adapters' `create` is already idempotent, so the
+  cheapest fix may be to drop the hash short-cut and apply every pass. What does that cost for a
+  provider that recreates an access key on every create, as the object store does?
+- Should the cache keep its users in an ACL file, so a restart does not lose them? That fixes this
+  instance and leaves the gap for the others.
+- Where does the record of what was applied live, if not in memory? ADR 0114, still
+  proposed, puts rotation state with the vault. The same place may answer this.