From b3bd50c58825386e1e3047d86e2b4f646f7aecb3 Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 26 Sep 2026 00:44:10 +0200 Subject: [PATCH] Issue 120: a provisioner remembers what it did, not what is there The harness compares against its own memory, so a backend that loses what was provisioned (the cache's ACL users on a server restart) is never provisioned again, silently. --- .../00-report.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 04-ISSUES/120-a-provisioner-remembers-what-it-did-not-what-is/00-report.md diff --git a/04-ISSUES/120-a-provisioner-remembers-what-it-did-not-what-is/00-report.md b/04-ISSUES/120-a-provisioner-remembers-what-it-did-not-what-is/00-report.md new file mode 100644 index 0000000..9b697f3 --- /dev/null +++ b/04-ISSUES/120-a-provisioner-remembers-what-it-did-not-what-is/00-report.md @@ -0,0 +1,62 @@ +--- +status: located +opened: 2026-09-26 +located-in: [mesh-sdk src/provisioner, mesh-catalog modules/redis] +fixed-by: +amended-design: +--- + +# 120 — A provisioner remembers what it did, not what is there + +## What was observed + +The provisioner harness every provider is built on keeps, in memory, a hash of what it last applied +for each consumer: the login, the password and the values. On each pass it skips a consumer whose +hash has not changed. It never asks the backend whether what it made is still there. + +The cache module shows what that allows. Its server is configured with a password and a data +directory, and **no ACL file**. So the per-consumer ACL users its provisioner creates exist only in +the server's memory. The server and the provisioner run in separate containers: + +1. the provisioner creates an ACL user for each consumer, and records it as applied; +2. the server restarts, for an upgrade or a crash, and comes back with no consumer users; +3. the provisioner, still running, sees nothing changed in what it receives, and does nothing; +4. every consumer of the cache fails to authenticate, and **nothing reports it**. The provisioner's + log is quiet, and the mesh's status is green. + +The consumers recover only when the provisioner itself restarts, because its memory is then empty. +Rotating the cache's administrative password happens to cover it, because that file is mounted into +the provisioner too and recreates it. Nothing else does. + +Evidence, from the catalogue's and the SDK's main branches: the harness's reconcile loop (`applied`, +keyed by login, compared by hash before `create`), and the cache module's rendered configuration, +which names no ACL file. Found during research 016, how a credential can be rotated, proposed +alongside to-be 27. + +## Why it matters beyond this instance + +The cache is the case where the backend forgets on its own. The same gap opens whenever a backend +loses what was provisioned while the provisioner keeps running: a store restored from a backup taken +before a consumer was added, a login removed by hand, a server recreated on an empty data directory. +In each of them the provisioner reports that everything is applied, because it compares against its +own memory and not against the backend. + +The harness's other half has the same shape. A consumer's contribution that disappears while the +provisioner is down is never removed, because only logins the running process applied are candidates +for removal. What the mesh wants and what the backend holds can drift in both directions, and the +harness sees neither. + +This is the design permitting a silent failure. *A provider makes what its consumers require true* +is stated in [to-be 13](../../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md), and nothing +checks it after the first pass. + +## Open questions + +- Should the harness check each consumer's credential against the backend on every pass, or + periodically, instead of trusting its memory? Most adapters' `create` is already idempotent, so the + cheapest fix may be to drop the hash short-cut and apply every pass. What does that cost for a + provider that recreates an access key on every create, as the object store does? +- Should the cache keep its users in an ACL file, so a restart does not lose them? That fixes this + instance and leaves the gap for the others. +- Where does the record of what was applied live, if not in memory? ADR 0114, still + proposed, puts rotation state with the vault. The same place may answer this. -- 2.54.0