Research 016: survey how each provider can rotate a credential

Overlap as drafted in 0113 would have deleted consumer data: seven of
eight providers name the resource after the login and five drop it on
remove. Rotation is now undecided in 0113 and to-be 27, pending the
survey. Also: a requirement naming a seat resolves to its holder, a
person chooses among remaining candidates at assignment, the controller's
secrets are requirements of its definition, genesis seals to the
control-node key, and moving the vault or broker is break-glass.

01-RESEARCH/016-how-a-credential-can-be-rotated/00-overview.md

@@ -0,0 +1,51 @@
+---
+status: active
+initiated: 2026-09-26
+touches:
+  - 02-DECISIONS/0113-the-vault-makes-every-secret.md
+  - 02-DECISIONS/0049-a-consumers-identity-fits-the-tightest-backend.md
+  - 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
+  - 03-DESIGN/01-to-be/13-credentials-and-their-rotation.md
+  - 03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md
+  - 04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md
+---
+
+# 016 — How a credential can be rotated
+
+**What.** Which rotation mechanisms the mesh's providers can actually support, measured against
+their code rather than assumed. Every provider in the catalogue was read: how it names what it
+makes for a consumer, what its remove destroys, whether it re-applies a password, whether its
+backend can hold two secrets for one login or two logins on one resource, and how its own
+administrative credential is set. The consumer side was read too: when a module reads a secret, and
+what makes it read a new one.
+
+**Why.** [ADR 0113](../../02-DECISIONS/0113-the-vault-makes-every-secret.md), as first drafted,
+chose *overlap*: add a second login beside the first, move every reader, then remove the old one,
+"through the adapter's existing create and remove", with "no consumer changes". A review showed that
+claim false. In most providers the consumer's data is named after its login, and remove drops the data
+with the login. Overlap as written would have deleted every consumer's database on its first
+rotation. The mechanism has to be chosen on what the providers do.
+
+**What it touches.** Rotation in 0113 and [to-be 27](../../03-DESIGN/01-to-be/27-a-module-requires-the-mesh-resolves.md),
+which both mark it undecided and point here. The identity budget in
+[ADR 0049](../../02-DECISIONS/0049-a-consumers-identity-fits-the-tightest-backend.md), if a consumer
+gets two logins. The rotation already implemented, which [to-be 13](../../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)
+describes.
+
+**Documents.**
+
+- [01 — The providers](01-the-providers.md): the survey, one row per provider, and what it shows.
+- [02 — The readers](02-the-readers.md): how a secret reaches a running process, and what already
+  recreates it.
+- [03 — The options](03-the-options.md): each rotation mechanism against those facts, and a
+  recommendation.
+
+**Finding, in one paragraph.** All eight credential providers already re-apply a consumer's password
+in place on every create, and the controller's `rotate` command relies on that. It is a working
+rotation with a stated window. Seven of the eight name the consumer's resource after its login,
+and five drop the resource when they remove the login, so a second login is impossible without
+changing the adapter. Only one backend holds two passwords on one login. But every backend can grant
+two logins the same rights over one resource. So overlap is possible everywhere, but only after each
+adapter separates *the consumer's resource* from *the login that reaches it*. Administrative
+credentials are a different case. They have one party, a fixed name, and in three providers they are
+taken only at first initialisation.

01-RESEARCH/016-how-a-credential-can-be-rotated/01-the-providers.md

@@ -0,0 +1,71 @@
+# 01 — The providers
+
+Read from the catalogue's main branch: each provider's provisioner adapter (`create`, `remove`),
+the client functions they call, and each definition's own credentials. The provisioner harness in
+`mesh-sdk` calls `create` for a consumer when its contribution appears or changes, and after the
+provisioner restarts, and `remove` when the contribution goes. Its record of what was applied is
+kept in memory.
+
+## The credential providers
+
+`login` is the consumer's derived identity, which the adapter receives as `as`
+([ADR 0049](../../02-DECISIONS/0049-a-consumers-identity-fits-the-tightest-backend.md)).
+
+| provider | the consumer's resource is named | remove destroys | create re-applies the password | two secrets on one login | two logins on one resource |
+|---|---|---|---|---|---|
+| postgres | a database named `login`, owned by the role `login` | the database and the role | yes, `ALTER ROLE … PASSWORD` when the role exists | no: a role has one password | yes, both as members of one role that owns the database. Not done today |
+| mssql | a database named `login`, with the login mapped into it | the database and the login | yes, `ALTER LOGIN … WITH PASSWORD` | no: a login has one password | yes, two logins mapped to users in `db_owner`. Not done today |
+| mongodb | a database named `login`, with a user holding `dbOwner` | the database and the user | yes, `updateUser` with the new password | no: a user has one credential | yes, two users with `dbOwner` on one database. Not done today |
+| redis | the key prefix `login:` on an ACL user named `login` | the user, **not** its keys | yes: `ACL SETUSER … reset … >password` replaces all of them | **yes**: an ACL user holds several passwords, added with `>` and removed with `<`. Today's `reset` discards all but the new one | yes, two users on one key prefix, once the prefix is not the login |
+| minio | a bucket derived from `login`, and a service account whose access key is `login` | the access key and the bucket | yes, by removing the access key and adding it again | no, but an access key *is* the login: a second key is a second login | yes, two service accounts with one bucket policy. The access key is capped at 20 characters |
+| lavinmq | a virtual host named `login`, and a user named `login` with permissions on it | the virtual host and the user | yes, the user is written again with the password | no: a user has one password | yes, permissions for two users on one virtual host |
+| mosquitto | a client named `login`, with a role named for it on the topic prefix `login/#` | the client and its role | yes, the password is set when the client exists | no: a client has one password | yes, two clients holding one role, once the prefix is not the login. An MQTT client identifier still has to be unique per connection |
+| gitea (npm) | a user named `login` on a team of an organisation that owns every package | the user; **packages survive**, because the organisation owns them | yes, the user's password is set on every run | no for the password; a user can hold several access tokens | yes, trivially: a second member of the same team |
+
+## The other providers
+
+| provider | answers with | credential |
+|---|---|---|
+| umami | a website, found by its public name | none. The site id it makes has no way back to the consumer today |
+| cloudflare-dns | a public name derived from `login` | none handed to the consumer; its own API token is an operator value |
+| showcase | a route | none |
+| mesh-vault | custody: it records and withdraws sealed values in a ledger | it holds secrets; it makes none today |
+
+## Each provider's own administrative credential
+
+| provider | identity | how the backend takes it |
+|---|---|---|
+| postgres | a fixed superuser | from a file **only at first initialisation**. Afterwards the file is read by the provisioner to connect, and changing it changes nothing in the database |
+| mssql | `sa` | from the environment at first setup. The image documents no file form, and the definition records that as a declared exception |
+| mongodb | a fixed `root` | from a file **only at first initialisation**, when the data directory is empty |
+| redis | the default user | from `requirepass` in a configuration the mesh renders, read when the server starts |
+| minio | a fixed root user | from a file, read when the server starts |
+| lavinmq | a fixed admin name | from the module's own secret, which its provisioner connects with |
+| mosquitto | a fixed admin client | from the module's own secret, which its provisioner connects with; the broker's dynamic-security file holds it |
+
+Every provider module also has its own bus account, an own secret, read at start.
+
+## What the table shows
+
+1. **Every credential provider already rotates in place.** All eight re-apply the password on the
+   same login each time `create` runs. The controller's `rotate` command relies on that: it replaces
+   the credential in the inventory and sends both ends in one push. Its own comments state the window,
+   between the provider applying and the consumer restarting, in which the consumer cannot
+   authenticate.
+2. **Seven of eight name the consumer's resource after its login.** Only gitea separates them,
+   because an organisation owns the packages. A second login therefore has no resource of its own to
+   reach, and cannot share the first one's without the adapter granting it.
+3. **Five of eight destroy the resource when they remove the login.** These are postgres, mssql, mongodb,
+   minio and lavinmq. In today's adapters, *retire a login* and *delete the consumer's data* are one call.
+   This is the fault the first overlap draft would have triggered.
+4. **Only redis holds two passwords on one login**, and gitea through tokens rather than its
+   password. A rotation built on two secrets per login would work for one provider out of eight.
+5. **Every backend can give two logins the same rights over one resource.** Group roles in postgres,
+   database roles in mssql and mongodb, permissions in lavinmq, a shared role in mosquitto, a shared
+   policy in minio, a shared key prefix in redis, a shared team in gitea. No adapter does it today.
+6. **The administrative credentials have one party and a fixed name.** The provider module is both the
+   one that applies the credential and the only one that reads it. In postgres, mongodb and mssql, a new
+   value only takes effect through a command run with the old value. A restart changes nothing.
+7. **A consumer's identity is already the resource's name.** The login is derived from the
+   assignment, which is a module on a node, so the current login and "the consumer" are the same
+   string today. A second login would need a new name. The resource can keep the one it has.

01-RESEARCH/016-how-a-credential-can-be-rotated/02-the-readers.md

@@ -0,0 +1,42 @@
+# 02 — The readers
+
+How a secret reaches a running process, and what makes the process take a new one.
+
+## No module watches a secret
+
+A search of every module's code in the catalogue found no file watching of any kind, and no
+re-reading of a secret while running. **Every reader reads a secret when it starts.** There is no
+consumer that takes a new value live, so every rotation that changes what a consumer presents ends in
+the consumer restarting.
+
+## The host already recreates what read a changed file
+
+The node host records, for every long-running container, the digest of each file it read when it
+was created: its env-files, and every file bind-mounted into it directly. When a digest changes, the
+host recreates the container, even though its spec is otherwise unchanged. This is the fix for
+[issue 103](../../04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md).
+It is on the host's main branch, while the issue is still recorded as located, not fixed.
+
+Two cases are deliberately left out and need `restart-on` in the definition:
+
+- a file read out of a **mounted directory**, because the host cannot know whether the service reads
+  it once or watches it (a route proxy re-reads its routes live; a provisioner polls what it receives);
+- a **process** rather than a container.
+
+## What the catalogue does with it
+
+22 definitions declare a secret they receive. In 16 of them it reaches the service through a
+rendered file, usually an env-file. That case the host already covers. 14 declare `restart-on` for
+something. Whether each of the 22 is fully covered depends on how its secret travels: through an
+env-file or a direct mount, which the host covers, or through a directory or into a process, which
+needs `restart-on`. **That was not classified module by module.** It is the check to run before a
+rotation mechanism relies on it.
+
+## What this means for rotation
+
+- The *read at start* half of 0113's recipient model is already true, and mostly already handled by
+  the host. The restart is derived from the files a container reads, not declared per secret.
+- Any mechanism, in place or overlapping, ends with the reader being recreated. What differs is
+  whether the credential it held until then still works.
+- For a single-party secret, a module's own, the reader is also the only holder. There is nobody to
+  overlap with, and delivering the new file recreates the reader.

01-RESEARCH/016-how-a-credential-can-be-rotated/03-the-options.md

@@ -0,0 +1,75 @@
+# 03 — The options
+
+Three mechanisms, weighed against [01](01-the-providers.md) and [02](02-the-readers.md).
+
+## A. In place, as today
+
+The vault makes a new value. Every applier re-applies it on the same login, which all eight
+providers already do. Every reader is recreated by the host.
+
+- **Works with:** every provider, unchanged. It is what `rotate` does now.
+- **Costs:** a window per consumer, from the provider applying to the consumer being recreated. They
+  are on different machines, and nothing orders them. A reader whose machine is unreachable from the
+  mesh but still reaches its provider stays locked out until the mesh reaches it again.
+- **Admin credentials:** the natural form. The provider module is the only party, and it has to
+  apply the new value with the old one anyway (finding 6).
+
+## B. Two secrets on one login
+
+The applier adds the new password beside the old one, readers move, and the old one is removed.
+
+- **Works with:** redis natively, and gitea through tokens. **Not** with the other six, whose backends
+  hold one password per login (finding 4).
+- **Verdict:** not a mechanism, a special case. Using it where it exists and something else
+  elsewhere is the "this way or that way" the design is trying to remove.
+
+## C. Two logins per consumer, over one resource
+
+The consumer has two logins derived by the mesh, and uses one at a time. The applier creates the other
+with the new value and grants it the same rights over the consumer's resource. Readers move to it,
+and then the old login is retired, which removes the login only, never the resource.
+
+- **Works with:** every backend (finding 5), **after** each adapter changes:
+  - the resource is named after the consumer, not the login. Today the two are the same string (finding
+    7), so existing resources keep their names, the current login stays one of the two, and only the
+    second is new;
+  - both logins get the same rights, through a group role or its equivalent;
+  - *retire a login* and *remove the consumer* become two operations. Today they are one call, and
+    in five providers that call destroys data (finding 3). This is the whole of the danger, and it has
+    to be split, whatever else is chosen.
+- **Costs:**
+  - seven adapters change;
+  - the harness learns the alternation and a confirmation per step;
+  - the second login's name must fit the tightest backend. That is 20 characters for a minio access
+    key ([ADR 0049](../../02-DECISIONS/0049-a-consumers-identity-fits-the-tightest-backend.md)), and
+    a suffix spends part of it;
+  - an MQTT client identifier stays unique per connection, so mosquitto needs the client id kept apart
+    from the login.
+- **Gains:** no window. A reader that cannot be reached keeps a working login until it can.
+- **Does not apply to** single-party secrets: admin credentials and a module's own secrets. There is
+  no second party to overlap with.
+
+## Independent of the choice
+
+- **Split remove.** Retiring a credential must never be able to destroy a consumer's data. That holds
+  under A too, because A's remove is the same call. A remove that drops a database should be a
+  separate, explicit operation, which [ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)
+  already implies: data outlives the declaration.
+- **Admin credentials are applied by their own provider**, using the old value. That happens in place
+  whichever mechanism consumers get. Where a backend takes the value only at first initialisation, the
+  file alone changes nothing, and rotation needs the provider's provisioner to run the change.
+- **Classify the 22 readers** ([02](02-the-readers.md)) before relying on derived restarts.
+
+## Recommendation
+
+- **Consumer credentials: C**, because it is the only mechanism every backend supports, and it closes
+  the window instead of shortening it. Its prerequisite, separating the resource from the login and
+  retiring a login from removing a consumer, is worth doing on its own, because it removes a
+  data-loss path that exists today.
+- **Single-party secrets (admin credentials, a module's own): A.** In place, applied by the provider
+  that holds them.
+- **Until the adapters are changed, A stays** as `rotate` implements it, with its window stated. It is
+  not replaced by a mechanism the providers cannot yet carry.
+
+This is two mechanisms, split by a property of the secret rather than by provider: whether it has one
+party or two. Every provider is treated the same way for the same kind of secret.