hq/02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md

---
topic: what runs on it
status: proposed
date: 2026-09-26
deciders: jochen
reconstructed: false
extends: 0113-the-vault-makes-every-secret.md
---

# 114. A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached

## Context

[ADR 0113](0113-the-vault-makes-every-secret.md) decides who asks for a rotation (an operator, or the
vault's policy) and who makes the new value (the vault). It leaves open how old and new change over.
[Research 016](../01-RESEARCH/016-how-a-credential-can-be-rotated/00-overview.md) read every
credential provider in the catalogue against its code. There are nine:

- **all nine re-apply a password in place**, on the same login, every time they run. The controller's
  `rotate` command relies on that, and states the window it leaves: between the provider applying the
  new value and the consumer restarting with it, the consumer cannot authenticate;
- **eight of nine name the consumer's resource after its login**: a database, a bucket, a virtual host,
  a key prefix, a topic prefix, a mailbox. Only the forge's npm registry keeps them apart, because an
  organisation owns the packages;
- **five of nine destroy the consumer's data when they remove its login**: postgres, mssql and mongodb
  drop the database, lavinmq drops the virtual host with its queued messages, and mailu deletes the
  mailbox with its mail. In those adapters, *retire a login* and *delete the consumer's data* are one
  call. minio drops a bucket only if it is empty. The provisioner harness makes it worse: a consumer
  whose derived login changed is removed under the old login and created under the new one, in one pass;
- **one backend holds two passwords on one login** (redis), and two hold several tokens beside one
  password (the forge and mailu);
- **eight of nine can give two logins the same rights over one resource**. mailu cannot, because a mail
  user *is* its mailbox. It can give one user several tokens. In postgres, a second login is not enough
  on its own: objects belong to whichever login created them, so the resource must be owned by a role of
  its own;
- **an administrative credential has one party and a fixed name.** The provider module both applies it
  and reads it. Five backends take it only at first initialisation: postgres, mssql, mongodb, mosquitto
  and lavinmq. Their credential file is mounted directly into both the server and the provisioner, so
  replacing the file recreates the provisioner holding only the new value, which the backend does not
  know yet. The provisioner is then locked out;
- **no module watches a secret.** Every reader reads at start, and the host recreates a container when
  a file it read at creation changes
  ([issue 103](../04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md)).

A first draft of 0113 chose to overlap old and new "through the adapter's existing create and
remove". In five providers, that remove deletes the consumer's data. The mechanism has to be chosen on
what the providers do, and the danger has to be closed whichever mechanism is chosen.

## Considered Options

**1. In place for everything, as today.** Works with every provider unchanged. Rejected for credentials
two parties hold. The window cannot be closed, only shortened, and the two ends are on different
machines with nothing ordering them. For an administrative credential, it locks the provisioner out.

**2. Two secrets on one login.** Rejected as the mechanism. It works for three providers out of nine,
and using it there and something else elsewhere would put the difference in the mesh instead of in the
adapter.

**3. Two logins over one resource.** Rejected as the mechanism. It works for eight of nine, and not for
mailu.

**4. Two credentials over one resource, with the adapter choosing what a credential is.** A credential
is what a consumer presents, a login and a secret. The mesh alternates between two of them. Each adapter
makes the second one the way its backend can: a second login for eight providers, a second token on the
same login for mailu. A credential a single party holds is staged in place instead, and retiring a
credential is separated from removing a consumer before either is used. Chosen.

## Decision

### Retiring a credential never removes what it reached

**A provider's adapter keeps two things apart that today are one:** the consumer's *resource* (its
database, bucket, virtual host, key or topic prefix, mailbox) and a *credential* that reaches it.
They get separate operations:

- **ensure the resource**, named after the consumer;
- **ensure a credential** with a value, holding the consumer's rights over its resource;
- **retire a credential**. Anything it owns moves first to the resource's owner, and any session it has
  open is ended. Then the credential is removed, and nothing else;
- **remove the consumer**, which is what removes the resource, and retires every credential it has.

**Remove the consumer runs only when the consumer no longer requires the provision from this provider.**
That happens when its assignment goes, when its definition drops the requirement, or when re-resolution
sends it to another provider. It never runs because a login or a value changed. The harness keys what it
applied by the consumer, not by the login, so a changed login is a credential change and never a removal.
What removing a resource does with the data in it stays
[ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)'s, and re-resolving to another provider
moves no data.

**The resource is named after the consumer, and owned by the resource, not by a login.** A consumer's
identity is derived from its assignment ([ADR 0049](0049-a-consumers-identity-fits-the-tightest-backend.md)),
and today its login is that same string, so **no existing resource is renamed**. Where a backend makes
whatever a login creates the login's own, as postgres does, the resource is owned by a role that cannot
log in, and each credential works as that role. Ownership of an existing resource moves to it once. A
credential is retired by handing what it owns to that role, never by dropping what it owns.

### A credential two parties hold rotates over two credentials

**Two parties** means an applier and a reader that are different modules, or a module and a node's
host. The vault's custody copy does not count, because the vault holds every secret. So this covers a
credential between a consumer and a provider, and every bus account: a module's or a host's, applied by
the broker's provisioner and read by its owner. **Each consumer has two credentials, one in use at a
time**, both holding the same rights over the one resource. For eight providers the second is a second
login, derived by the mesh as the consumer's identity with a short fixed suffix. For mailu it is a
second token on the same login.

**The vault drives each rotation and records every step durably.** A provisioner learns which
credentials to hold from what it receives: both of them, for as long as a rotation is under way. It
never learns them from its own memory, so a provisioner restarted mid-rotation resumes from the step the
vault has recorded.

1. **The vault makes the new value.**
2. **Each applier ensures the unused credential with it**, with the consumer's rights, and leaves the
   one in use untouched. It verifies that the new credential authenticates and the old one still does,
   and confirms. It repeats the confirmation on every reconcile pass until the vault acknowledges it, so
   a lost message costs one pass.
3. **Only then does the vault release the new credential to the readers.** The mesh delivers it and the
   value together, and the host recreates each reader, because a file it read at creation changed. A
   node's host is its own reader: it reconnects to the bus with the new login, and confirms over it.
4. **Each reader confirms by authenticating with the new credential.** It shows this through its
   health check, where its definition declares one, or the applier sees the new credential in use,
   where its backend reports that. A reader for which neither is possible is confirmed by an operator.
   It is never assumed from the reader having restarted.
5. **Only when every reader has confirmed is the old credential retired**, as above, and verified to no
   longer authenticate.

**A reader that goes away leaves the rotation.** A reader unassigned, or re-resolved to another
provider, is no longer waited for. A consumer removed mid-rotation has both of its credentials retired
with it.

**A rotation can be abandoned until the old credential is retired.** An operator abandons it. Readers
that moved are given the old credential back, and recreated. The new credential is retired. Nothing is
lost, because the old one was never removed.

`status` shows a rotation as waiting on whichever applier or reader has not moved, and it is not done
until the old credential is gone. A reader that cannot be reached keeps working on the old credential
until it can, and the rotation waits for it. That wait is shown, never hidden.

**Queues and permissions belong to the consumer, not to a login.** A module's queue on the bus is named
for the module on its node, and both of its logins get the same permissions over it
([ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md)). An MQTT client
identifier is chosen by the consumer and is independent of its login. A reader recreated with a new
login keeps it, and the broker hands the session over.

### A credential a single party holds rotates in place, staged

This covers a provider's administrative credential and a module's own secret, which only that module
reads. The vault makes the new value, and the one party takes it:

- **applied**: the vault delivers the new value **staged, beside the current one**, and the current file
  is left as it is. The party's provisioner changes the backend using the current value, verifies the
  new one, and confirms. Only then does the vault make the new value current. This is the only form for
  a backend that takes its administrative credential only at first initialisation. Replacing the file
  first would lock the provisioner out;
- **read at start**: the vault delivers the new value as current, and the host recreates the party.

There is no window between two parties, because there is only one. Where neither form can change the
value, the requirement is marked not rotatable by the mesh, and a rotation is refused, saying why
([ADR 0113](0113-the-vault-makes-every-secret.md)).

### One rule decides which

**The number of parties decides, never the provider.** The resolver knows it from the requirement's
recipients, leaving out the vault's custody copy, so no definition declares it.

### Until an adapter can

**An adapter that cannot yet ensure a second credential says so.** The two-party credentials it applies
rotate in place, as today, and the window is stated when the rotation is asked for. So does a
two-party credential whose backend has one fixed name and no second credential for it. These are listed
by a check, and the list is meant to shrink. Separating *retire a credential* from *remove the
consumer*, and keying the harness by consumer, come first. They close a data-loss path that exists
today, whatever rotation does.

## What this changes in earlier records

On acceptance, each of these is amended by this record, not edited:

- [ADR 0113](0113-the-vault-makes-every-secret.md): the changeover it left open is decided here.
- [ADR 0049](0049-a-consumers-identity-fits-the-tightest-backend.md): a consumer's identity leaves room
  for the second login's suffix within the tightest backend it reaches, and both logins are checked
  against it.
- [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md): a module's broker
  account is two logins with the same permissions over the same queue, one in use at a time. Its scoping
  is unchanged.
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md), already superseded by 0113:
  a provider now ensures and retires credentials over a resource it owns separately.
- [To-be 13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md): rotation of a two-party
  credential is no longer all-or-nothing with a window. It overlaps, with each step confirmed.
  A single-party credential is staged, not replaced.

## Consequences

- **Every credential provider's adapter changes**, in two steps. The first separates *retire a
  credential* from *remove the consumer*. It names and owns the resource after the consumer, which
  keeps the name it has but moves ownership once in postgres and mssql, and the harness is keyed by
  consumer. The second ensures a second credential with the same rights.
- **The vault gains rotation state**: each rotation's step, per applier and reader, recorded durably.
  Staged delivery is added for single-party secrets. The SDK harness carries the alternation and the
  repeated confirmation, so no adapter implements them.
- **No consumer module changes.** It reads one credential at start, as today, and is recreated by the
  host when it changes. The exception is a reader that has neither a health check nor a backend that
  reports use: its rotations wait for an operator until it declares one.
- **The derived identity is two characters tighter** in the tightest backend, a minio access key of 20
  characters.
- **What got harder:**
  - a provider briefly holds two credentials per consumer;
  - a rotation lasts until its slowest reader moves, so an unreachable reader keeps the old credential
    valid until it is reached;
  - an adapter has four operations where it had two;
  - retiring a login in mssql has to end its sessions first.

## How it is checked

| Rule | Checked by |
|---|---|
| Retiring a credential never removes a resource | A provider test per credential provider: retiring one of a consumer's credentials leaves its resource and data intact, reachable through the other. |
| What a retired login owned survives it | A postgres and an mssql test: objects created under login A, tables included, are still there and alterable under login B after A is retired. |
| A changed login is not a removal | A harness test: changing a consumer's derived login ensures a credential and never calls remove. |
| Remove runs only when the requirement goes | Harness tests: unassigning, dropping the requirement and re-resolving each remove the consumer once; a rotation and a login change never do. |
| No existing resource is renamed | A provider test: a consumer created before the change keeps its resource, with ownership moved to the resource's own role where the backend needs one. |
| Both credentials hold the same rights | A provider test per credential provider: data and structure created under one credential are read, changed and altered under the other. |
| Readers move only after the applier confirms | A rotation test: readers receive nothing until both credentials authenticate at every applier. |
| A reader confirms by authenticating | A rotation test: a reader recreated but failing to authenticate with the new credential does not confirm, and the old credential is not retired. |
| The old credential is retired only after every reader confirms | A rotation test with one reader's node unreachable: it keeps authenticating with the old credential, the rotation shows waiting on it, and it completes when the reader returns and confirms. |
| A reader that goes away leaves the rotation | A rotation test: unassigning a waiting reader lets the rotation complete; removing the consumer mid-rotation retires both credentials. |
| A rotation can be abandoned | A rotation test: abandoning after readers moved gives them the old credential back and retires the new one. |
| Rotation state survives a restart | A test restarting the applier's provisioner, and then the vault, between steps: the rotation resumes from the recorded step. |
| A single-party applied secret is staged | A rotation test on a first-initialisation administrative credential: the provisioner receives the new value beside the current one, applies it, and only then does the new value become current. At no point does it lose its connection. |
| The number of parties decides | A resolution test: a secret with an applier and a reader in different parties is marked for two credentials, and one held by one module for in place. The vault's copy is not counted. |
| Both logins fit the tightest backend | A controller test: both derived logins for the longest node and module names fit the limit ADR 0049 sets. |
| A host rotates its bus login | A rotation test on a node's bus account: the host reconnects with the new login and confirms over the bus before the old one is retired. |
| What still rotates in place is listed | A catalogue test lists every adapter that cannot yet ensure a second credential, and every two-party credential with one fixed name. A rotation of these states its window. |

## References

- [Research 016](../01-RESEARCH/016-how-a-credential-can-be-rotated/00-overview.md): the survey this
  rests on, provider by provider
- [ADR 0113](0113-the-vault-makes-every-secret.md): who asks and who makes
- [ADR 0049](0049-a-consumers-identity-fits-the-tightest-backend.md), [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md),
  [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md): identity, bus accounts, and data outliving
  its declaration
- [To-be 13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md): rotation as implemented
- [Issue 103](../04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md):
  why a reader's restart can be derived