hq/02-DECISIONS/0114-a-shared-credential-rotates-over-two-credentials.md

topic, status, date, deciders, reconstructed, extends

topic	status	date	deciders	reconstructed	extends
what runs on it	proposed	2026-09-26	jochen	false	0113-the-vault-makes-every-secret.md

114. A credential two parties hold rotates over two credentials; one a single party holds rotates in place, staged; and retiring a credential never removes what it reached

Context

ADR 0113 decides who asks for a rotation (an operator, or the vault's policy) and who makes the new value (the vault). It leaves open how old and new change over. Research 016 read every credential provider in the catalogue against its code. There are nine:

• all nine re-apply a password in place, on the same login, every time they run. The controller's rotate command relies on that, and states the window it leaves: between the provider applying the new value and the consumer restarting with it, the consumer cannot authenticate;
• eight of nine name the consumer's resource after its login: a database, a bucket, a virtual host, a key prefix, a topic prefix, a mailbox. Only the forge's npm registry keeps them apart, because an organisation owns the packages;
• five of nine destroy the consumer's data when they remove its login: postgres, mssql and mongodb drop the database, lavinmq drops the virtual host with its queued messages, and mailu deletes the mailbox with its mail. In those adapters, retire a login and delete the consumer's data are one call. minio drops a bucket only if it is empty. The provisioner harness makes it worse: a consumer whose derived login changed is removed under the old login and created under the new one, in one pass;
• one backend holds two passwords on one login (redis), and two hold several tokens beside one password (the forge and mailu);
• eight of nine can give two logins the same rights over one resource. mailu cannot, because a mail user is its mailbox. It can give one user several tokens. In postgres, a second login is not enough on its own: objects belong to whichever login created them, so the resource must be owned by a role of its own;
• an administrative credential has one party and a fixed name. The provider module both applies it and reads it. Five backends take it only at first initialisation: postgres, mssql, mongodb, mosquitto and lavinmq. Their credential file is mounted directly into both the server and the provisioner, so replacing the file recreates the provisioner holding only the new value, which the backend does not know yet. The provisioner is then locked out;
• no module watches a secret. Every reader reads at start, and the host recreates a container when a file it read at creation changes (issue 103).

A first draft of 0113 chose to overlap old and new "through the adapter's existing create and remove". In five providers, that remove deletes the consumer's data. The mechanism has to be chosen on what the providers do, and the danger has to be closed whichever mechanism is chosen.

Considered Options

1. In place for everything, as today. Works with every provider unchanged. Rejected for credentials two parties hold. The window cannot be closed, only shortened, and the two ends are on different machines with nothing ordering them. For an administrative credential, it locks the provisioner out.

2. Two secrets on one login. Rejected as the mechanism. It works for three providers out of nine, and using it there and something else elsewhere would put the difference in the mesh instead of in the adapter.

3. Two logins over one resource. Rejected as the mechanism. It works for eight of nine, and not for mailu.

4. Two credentials over one resource, with the adapter choosing what a credential is. A credential is what a consumer presents, a login and a secret. The mesh alternates between two of them. Each adapter makes the second one the way its backend can: a second login for eight providers, a second token on the same login for mailu. A credential a single party holds is staged in place instead, and retiring a credential is separated from removing a consumer before either is used. Chosen.

Decision

Retiring a credential never removes what it reached

A provider's adapter keeps two things apart that today are one: the consumer's resource (its database, bucket, virtual host, key or topic prefix, mailbox) and a credential that reaches it. They get separate operations:

• ensure the resource, named after the consumer;
• ensure a credential with a value, holding the consumer's rights over its resource;
• retire a credential. Anything it owns moves first to the resource's owner, and any session it has open is ended. Then the credential is removed, and nothing else;
• remove the consumer, which is what removes the resource, and retires every credential it has.

Remove the consumer runs only when the consumer no longer requires the provision from this provider. That happens when its assignment goes, when its definition drops the requirement, or when re-resolution sends it to another provider. It never runs because a login or a value changed. The harness keys what it applied by the consumer, not by the login, so a changed login is a credential change and never a removal. What removing a resource does with the data in it stays ADR 0030's, and re-resolving to another provider moves no data.

The resource is named after the consumer, and owned by the resource, not by a login. A consumer's identity is derived from its assignment (ADR 0049), and today its login is that same string, so no existing resource is renamed. Where a backend makes whatever a login creates the login's own, as postgres does, the resource is owned by a role that cannot log in, and each credential works as that role. Ownership of an existing resource moves to it once. A credential is retired by handing what it owns to that role, never by dropping what it owns.

A credential two parties hold rotates over two credentials

Two parties means an applier and a reader that are different modules, or a module and a node's host. The vault's custody copy does not count, because the vault holds every secret. So this covers a credential between a consumer and a provider, and every bus account: a module's or a host's, applied by the broker's provisioner and read by its owner. Each consumer has two credentials, one in use at a time, both holding the same rights over the one resource. For eight providers the second is a second login, derived by the mesh as the consumer's identity with a short fixed suffix. For mailu it is a second token on the same login.

The vault drives each rotation and records every step durably. A provisioner learns which credentials to hold from what it receives: both of them, for as long as a rotation is under way. It never learns them from its own memory, so a provisioner restarted mid-rotation resumes from the step the vault has recorded.

1. The vault makes the new value.
2. Each applier ensures the unused credential with it, with the consumer's rights, and leaves the one in use untouched. It verifies that the new credential authenticates and the old one still does, and confirms. It repeats the confirmation on every reconcile pass until the vault acknowledges it, so a lost message costs one pass.
3. Only then does the vault release the new credential to the readers. The mesh delivers it and the value together, and the host recreates each reader, because a file it read at creation changed. A node's host is its own reader: it reconnects to the bus with the new login, and confirms over it.
4. Each reader confirms by authenticating with the new credential. It shows this through its health check, where its definition declares one, or the applier sees the new credential in use, where its backend reports that. A reader for which neither is possible is confirmed by an operator. It is never assumed from the reader having restarted.
5. Only when every reader has confirmed is the old credential retired, as above, and verified to no longer authenticate.

A reader that goes away leaves the rotation. A reader unassigned, or re-resolved to another provider, is no longer waited for. A consumer removed mid-rotation has both of its credentials retired with it.

A rotation can be abandoned until the old credential is retired. An operator abandons it. Readers that moved are given the old credential back, and recreated. The new credential is retired. Nothing is lost, because the old one was never removed.

status shows a rotation as waiting on whichever applier or reader has not moved, and it is not done until the old credential is gone. A reader that cannot be reached keeps working on the old credential until it can, and the rotation waits for it. That wait is shown, never hidden.

Queues and permissions belong to the consumer, not to a login. A module's queue on the bus is named for the module on its node, and both of its logins get the same permissions over it (ADR 0043). An MQTT client identifier is chosen by the consumer and is independent of its login. A reader recreated with a new login keeps it, and the broker hands the session over.

A credential a single party holds rotates in place, staged

This covers a provider's administrative credential and a module's own secret, which only that module reads. The vault makes the new value, and the one party takes it:

• applied: the vault delivers the new value staged, beside the current one, and the current file is left as it is. The party's provisioner changes the backend using the current value, verifies the new one, and confirms. Only then does the vault make the new value current. This is the only form for a backend that takes its administrative credential only at first initialisation. Replacing the file first would lock the provisioner out;
• read at start: the vault delivers the new value as current, and the host recreates the party.

There is no window between two parties, because there is only one. Where neither form can change the value, the requirement is marked not rotatable by the mesh, and a rotation is refused, saying why (ADR 0113).

One rule decides which

The number of parties decides, never the provider. The resolver knows it from the requirement's recipients, leaving out the vault's custody copy, so no definition declares it.

Until an adapter can

An adapter that cannot yet ensure a second credential says so. The two-party credentials it applies rotate in place, as today, and the window is stated when the rotation is asked for. So does a two-party credential whose backend has one fixed name and no second credential for it. These are listed by a check, and the list is meant to shrink. Separating retire a credential from remove the consumer, and keying the harness by consumer, come first. They close a data-loss path that exists today, whatever rotation does.

What this changes in earlier records

On acceptance, each of these is amended by this record, not edited:

• ADR 0113: the changeover it left open is decided here.
• ADR 0049: a consumer's identity leaves room for the second login's suffix within the tightest backend it reaches, and both logins are checked against it.
• ADR 0043: a module's broker account is two logins with the same permissions over the same queue, one in use at a time. Its scoping is unchanged.
• ADR 0048, already superseded by 0113: a provider now ensures and retires credentials over a resource it owns separately.
• To-be 13: rotation of a two-party credential is no longer all-or-nothing with a window. It overlaps, with each step confirmed. A single-party credential is staged, not replaced.

Consequences

• Every credential provider's adapter changes, in two steps. The first separates retire a credential from remove the consumer. It names and owns the resource after the consumer, which keeps the name it has but moves ownership once in postgres and mssql, and the harness is keyed by consumer. The second ensures a second credential with the same rights.
• The vault gains rotation state: each rotation's step, per applier and reader, recorded durably. Staged delivery is added for single-party secrets. The SDK harness carries the alternation and the repeated confirmation, so no adapter implements them.
• No consumer module changes. It reads one credential at start, as today, and is recreated by the host when it changes. The exception is a reader that has neither a health check nor a backend that reports use: its rotations wait for an operator until it declares one.
• The derived identity is two characters tighter in the tightest backend, a minio access key of 20 characters.
• What got harder:
  • a provider briefly holds two credentials per consumer;
  • a rotation lasts until its slowest reader moves, so an unreachable reader keeps the old credential valid until it is reached;
  • an adapter has four operations where it had two;
  • retiring a login in mssql has to end its sessions first.

How it is checked

Rule	Checked by
Retiring a credential never removes a resource	A provider test per credential provider: retiring one of a consumer's credentials leaves its resource and data intact, reachable through the other.
What a retired login owned survives it	A postgres and an mssql test: objects created under login A, tables included, are still there and alterable under login B after A is retired.
A changed login is not a removal	A harness test: changing a consumer's derived login ensures a credential and never calls remove.
Remove runs only when the requirement goes	Harness tests: unassigning, dropping the requirement and re-resolving each remove the consumer once; a rotation and a login change never do.
No existing resource is renamed	A provider test: a consumer created before the change keeps its resource, with ownership moved to the resource's own role where the backend needs one.
Both credentials hold the same rights	A provider test per credential provider: data and structure created under one credential are read, changed and altered under the other.
Readers move only after the applier confirms	A rotation test: readers receive nothing until both credentials authenticate at every applier.
A reader confirms by authenticating	A rotation test: a reader recreated but failing to authenticate with the new credential does not confirm, and the old credential is not retired.
The old credential is retired only after every reader confirms	A rotation test with one reader's node unreachable: it keeps authenticating with the old credential, the rotation shows waiting on it, and it completes when the reader returns and confirms.
A reader that goes away leaves the rotation	A rotation test: unassigning a waiting reader lets the rotation complete; removing the consumer mid-rotation retires both credentials.
A rotation can be abandoned	A rotation test: abandoning after readers moved gives them the old credential back and retires the new one.
Rotation state survives a restart	A test restarting the applier's provisioner, and then the vault, between steps: the rotation resumes from the recorded step.
A single-party applied secret is staged	A rotation test on a first-initialisation administrative credential: the provisioner receives the new value beside the current one, applies it, and only then does the new value become current. At no point does it lose its connection.
The number of parties decides	A resolution test: a secret with an applier and a reader in different parties is marked for two credentials, and one held by one module for in place. The vault's copy is not counted.
Both logins fit the tightest backend	A controller test: both derived logins for the longest node and module names fit the limit ADR 0049 sets.
A host rotates its bus login	A rotation test on a node's bus account: the host reconnects with the new login and confirms over the bus before the old one is retired.
What still rotates in place is listed	A catalogue test lists every adapter that cannot yet ensure a second credential, and every two-party credential with one fixed name. A rotation of these states its window.

References

• Research 016: the survey this rests on, provider by provider
• ADR 0113: who asks and who makes
• ADR 0049, ADR 0043, ADR 0030: identity, bus accounts, and data outliving its declaration
• To-be 13: rotation as implemented
• Issue 103: why a reader's restart can be derived