hq/02-DECISIONS/0113-the-vault-makes-every-secret.md

topic, status, date, deciders, reconstructed

topic	status	date	deciders	reconstructed
what runs on it	proposed	2026-09-25	jochen	false

113. The vault makes every shared secret, a provider makes resources and data, and the mesh carries both

Context

A shared secret comes into being many different ways today, counted across the catalogue and the controller on 2026-09-25:

kind	made by	used by
a credential between a consumer and a provider	the controller	19 modules
a module's own secret (own-secrets)	the controller, as a random value nothing owns	54 modules
a module's broker account	the controller, but only when a person runs a separate command; otherwise the random value above, which cannot work (issue 095)	49 modules
a node's and the builder's broker accounts	the controller, each in its own code path	every node, the builder
an enrolment token	the controller	every node joining
a secret from the vault	the controller mints it, and the vault only records it (ADR 0085, as amended)	6 modules
a value an operator accepts	a person (ADR 0092)	where accepted
a licence for model access	a separate controller context with its own store	model consumers
the foundation's root secrets	genesis, sealed to the operator key	the foundation

The vault was built to end the second row, and did not. ADR 0085 says a module's own secret "stops being a generated value that nothing owns". 54 modules still use one, and 6 use the vault. The replacement was added and the old path was never retired.

ADR 0085 considered and rejected making the vault the only maker, because "the controller must mint in order to deliver any provision — the vault's own credential among them": the vault cannot make the credentials that exist before it does. That objection is real, and this record has to answer it rather than step around it.

Rotation has gaps. To-be 13 makes rotation one command, all-or-nothing, with a stated window in which a consumer cannot authenticate. A consumer restarts only if its definition remembered to say so; a container fed by an env-file is not recreated when that file changes (issue 103); and some secrets are read only when a service first initialises, where a restart changes nothing.

And providers cannot answer with data. ADR 0048 left "delivering provider-generated data back to a consumer" to a separate decision. The analytics provider's site id and the DNS provider's record have no way back, and say so in their code.

Considered Options

1. Keep the controller minting, and tidy the paths. Rejected. The paths are the problem: each is made, kept, rotated and audited differently, and tidying keeps them all.

2. Every provider mints its own secrets, with one shared function in the SDK. Rejected. Generation becomes uniform, but custody stays spread over every provider's machine, so rotation, audit and the operator's break-glass copies cover only some secrets. Each SDK language needs its own implementation.

3. Raise the vault first at genesis, so it makes even the first secrets. Rejected. The vault is built on the shared runtime base, which the installation makes only after the store, the broker and the controller exist, and the vault learns what to answer from the controller over the bus. Running it first means reordering the whole installation and giving the vault a second way of being asked.

4. The vault makes every shared secret; genesis delivers the first ones to it. Chosen. It answers 0085's objection with a mechanism the mesh already has: a value delivered to the vault.

Decision

There are two kinds of secret, and each has one rule.

• A shared secret is a value more than one party must hold: a password, a token, an API key. The vault makes every one. Nothing else in the mesh generates a shared secret.
• A private key is made where it is used and never leaves: a node's sealing key, the operator's key, the mesh's certificate authority. This is not a second way of making secrets. A private key any other party ever held would no longer be private.

Every shared secret is a secret requirement, answered by the vault:

• a credential between a consumer and a provider. A provision's contract declares for each consumer, one secret, and resolution expands it into one requirement per consumer. So gitea requiring a database makes the database's provider require a secret for gitea, and the vault answers it. The provider's own code does not change: it is handed a login and a password, as today;
• a module's own secret. own-secrets is retired;
• every broker account on the mesh's bus: a module's, a node agent's, the builder's, the controller's. The broker holding mesh-broker carries the mesh's bus (ADR 0110), and its own provisioner creates each account from the vault's secret, like any provider. The controller no longer creates accounts, and there is no separate command to forget;
• an enrolment token. The vault makes it; the operator receives the token, sealed to the operator key, to hand to the joining machine; the controller receives only what it needs to verify it, never the token itself;
• a secret operator value, such as an external API key, which the operator delivers to the vault (ADR 0092). A licence's credential is one of these. What the licences context adds, refreshing a token, is provider behaviour, decided in its own record;
• a secret a backend issues itself, such as an API token a forge hands out exactly once when asked. The vault cannot make that value. The module that received it delivers it to the vault, which keeps it and provides it like any other; rotating it means asking the backend again.

Parties that are not modules take the same path. The controller's own store login and bus account, and each node agent's bus account, have no definition to require them. The controller asks the vault on its own behalf, or a node's, and the vault answers the way it answers any requirement: made by the vault, sealed to the recipient, carried by the mesh. The requirement is not written in a definition, because the controller and a node agent are the mesh itself, but it is answered no differently.

Only the vault may provide secret. An assignment providing it must hold the mesh-vault seat. The parser refuses a definition that provides it and cannot hold the seat, and a pin cannot route a secret requirement anywhere else, because there is nowhere else.

A secret has recipients, and the vault delivers to each. The database credential has two: the provider, which applies it by creating the login, and the consumer, which reads it and presents it when it connects. The vault hands the value to the mesh sealed to each recipient's node. The controller and the broker carry sealed values they cannot open.

Genesis delivers, and the vault adopts. The vault is built on the shared runtime base, which the installation makes only after the store, the broker and the controller are running (to-be 21). So the vault is installed as soon as that base exists, before any other module built on it, and everything needed before that moment is generated by genesis:

• the store's superuser, and the broker's admin in the hashed form the broker needs;
• the bus accounts of the temporary and permanent controller, the control-node's agent, the builder, the broker's own provisioner and the vault;
• the controller's store login, and the first enrolment token.

Until the broker's provisioner runs, genesis creates the bus accounts it generated, with the broker's admin, as the controller does today. Genesis seals all of it to the operator key, and when the vault is installed it delivers the values to the vault, recorded as the mesh's own, not as an operator's. That distinction matters: an operator's value is never replaced (ADR 0092), and these are, because the vault can make their replacements. The broker's provisioner then adopts the accounts genesis created. From then on the vault makes every shared secret, and genesis has made its last one.

A provider makes resources and data, and the mesh carries data back. A provider's adapter may answer with its contract's non-secret fields: a site id, a registered name. The mesh delivers them to the consumer as resolved values. Who a consumer is stays the mesh's: a provider makes what a consumer is given, never what it is called (ADR 0049).

Rotation

It is asked of the vault, by an operator or by the vault's policy, such as a maximum age in the secret's contract. A delivered value the vault cannot replace, such as an external API key, is not rotated by the vault: rotating it means an operator delivering a new one.

A secret's contract says how each recipient takes a new value:

recipient takes it by	example	what happens on rotation
applying it	a provider creating the login; the broker's provisioner updating an account; the store's own provisioner changing its superuser	its provisioner applies the new value; it is never restarted for it
reading it at start	a consumer reading its password when it starts	the host restarts it, or recreates a container whose env-file carries it

A secret read only when a service first initialises cannot be rotated by a restart. Its contract marks it applied, and a provisioner makes the change. Where no provisioner exists to make it, such as a module's own bootstrap password, the contract marks the secret not rotatable by the mesh, and a rotation request is refused, saying why, rather than restarting a service that would carry on with the old value. The host derives which recipients read a secret at start from the requirement their definition reads it through, so no definition declares a restart for a secret. A provider's per-consumer secrets are applied, never read at start, so the host never restarts a provider for one.

Old and new overlap: nobody is ever without a credential that works. A credential is never changed in place. The new one is added beside the old, every reader moves to it, and only then is the old one removed. There is one mechanism, the same for every provider:

1. The vault makes the new value.
2. Each applier adds it beside the old. A consumer has two logins, both derived by the mesh, and it uses one at a time. The provider's loop creates the other with the new value, through the adapter's existing create, and leaves the one in use untouched. It verifies that the new login works and the old one still does, and confirms. It repeats that confirmation on every reconcile pass until the vault acknowledges it, so a lost message costs one pass.
3. Only then is the new login released to the readers. A reader receives the new login and its value together. The host restarts it, or recreates a container whose env-file carries it.
4. Each reader confirms, by restarting with the new login and passing its health check, where its definition declares one.
5. Only when every reader has confirmed is the old login retired. Each applier removes it, through the adapter's existing remove, and verifies that it no longer authenticates.

What overlap closes:

• a reader whose machine is offline keeps the old login, which still works, until it returns and moves; the rotation shows as waiting on that reader, and nobody is locked out;
• a bus account's owner keeps its old account until it has confirmed the new one over the bus it still has, so no party can lose the bus it would hear the new value on;
• a provisioner restarted mid-rotation is still delivered both values until the old is retired, so it can verify either.

What overlap costs.

• Consumer modules: nothing. A consumer reads one login at a time and changes it when it restarts.
• Providers: one duty. Both of a consumer's logins must have the same rights over its data, because the consumer's data was written under one login and is read under the other. In postgres, both are members of one role that owns the data. That is the adapter's part, and the only place overlap touches provider code. The alternation itself is the provider loop's, so every provider gets it by using the harness.
• The mesh: it derives two logins per consumer, and both must still fit the tightest backend (ADR 0049).
• Secrets with no applier, such as a module's own secret read only by itself, have no second party to overlap with. They are delivered and the reader restarted, where their contract allows rotation at all.

step	who
asks	an operator, or the vault's policy
makes the value	the vault
adds the new login beside the old	each applier's provisioner, confirming on every pass until acknowledged
moves each reader	the host, restarting or recreating what reads the secret
confirms each reader	its restart and health check
retires the old login	each applier's provisioner, once every reader has confirmed
shows progress	status: waiting on which applier or reader, never done until the old is retired

What this changes in earlier records

On acceptance, each of these is superseded or amended by this record, not edited:

• ADR 0048 is superseded: the controller no longer mints a provider's credential; the vault makes it. That a provider is handed its credential and seals nothing stands.
• ADR 0085 is amended: the vault makes every shared secret, own secrets are retired, and its rejection of vault-only minting is answered by genesis delivering the first secrets. "The vault stores no plaintext, ever" stands.
• ADR 0092 is amended: an operator delivers a secret to the vault. Genesis's values reach the vault by delivery too, but are recorded as the mesh's own, so 0092's rule that an operator's value is never replaced does not apply to them.
• ADR 0043 is amended: a broker account is created by the broker's provisioner, not the controller. Its scoping stands.
• ADR 0049 is amended: the mesh derives two logins per consumer, and both fit the tightest backend.
• To-be 13, to-be 21 and to-be 24 are amended: rotation overlaps old and new instead of being all-or-nothing, with restarts derived and each step confirmed; the vault is installed as soon as the shared runtime base exists, and genesis delivers its secrets to it; the vault is the only maker.
• The glossary gains shared secret, recipient, applies and reads at start, and reserved provision, once this record is accepted.
• Issue 103 becomes a prerequisite: rotation cannot be trusted while a changed env-file leaves a container on its old value.

Consequences

• The vault is on the path of every new or rotated shared secret. Today the controller holds that place, on the same node. A secret can no longer be made while the vault is down.
• Resolution expands per-consumer requirements from a provision's contract. The contract declares them, never the provider's code, so what a provider requires stays predictable from the catalogue.
• The SDK's provider loop gains the alternation of two logins per consumer and repeated confirmation of each step. A credential provider's adapter gains one duty, giving both logins the same rights over the consumer's data. A data provider's adapter gains a return value. No consumer module changes.
• The broker's provisioner gains every bus account, and the controller loses five separate places it generates a secret today.
• 54 modules move from own secrets to vault requirements. Six provider clients export a password generator nothing uses any more; it is removed, so no module can quietly start minting again.
• The installation changes order: the vault is installed as soon as the shared runtime base exists, before any other module built on it.
• What got harder: a rotation lasts until its slowest reader has moved, so a reader offline for a week keeps the old login valid for a week. That is shown, and it is the price of never locking anyone out. A provider briefly holds two logins per consumer. A secret some services read only at first start can no longer be "rotated" by a restart that quietly changes nothing; it is refused instead.

How it is checked

Rule	Checked by
Only the vault generates a shared secret after genesis	A controller test: no code path generates a shared secret. A catalogue test: no module's code generates one, found by scanning for generation calls, with none exempt. An installer test: genesis generates exactly the list above and delivers it to the vault, recorded as the mesh's own.
A private key is made where it is used	A test per key: a node's sealing key never leaves the node, the operator's private key never enters the mesh, and the certificate authority's private key never leaves the controller's identity store.
Only the vault provides secret	The parser refuses a definition providing secret that cannot hold mesh-vault, and resolution refuses a pin on a secret requirement.
Parties that are not modules take the same path	Controller tests: its own store login, its bus account and a node agent's bus account are each made by the vault and delivered sealed; an enrolment token reaches the controller only as what verifies it.
A backend-issued secret enters through the vault	A vault test: a value delivered as issued is provided like any other, and rotating it is refused as the vault's act.
A secret with no provisioner to apply it is not rotated by restart	A vault test: rotating a secret marked not rotatable by the mesh is refused, naming why.
A provider's per-consumer secret comes from the vault	A resolution test: a consumer requiring a database expands to a secret requirement for it, answered by the vault and delivered to both recipients.
Values are carried sealed	A controller test: each recipient's copy opens with that recipient's node key and no other; neither the controller nor a message on the broker can open one.
Own secrets are retired	A catalogue test: no definition declares an own secret, with a declared list of exceptions that shrinks to empty.
Bus accounts come from the broker's provisioner	A resolution test: assigning a module that speaks on the bus yields its account, created by the broker's provisioner with no separate command.
An operator's value is never rotated by the vault, and genesis's values are	Vault tests: a rotation request on an operator's external key is refused, naming the operator; the same request on a value genesis delivered makes a replacement.
Old and new overlap	A rotation test: after an applier adds the new login, both authenticate; readers are released only after it confirms; the old login is removed only after every reader confirms, and then no longer authenticates.
An offline reader is never locked out	A rotation test with one reader's node offline: it keeps authenticating with the old login throughout, the rotation shows waiting on it, and completes when it returns.
A bus account's owner keeps the bus	A rotation test on a node agent's bus account: the agent stays connected on the old account until it has confirmed the new one.
A restarted provisioner can still verify	A rotation test restarting the applier's provisioner mid-rotation: it is delivered both values and confirms.
Both logins have the same rights	A provider test per credential provider: data written under one of a consumer's logins is read and changed under the other.
Both logins fit the tightest backend	A controller test: the two derived logins for the longest node and module names fit the limit ADR 0049 sets.
Restarts are derived from how a secret is read	A host test: a secret read at start restarts its reader, and recreates a container whose env-file carries it; an applied secret restarts nothing.
Rotation is confirmed	A rotation test: the rotation shows unconfirmed until every reader has restarted with the new login and passed its health check, and the old login is retired.
A provider answers data back	A lab test with a consumer requiring analytics: the provider's site id reaches it as a resolved value.

References

• ADR 0048: the decision this supersedes, and the return path it left open
• ADR 0085: the vault, and the objection this record answers
• ADR 0092, ADR 0049, ADR 0043: delivered values, identity, and broker accounts
• ADR 0112, to-be 27: everything a module needs is a requirement
• Issue 095, issue 103: what fails today