novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge pull request 'Graduate issues 067 and 068 — provider scoping and the secrets vault' (#57) from multi-node/harden-and-prove into main

Browse Source
This commit was merged in pull request #57.
This commit is contained in:
jschoubben
2026-09-20 21:30:12 +02:00
parent df2c3f63a5 fc4ab370d6
commit 731027c009
9 changed files with 625 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0084-which-provider-serves-a-consumer.md
+108
View File
@@ -0,0 +1,108 @@
---
topic: what runs on it
status: accepted
date: 2026-09-20
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
---
# 84. Which provider serves a consumer, when the mesh runs more than one
## Context
[ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) settled what a provision
is *named* for — the thing the consumer's code is coupled to, so `postgres-database` and
`mssql-database` are different provisions and a wrong match is refused at resolution. It said one
thing more, in passing, and left it: *"Two providers of `postgres-database` — a container on this
node and a managed instance elsewhere — are interchangeable and should both match."* Naming was
decided; **which of several providers serves a given consumer was not.**
The mesh assumes there is only one to choose. Provisions are mesh-scoped: the adopted store is a
single `mesh-store` a consumer on any node reaches, and `scope: "mesh"` is written into the
provision definitions. **That assumption is false on day one, and was always meant to be.**
Node-specific services delivered to the mesh is the plan, not an edge case:
- Both control-capable nodes already run their **own** general-purpose relational store (the same
engine, one instance each), serving that node's own applications.
- Each runs its **own** SQL server, its **own** cache, its **own** object store. One node alone
runs six separate relational-store instances, each raised by the module that needed it.
- The one provision that is currently single — the identity provider, one instance on one node —
already authenticates applications whose home is a **different** node.
So several providers of one provision name genuinely coexist, and they are **not** interchangeable
the way 0027's aside supposed. They differ by node, by the data they hold, and by locality. A
consumer bound to the wrong one reads the wrong database, or takes a cross-node hop it did not
need, or cannot be moved without silently rebinding. The model has no field in which to say which
one. This is 0027's own fault — *a match that resolves and is wrong* — one level up: 0027 refused
the wrong **dialect**; nothing refuses, or even asks about, the wrong **instance**.
## Considered Options
1. **Keep `scope: "mesh"` — one provider per provision, mesh-wide.** Rejected: it is false on day
one, and making it true would force every node's applications onto one node's server — the
exact opposite of node-specific services delivered to the mesh, and a single point of failure
the topology was built to avoid.
2. **Resolve to any provider of the name (0027's "both match").** Rejected: when providers hold
different data and live on different nodes they are not interchangeable, and picking one
arbitrarily is a wrong-instance match — the confidently-wrong answer 0027 exists to prevent,
restated at the level of the instance rather than the dialect.
3. **Always require the consumer to name the provider explicitly.** Rejected: needless ceremony in
the common case, where the consumer wants the provider on its own node; and a field every
manifest must carry is a field an author forgets, which then matches everything again — the
failure 0027 warned about for qualifiers.
4. **A provision is node-scoped; the consumer selects the provider, defaulting to co-location.**
Adopted.
## Decision
**A provision is served by a provider identified by its node, and the consumer selects which one.**
A provider is a (node, module) pair, not a mesh-wide singleton. A consumer's binding resolves to a
specific provider, and the selection is part of the assignment
([ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)), not the
manifest.
**The default is co-location.** A consumer that names no provider is served by the provider of
that provision on **its own node**. This is the common case and needs nothing said. A mesh with
one provider of a kind is just the case where co-location and "the only one" coincide — expressed
as *there happens to be one*, not as a scope.
**A consumer coupled to a provider's data names it.** Where two consumers must share one database,
or a consumer must reach a provider on another node, the assignment names that provider — because
that coupling is exactly what may not be guessed, and naming it is what makes a later move safe.
**A module need not consume a provider at all.** It may carry its **own** instance inside its own
composition — on its own module network, publishing no host port, **not** declared as a provision —
when a genuine engine fork or a pinned server version makes the shared provider unusable. Such an
instance is invisible to resolution and can be bound by nothing else. The rule is *share by
default; embed only when a fork or a version forces it* — most of the per-module stores that exist
today are vanilla engines on stale pins that a consolidation onto the node's provider would absorb.
## Consequences
The mesh can carry its real topology **deliberately** rather than by the accident of which
provider happened to be the single one. A consumer's data-coupling becomes a stated fact, which is
what lets a provider be moved without a consumer silently following the wrong one — provided a
provider keeps its identity across a relocation, which is a follow-up this record opens rather than
closes. Rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)) addresses a
specific provider's holders, not "the provision's".
What got harder: an assignment now may carry a provider selection, and a wrong one is a new way to
misconfigure. It is mitigated the way 0027 mitigated its own: the co-location default removes the
choice in the common case, and genuine ambiguity — several providers, none named, none co-located —
is refused with the candidates named, never resolved by picking.
## References
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — the naming this extends;
its "both match" aside is the gap closed here.
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is exactly such a
provider, and already serves consumers on another node.
- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md) — where the
selection lives.
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the adopted store, whose mesh-wide
binding is the single-provider assumption this record replaces.
- [issue 067](../04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md) — the
gap, and the day-one evidence.
- [`03-DESIGN/01-to-be/23-choosing-a-provider.md`](../03-DESIGN/01-to-be/23-choosing-a-provider.md)
— the design.
02-DECISIONS/0085-a-secret-is-a-provision.md
+107
View File
@@ -0,0 +1,107 @@
---
topic: what runs on it
status: accepted
date: 2026-09-20
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
---
# 85. A secret is a provision, and the vault is the module that provides it
## Context
[ADR 0031](0031-the-control-plane-authenticates-nobody.md) decided that identity *runs on the
mesh, not of it* — a module other modules require, rather than a privileged part of the
controller. [ADR 0078](0078-the-store-and-broker-are-modules.md) did the same for the store and
the broker: the twelve-module floor has no specialty left in it. **Secrets are the exception that
survived.** No module owns a secret.
Secret handling is smeared across three built-in parts of the runtime:
- the **controller mints** one credential per consumer↔provider pair and seals it to both node
keys ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md));
- the **mesh generates local secrets** — *"generated secrets are the mesh's, never authored"*
(as-is, `06-configuration-and-secrets.md`) — for a value a single module needs for its own use;
- the **synchroniser injects** both into a node's generated files.
Nothing is the owner of "a secret" the way the store module is the owner of "a database", and the
cost is recorded rather than hypothetical. The as-is design names the weakness in its own words:
*"Rotation is not a mesh operation… there is no mechanism that rotates one and informs everything
holding it."* A `rotate` command has since been built and proven, but it reaches **only** the
provisioned pairs; a secret a module generates for its own fully-local use — the password of a
version-pinned embedded store ([ADR 0084](0084-which-provider-serves-a-consumer.md)), an internal
token — is minted by the mesh and then has no operation that can remake it. Three species of
secret, and only the first has an owner:
| species | minted by | rotates? |
|---|---|---|
| a provisioned credential (a database login) | controller, sealed to nodes (0048) | yes — `rotate`, per pair |
| a module's own local secret | the mesh, as a generated value | **no owner, no rotation** |
| an operator-delivered secret (an external key) | a person, sealed in (`secret accept`) | no rotation, no audit |
## Considered Options
1. **Leave it a property of the controller.** Rejected: it is the smear above — no owner, local
secrets that cannot be rotated, operator secrets that cannot be audited — and it is exactly the
specialty 0078 removed for the store and broker, kept here for no reason anyone recorded.
2. **A dedicated vault built into the foundation, not a module.** Rejected: it reintroduces a
privileged built-in, the thing 0031 and 0078 went out of their way to remove, and a mesh that
wants none would still carry it.
3. **Fold all minting, the controller's provisioning credentials included, into the vault.**
Rejected: the controller must mint in order to **deliver** any provision — the vault's own
credential among them — so making the vault mint the credential of its own delivery is the
store/broker chicken-and-egg for no gain. The provisioned-pair credential already has an owner
and a rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)); this record
does not disturb it.
4. **A secret is a provision; the vault is an ordinary module that provides it.** Adopted.
## Decision
**Secret-holding is a module, parallel to identity.** A module that needs a secret **for its own
use** — a local service's password, an internal token, an external key it was handed — requires a
`secret` provision from a vault provider, exactly as it requires a database from the store. The
vault generates the value (or holds one it was given), and because the credential belongs to the
consumer↔vault pair it **rotates, backs up and is audited through the same per-pair machinery**
[13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md) already defines. That is what
gives the second and third species the rotation and audit they lack. A module's own secret stops
being a generated value that nothing owns and becomes an ordinary provision with a provider.
**The controller's minting of provisioning-pair credentials is unchanged** ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)):
it is how every provision, the vault's own included, is delivered. The vault does not mint the
mesh's delivery credentials; it provides secrets to modules, and is itself provisioned the ordinary
way.
**The vault is node-scoped like every provider** ([ADR 0084](0084-which-provider-serves-a-consumer.md)):
each node its own, selected the same way, so a module's own secret is held by the vault on the
module's node. **A mesh that wants no vault runs none** — a module requiring no secret needs
nothing, which is the same test 0031 applied to identity.
## Consequences
The gap that opened this — a generated local secret with no rotation — **closes without new
machinery**: rotating such a secret is the vault's provisioner remaking a pair credential, the
operation 13 already specifies. Backup, audit and break-glass gain an owner — the vault module —
and become things a design specifies rather than absences. The as-is sentence *"rotation is not a
mesh operation"* is already false for provisioned pairs and, once the vault ships, for local
secrets too; the as-is document is updated when it does, not before.
What got harder: a break-glass path — recovering a secret when the sealed delivery path is
unavailable — must not reintroduce a key that one place holds, which is the property
[ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) was built to preserve; how
the vault offers recovery without it is left to the design as an open question. And a module that
today bakes a password into its own composition must instead require it from the vault — a
migration taken module by module, not a flag day.
## References
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is a module; this is the
same move for secrets.
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) — the provisioning-credential
path, left unchanged.
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the de-specialisation this completes.
- [ADR 0084](0084-which-provider-serves-a-consumer.md) — the node-scoping the vault obeys.
- [issue 068](../04-ISSUES/068-secrets-have-no-owning-module/00-report.md) — the gap.
- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md) —
the design. The source mesh's `secret_locate` / `secret_backup` / `secret_verify` /
`secret_breakglass` subsystem is the prior art it draws on.
02-DECISIONS/README.md
+2
View File
@@ -136,6 +136,8 @@ python3 00-META/checks/index.py fail if stale
- **0053** — [A scheduled step is a container run on a recurring schedule](0053-a-step-that-runs-on-a-schedule.md)
- **0054** — [Model usage is a vendor-neutral record, produced by the adapter, at two grains](0054-model-usage-is-recorded-at-two-grains.md)
- **0055** — [Model access is answered by a licence, or by a node that hosts the model](0055-model-access-is-answered-by-a-licence-or-a-node.md)
- **0084** — [Which provider serves a consumer, when the mesh runs more than one](0084-which-provider-serves-a-consumer.md)
- **0085** — [A secret is a provision, and the vault is the module that provides it](0085-a-secret-is-a-provision.md)
### How it is built
03-DESIGN/01-to-be/13-credentials-and-their-rotation.md
+18 -1
View File
@@ -5,10 +5,11 @@ code:
- mesh-controller internal/inventory/secrets.go
- mesh-controller cmd/mesh-controller/rotate.go
- mesh-controller examples/postgres-provisioner
updated: 2026-09-01
updated: 2026-09-20
decisions:
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
- 02-DECISIONS/0009-modules-and-the-graph.md
- 02-DECISIONS/0085-a-secret-is-a-provision.md
---
# 13 — Credentials, and moving them
@@ -105,3 +106,19 @@ a provider that added a password and removed nothing.
**Not over loopback.** `pg_hba` trusts anything there, so every password looks correct — a
deliberately wrong one returned a row for an afternoon before that was noticed.
## The secret that is not a pair
Everything on this page is about the credential *between a consumer and a provider* — the login
one module uses against another. A mesh also holds secrets that are not that: a value a single
module needs for **its own** use, and a value only an operator can supply. Those had no owner, and
so no rotation — the gap this page's machinery could not reach because there was no pair to rotate.
[ADR 0085](../../02-DECISIONS/0085-a-secret-is-a-provision.md) gives them one. Such a secret is
provided by a **vault** module ([24](24-the-secrets-vault.md)): a module requires a `secret`
provision, and the credential of that consumer↔vault pair is an ordinary pair credential — so it
rotates, and is queried for who holds it, through exactly the machinery described above, unchanged.
The rotation a module's own secret lacks is not a second mechanism; it is this one, pointed at a
secret the vault provides. What this page proves for a database password holds, by construction,
for a secret from the vault.
03-DESIGN/01-to-be/23-choosing-a-provider.md
+87
View File
@@ -0,0 +1,87 @@
---
layer: to-be
status: designed
code: []
updated: 2026-09-20
decisions:
- 02-DECISIONS/0084-which-provider-serves-a-consumer.md
- 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
---
# 23 — Choosing a provider
A provision is named for what the consumer's code is coupled to
([ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md)):
`postgres-database`, not `database`. That decides *what kind* of provider satisfies a requirement.
It does not decide *which* provider, and the mesh runs more than one of most kinds.
## Why there is a choice at all
Node-specific services delivered to the mesh is the design, not an exception. Every control-capable
node runs its own relational store, its own cache, its own object store; a single node may run
several relational stores, each raised by the module that needed a particular engine or version.
The one provision that is single today — the identity provider — already serves applications whose
home is another node. So for a given provision name there are usually several providers, one per
node, and they are **not** interchangeable: each holds different data and lives in a different
place. A consumer bound to the wrong one reads the wrong database or takes a network hop it did not
need.
Naming the kind is therefore only half of "how a consumer gets what it needs". The other half is
which provider, and it has two shapes: **consume a provider**, or **carry your own**.
## Consuming a provider
A provider is not a mesh-wide singleton. It is identified by the node it runs on together with the
module that provides it — a (node, module) pair. A consumer's requirement resolves to one such
provider, and which one is part of the **assignment**, not the manifest
([ADR 0046](../../02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md)):
the same module, assigned twice, may be served by two different providers.
**The default is co-location.** A consumer that names no provider is served by the provider of that
provision on its own node. This is the ordinary case and is meant to need nothing said — a module
that wants a database wants, almost always, the database on the machine it runs on. A mesh that
happens to run exactly one provider of a kind is simply the case where co-location and "the only
one there is" name the same thing; that is *there happens to be one*, not a mesh-wide scope written
into the provision.
**Coupling to data is named.** The exception to co-location is a consumer coupled to a *particular
provider's contents*: two modules that must share one database, or a consumer that must reach a
provider on a different node. That coupling is exactly what may not be guessed, so the assignment
names the provider. Naming it is also what makes a later move safe — the mesh knows the binding is
to that provider and not to whichever one is nearest.
**Ambiguity is refused, never resolved by picking.** If several providers of a kind exist, none is
named, and none is co-located, the requirement is unsatisfiable and is refused with the candidates
shown — the same stance
[ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md) took
against a confidently-wrong match, applied to the instance rather than the dialect. A wrong answer
delivered quietly costs more than a refusal.
## Carrying your own
A module need not consume a provider at all. It may carry its **own** instance of an engine inside
its own composition — reachable only on the module's own network, publishing no host port, and
**not** declared as a provision. Nothing else in the mesh can see it or bind to it, and it cannot
collide with anything on a well-known port. To resolution it does not exist; it is an internal part
of the module, like any other container the module runs.
This is legitimate but it is the exception, and the design says when: **only when a genuine engine
fork or a pinned server version makes the shared provider unusable.** A module written against a
customised engine, or one that needs an extension the node's provider does not carry, has no choice
but to carry its own. A module that merely pins an old image of an ordinary engine does not — the
version on a compose file is the *server's*, and the application talks to a newer shared server
perfectly well once its data is migrated in. The rule is *share by default; embed only when a fork
or a version forces it*. Most of the per-module stores that exist in the mesh being migrated onto
are the first kind wearing the second's clothes, and consolidate onto the node's provider.
The distinction is worth stating because the two cases look identical from outside — a module with
a database either way — and the mesh must be able to tell them apart to reason about either. A
consumed provider is a binding the mesh records, rotates and can move. An embedded instance is a
private detail the mesh does not manage and must not mistake for a provider.
## What is not settled here
A provider that moves between nodes must keep its identity, so that a consumer's recorded choice
does not silently rebind to a different provider that inherited its place. That is a property the
provider lifecycle must supply, and this document names it as a requirement rather than describing
its mechanism.
03-DESIGN/01-to-be/24-the-secrets-vault.md
+98
View File
@@ -0,0 +1,98 @@
---
layer: to-be
status: designed
code: []
updated: 2026-09-20
decisions:
- 02-DECISIONS/0085-a-secret-is-a-provision.md
- 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
- 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md
---
# 24 — The secrets vault
Identity runs on the mesh, not of it — a module other modules require, not a part of the
controller ([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)). The
store and the broker are ordinary modules too
([ADR 0078](../../02-DECISIONS/0078-the-store-and-broker-are-modules.md)). Secrets are the piece
that never got the same treatment: nothing owns a secret. This document describes the module that
does — a **vault** that provides a `secret` provision — and, as importantly, the boundary of what
it owns and what it deliberately does not.
## Three kinds of secret, and which the vault owns
A mesh handles three species of secret, and confusing them is how the current arrangement went
wrong.
- **A provisioned credential** is the login one module uses against another — a database password,
a broker account. The mesh mints it per consumer↔provider pair and seals it to the machines that
must hold it ([ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md)),
and its provider makes it true. **This is not the vault's**, and the vault does not change it. It
already has an owner and a rotation
([13](13-credentials-and-their-rotation.md)); disturbing it would buy nothing.
- **A module's own secret** is a value a single module needs for itself — the password of a store
it runs privately, an internal signing token. Today the mesh generates this as a value nothing
owns, and so it cannot be rotated. **This is the vault's.**
- **An operator-delivered secret** is a value only a person can supply — a credential for something
outside the mesh. Today it is sealed in and then held, un-audited and un-rotatable. **The vault
holds this**, and can hand it out and audit it, though it cannot generate it.
The vault, then, is the provider a module turns to for a secret that is **not** the byproduct of
some other provision. It is the answer to *"this module needs a password, and there is no provider
whose job it is to give it one."*
## A secret as a provision
A module that needs a secret for its own use requires a `secret` provision, exactly as it requires
a database from the store. The vault generates the value — or takes custody of one an operator
delivered — and the credential belongs to the consumer↔vault pair. Because it is an ordinary pair
credential, **everything already built for pair credentials applies to it unchanged**: it rotates
with the one command that discards a credential and delivers both ends together, it is one secret
per holder so rotating one touches nothing else, and *who holds this* is a query rather than an
assumption ([13](13-credentials-and-their-rotation.md)). The rotation a module's own secret lacks
today is not new machinery; it is the machinery that already moves a database password, pointed at
a secret the vault provides.
This is what replaces the "generated value that nothing owns". A module's own password stops being
a special kind of thing injected by the synchroniser and becomes a provision with a provider, a
holder, and a lifecycle — the same shape as everything else the mesh grants.
## The vault is a module, and node-scoped
The vault is an ordinary module. A mesh that wants one runs it; a mesh whose modules require no
secret of their own runs none — the same test identity meets
([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)). It is provisioned
the ordinary way, and it does **not** mint the mesh's delivery credentials — the controller does
that, because the controller must mint in order to deliver any provision, the vault's own included.
The vault mints secrets *for modules*, downstream of its own existence, never the credential that
delivers it.
Like every provider it is node-scoped
([ADR 0084](../../02-DECISIONS/0084-which-provider-serves-a-consumer.md), [23](23-choosing-a-provider.md)):
each node may run its own vault, and a module's own secret is held by the vault on the module's
node, selected the same way any provider is. There is no single mesh vault holding everything, for
the same reason there is no single mesh store.
## Beyond generate and hold
Owning a secret means owning more than its creation. The mesh being migrated onto has a working
secrets subsystem whose surface names the operations a vault is responsible for — locating a secret,
backing it up, verifying it is what it should be, and a break-glass recovery for when the normal
path is unavailable. These become the vault module's, specified against it rather than scattered.
One of them is left open on purpose. **Break-glass must not reintroduce a key that one place
holds.** The whole point of sealing a secret to the machine that needs it, asymmetrically, is that
no single place can open everything
([ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md)); a
recovery path that keeps a master key would undo exactly that. How the vault lets an operator
recover a secret without becoming the thing the sealing was designed to prevent is a question this
design opens and does not yet answer.
## What this changes for a module
A module that today writes a password into its own composition — an embedded store's login, an
internal token — instead requires it from the vault and reads it where the mesh puts it. The change
is taken module by module, not as a flag day, and it is the same change in each: a value the module
authored becomes a value the vault provides. When it is done, the guarantee the as-is design already
makes for generated secrets — *nothing in the repository contains a credential* — holds for a
module's own secrets not by convention but because there is a provider whose job it is to keep them.
03-DESIGN/01-to-be/README.md
+2
View File
@@ -32,6 +32,8 @@ document is written and this one's status becomes `implemented`.
| [`20-writing-a-module.md`](20-writing-a-module.md) | A worked guide: one module, four capabilities, four languages, and the packages it publishes | [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md), [ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md), [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md) |
| [`21-the-installation-in-full.md`](21-the-installation-in-full.md) | Every step from a bare machine to a mesh that maintains itself, and what is not yet true | [ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md), [ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md), [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) |
| [`22-the-work-ahead.md`](22-the-work-ahead.md) | Everything decided and not yet built, in dependency order, each phase ending at a run | [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md), [ADR 0075](../../02-DECISIONS/0075-two-stores-and-which-provides-what.md), [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) |
| [`23-choosing-a-provider.md`](23-choosing-a-provider.md) | Which of several providers of a kind serves a consumer, and when a module carries its own instead | [ADR 0084](../../02-DECISIONS/0084-which-provider-serves-a-consumer.md), [ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md) |
| [`24-the-secrets-vault.md`](24-the-secrets-vault.md) | The module that owns a secret — a `secret` provision, and the boundary of what it owns | [ADR 0085](../../02-DECISIONS/0085-a-secret-is-a-provision.md), [ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md), [ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md) |
## Not yet written
04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md
+113
View File
@@ -0,0 +1,113 @@
---
status: resolved
opened: 2026-09-20
located-in: []
fixed-by:
amended-design: 03-DESIGN/01-to-be/23-choosing-a-provider.md
---
# A provision cannot name which provider serves it
## Symptom, as observed
The mesh models every provision — `postgres-database`, `s3-bucket`, `amqp`, a
future `oidc` — as **mesh-scoped**: there is one provider of a given kind for the
whole mesh, and a consumer that requires the provision is bound to *that* one.
`scope: "mesh"` is written into the provision definitions, and the adopted store
is a single `mesh-store`.
The mesh being migrated onto is not shaped that way, and never was meant to be.
**Node-specific services delivered to the mesh was the plan from the start.** Each
node already runs its own provider of the same kinds:
- Both control-capable nodes run their **own general-purpose postgres server**
(the same image, one per node), serving that node's own applications.
- Each node runs its **own** SQL server, its **own** redis, its **own** object
store — infrastructure is per node, by design, not a single mesh-wide instance.
- Identity is the *only* provision that is currently single (one realm on one
node), and even that already serves applications hosted on a **second** node.
So the multi-provider reality is not a future edge case that appears "the day a
second provider is added" — it is the founding topology, true today, on every
provision kind. What is missing is any way to **say it**. A consumer requires
`postgres-database`; it cannot require *this node's* postgres rather than *that
node's*. It requires `oidc`; it cannot name which node's identity provider. The
mesh model collapses a deliberately per-node fleet down to one mesh-scoped
provider, and a consumer has no field in which to choose.
## Why it matters beyond this instance
- **The model regressed an intended topology, it did not merely miss a corner
case.** "Node-specific services delivered to the mesh" is the design; `scope:
"mesh"` with a single `mesh-store` expresses the opposite. This is a gap between
a stated intent and what the manifests can represent, which is exactly what the
issues process is for.
- **It is not an identity special case.** The missing concept — *a provision has a
provider, providers are per-node, and a consumer names the provider* — is the
same for databases, object stores, brokers and identity. A fix aimed only at SSO
would leave the same wall standing behind postgres and minio, both of which are
*already* multi-provider on the live mesh.
- **The single-provider assumption is silent.** Nothing rejects a second provider
of a mesh-scoped provision; the model just cannot address it, so a consumer binds
to whichever one is "the" provider — by accident of there being one, or by a race
when there are two. A rule enforced by nothing ("there is one provider per
provision") reads as true until the second node's provider makes it false, with
no diagnostic at the seam.
- **It blocks the migration concretely.** The mesh already has applications on one
node depending on another node's provider (identity today; databases the moment
an app is assigned to a node whose local postgres is not "the" mesh store).
Modelled as mesh-scoped, that topology is expressible only by accident. To carry
it deliberately the provision must be able to name its provider.
## A second axis: provisioned against a provider, or embedded and private
Naming *which* provider is only half of "how a module gets a database". There is a
second, distinct case the model also cannot express: a module that does **not**
consume a shared provider at all, but carries its **own** instance inside its own
composition — on its own module network, publishing no host port, visible to
nothing else in the mesh. This is legitimate and sometimes necessary: some
containers pin a database *server* version or need a fork or extension set (a
customised postgres, a vector extension) that the node's shared provider does not
offer, so they must run their own alongside the main container.
The distinction that matters:
- **Provisioned** — the module requires a provision and is bound to a *named
node-scoped provider* (the first axis above). This should be the default; on the
mesh being migrated onto, most per-module databases are vanilla servers on old
version pins that could simply be consolidated onto the node's shared provider.
- **Embedded and private** — the module ships its own instance because a fork or
version genuinely forces it. Its credential is still a mesh-generated secret, not
a module-authored password; but the *instance* is module-internal — same module
network only, no published port, **not registered as a provision**, so nothing
else can bind to it and it cannot collide on a well-known port.
The model has no word for the second case. A module that carries a private instance
looks, to the mesh, either like nothing (an undeclared container) or like a provider
it must not be treated as. "Embed only when a fork or version forces it; otherwise
provision against the node's provider" is the rule the design should be able to
state and check — and the invisibility of an embedded instance (own network, no host
port, not a provision) should be an enforceable property, not a convention.
## Open questions
- Where does the provider name live — on the provision definition (`scope: "node"`
with a provider identity), on the requirement in the consumer's manifest, or
supplied only at assignment time so the same module can be bound to different
providers on different assignments?
- Is "mesh-scoped" still a legitimate scope for some provisions (a single mesh CA,
say), or does every provision become node-scoped, with a single instance
expressed as "there happens to be one"?
- What is the default when a consumer names no provider — bind to the node the
consumer is assigned to (co-located provider), require the name always, or fall
back to a mesh-wide default provider where one is declared?
- How does a provider's identity survive being moved between nodes, so a consumer's
recorded choice does not silently rebind when the provider relocates?
- Does this interact with secret rotation (the issue-scope of `rotate`) — must
rotation address a specific provider's credential holders rather than "the
provision's"?
- When a module carries an **embedded, private** instance rather than consuming a
provider, how is that declared so the mesh knows it is module-internal — not a
provision, not published, not bindable by anything else — and can enforce it?
- What decides embed-vs-provision — is it the module's declaration alone, or may an
operator override at assignment (consolidate this one onto the node's provider)?
04-ISSUES/068-secrets-have-no-owning-module/00-report.md
+90
View File
@@ -0,0 +1,90 @@
---
status: resolved
opened: 2026-09-20
located-in: []
fixed-by:
amended-design: 03-DESIGN/01-to-be/24-the-secrets-vault.md
---
# Secrets have no owning module
## Symptom, as observed
Every other piece of shared infrastructure in the mesh has been made an ordinary
module that claims a seat: the store seat and the broker seat are each filled by a
module that runs its own code, provisions for its consumers, and is built and
delivered like any other ([ADR 0047](../../02-DECISIONS/0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md),
[ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md),
issue 051). Secrets are the exception. There is **no module that owns a secret.**
Secret handling is instead smeared across three built-in parts of the controller
and node runtime:
- the **controller mints** — one password per consumer↔provider pair, sealed to
both node keys (ADR 0048);
- the **mesh database holds** — a secret is a database record, never a file in the
repository (as-is design, `06-configuration-and-secrets.md`);
- the **synchroniser injects** — a generated secret is written into a node's
generated env file and kept stable across regenerations.
Nothing is the owner of "a secret" the way the store module is the owner of "a
database." The consequences are the gaps we already have written down separately:
- **Generated local secrets exist but cannot be rotated by a mesh operation.** The
as-is design records the weakness verbatim — *"Rotation is not a mesh operation…
there is no mechanism that rotates one and informs everything holding it"*. A
`rotate <provision>` command has since been added and proven for provisioned
provider↔consumer credentials, but it reaches **only** those pairs; a secret a
module generates for its own fully-local use (the password of a version-pinned
embedded database, for instance — see issue 067) is minted by the mesh and then
has no operation that can remake it.
- **Three secret paths, no single provider behind them.** Provisioned credentials
(ADR 0048), generated local secrets (a manifest's generated-secret env var), and
operator-delivered secrets (`secret accept`, sealed to a node) are three separate
mechanisms. Nothing unifies "the mesh has a secret and is responsible for its
whole life."
## Why it matters beyond this instance
- **It is the same de-specialisation the mesh already committed to, left half
done.** Making the store and broker seats ordinary modules was a deliberate
decision precisely so infrastructure would not be a privileged property of the
controller that no module owns, cannot be reasoned about as a module, and cannot
be built or replaced like one. Secret-minting is still exactly that privileged
controller property. Either the store/broker decision was right and this should
follow it, or it was wrong — but the mesh should not be half one and half the
other with no record of why.
- **Rotation, backup, audit and break-glass have nowhere to live.** These are
provider responsibilities everywhere else (the store provisioner rotates a DB
credential; a provider is where a resource's lifecycle lives). With no secrets
provider, each of these is either absent or a one-off in the controller. The mesh
being migrated onto has a working secrets subsystem to learn from — the source
mesh exposes locate / backup / verify / break-glass / preflight / generate
operations — none of which has an owner on the nox side.
- **It blocks the "assume every old secret leaked" step of the migration.** The
cutover plan ends with a full rotation of every credential on the assumption the
old ones are compromised. Today that step is only expressible for provisioned
pairs; app-internal and generated-local secrets must be rotated by hand, which is
the exact operation the design says has taken services down when done wrong.
## Open questions
- Is the vault a **module that claims a seat** (like store and broker), a provider
that offers a `secret` provision that other modules `require`, or both — a seat
whose claimant is also the provider of secrets to everyone else?
- Does a consumer request a secret the way it requests a database (`requires:
secret`, the vault mints and delivers it), so that a fully-local embedded
service's password is a provisioned secret rather than a magic generated env var?
- Does the vault **subsume** the three existing paths (provisioned credentials,
generated local secrets, operator-delivered secrets), or sit beside them owning
only rotation/backup/audit? Subsuming is cleaner but is a migration of every
provider that mints today.
- Is the vault **node-scoped** like every other provider (per issue 067) — each
node's own vault — or is there a case for a single mesh vault, and if so how does
that survive the same objections that made the store node-scoped?
- What does rotation of a secret mean when the holder is a local-only service the
mesh cannot reach as a consumer — does the vault restart the holder, or hand the
new value to the holding module's own runtime to apply?
- How does break-glass work — recovering a secret when the normal sealed path is
unavailable — without reintroducing a key some single place holds, which ADR 0048
went out of its way to avoid?