From 1ce9ffe79f91f4079d93ca2012bf1f35d311476b Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 20 Sep 2026 20:29:18 +0200 Subject: [PATCH 1/4] =?UTF-8?q?Issue=20067=20=E2=80=94=20a=20provision=20c?= =?UTF-8?q?annot=20name=20which=20provider=20serves=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mesh models provisions as mesh-scoped (one provider of a kind, a single mesh-store). But node-specific services delivered to the mesh was the plan from the start: both nodes already run their own postgres, SQL server, redis and object store, and identity — currently single — already serves apps on a second node. The model cannot express which provider serves a consumer, so it collapses a deliberately per-node fleet to one. Provider scoping is a whole-mesh decision across postgres/s3-bucket/oidc, not an SSO patch. Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx --- .../00-report.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md diff --git a/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md b/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md new file mode 100644 index 0000000..20a7bcd --- /dev/null +++ b/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md @@ -0,0 +1,78 @@ +--- +status: open +opened: 2026-09-20 +located-in: [] +fixed-by: +amended-design: +--- + +# A provision cannot name which provider serves it + +## Symptom, as observed + +The mesh models every provision — `postgres-database`, `s3-bucket`, `amqp`, a +future `oidc` — as **mesh-scoped**: there is one provider of a given kind for the +whole mesh, and a consumer that requires the provision is bound to *that* one. +`scope: "mesh"` is written into the provision definitions, and the adopted store +is a single `mesh-store`. + +The mesh being migrated onto is not shaped that way, and never was meant to be. +**Node-specific services delivered to the mesh was the plan from the start.** Each +node already runs its own provider of the same kinds: + +- Both control-capable nodes run their **own general-purpose postgres server** + (the same image, one per node), serving that node's own applications. +- Each node runs its **own** SQL server, its **own** redis, its **own** object + store — infrastructure is per node, by design, not a single mesh-wide instance. +- Identity is the *only* provision that is currently single (one realm on one + node), and even that already serves applications hosted on a **second** node. + +So the multi-provider reality is not a future edge case that appears "the day a +second provider is added" — it is the founding topology, true today, on every +provision kind. What is missing is any way to **say it**. A consumer requires +`postgres-database`; it cannot require *this node's* postgres rather than *that +node's*. It requires `oidc`; it cannot name which node's identity provider. The +mesh model collapses a deliberately per-node fleet down to one mesh-scoped +provider, and a consumer has no field in which to choose. + +## Why it matters beyond this instance + +- **The model regressed an intended topology, it did not merely miss a corner + case.** "Node-specific services delivered to the mesh" is the design; `scope: + "mesh"` with a single `mesh-store` expresses the opposite. This is a gap between + a stated intent and what the manifests can represent, which is exactly what the + issues process is for. +- **It is not an identity special case.** The missing concept — *a provision has a + provider, providers are per-node, and a consumer names the provider* — is the + same for databases, object stores, brokers and identity. A fix aimed only at SSO + would leave the same wall standing behind postgres and minio, both of which are + *already* multi-provider on the live mesh. +- **The single-provider assumption is silent.** Nothing rejects a second provider + of a mesh-scoped provision; the model just cannot address it, so a consumer binds + to whichever one is "the" provider — by accident of there being one, or by a race + when there are two. A rule enforced by nothing ("there is one provider per + provision") reads as true until the second node's provider makes it false, with + no diagnostic at the seam. +- **It blocks the migration concretely.** The mesh already has applications on one + node depending on another node's provider (identity today; databases the moment + an app is assigned to a node whose local postgres is not "the" mesh store). + Modelled as mesh-scoped, that topology is expressible only by accident. To carry + it deliberately the provision must be able to name its provider. + +## Open questions + +- Where does the provider name live — on the provision definition (`scope: "node"` + with a provider identity), on the requirement in the consumer's manifest, or + supplied only at assignment time so the same module can be bound to different + providers on different assignments? +- Is "mesh-scoped" still a legitimate scope for some provisions (a single mesh CA, + say), or does every provision become node-scoped, with a single instance + expressed as "there happens to be one"? +- What is the default when a consumer names no provider — bind to the node the + consumer is assigned to (co-located provider), require the name always, or fall + back to a mesh-wide default provider where one is declared? +- How does a provider's identity survive being moved between nodes, so a consumer's + recorded choice does not silently rebind when the provider relocates? +- Does this interact with secret rotation (the issue-scope of `rotate`) — must + rotation address a specific provider's credential holders rather than "the + provision's"? From c4ff0478b7e4955860658a4e0505c04caad2e93e Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 20 Sep 2026 21:11:35 +0200 Subject: [PATCH 2/4] =?UTF-8?q?Issue=20067=20=E2=80=94=20fold=20in=20the?= =?UTF-8?q?=20embedded-vs-provisioned=20axis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Naming which provider is only half of how a module gets a database. The other half: a module may carry its own version/fork-pinned instance, module-network only, no published port, not a provision — and the model has no word for it. Add it as a second axis with its own open questions. Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx --- .../00-report.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md b/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md index 20a7bcd..72b3108 100644 --- a/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md +++ b/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md @@ -59,6 +59,36 @@ provider, and a consumer has no field in which to choose. Modelled as mesh-scoped, that topology is expressible only by accident. To carry it deliberately the provision must be able to name its provider. +## A second axis: provisioned against a provider, or embedded and private + +Naming *which* provider is only half of "how a module gets a database". There is a +second, distinct case the model also cannot express: a module that does **not** +consume a shared provider at all, but carries its **own** instance inside its own +composition — on its own module network, publishing no host port, visible to +nothing else in the mesh. This is legitimate and sometimes necessary: some +containers pin a database *server* version or need a fork or extension set (a +customised postgres, a vector extension) that the node's shared provider does not +offer, so they must run their own alongside the main container. + +The distinction that matters: + +- **Provisioned** — the module requires a provision and is bound to a *named + node-scoped provider* (the first axis above). This should be the default; on the + mesh being migrated onto, most per-module databases are vanilla servers on old + version pins that could simply be consolidated onto the node's shared provider. +- **Embedded and private** — the module ships its own instance because a fork or + version genuinely forces it. Its credential is still a mesh-generated secret, not + a module-authored password; but the *instance* is module-internal — same module + network only, no published port, **not registered as a provision**, so nothing + else can bind to it and it cannot collide on a well-known port. + +The model has no word for the second case. A module that carries a private instance +looks, to the mesh, either like nothing (an undeclared container) or like a provider +it must not be treated as. "Embed only when a fork or version forces it; otherwise +provision against the node's provider" is the rule the design should be able to +state and check — and the invisibility of an embedded instance (own network, no host +port, not a provision) should be an enforceable property, not a convention. + ## Open questions - Where does the provider name live — on the provision definition (`scope: "node"` @@ -76,3 +106,8 @@ provider, and a consumer has no field in which to choose. - Does this interact with secret rotation (the issue-scope of `rotate`) — must rotation address a specific provider's credential holders rather than "the provision's"? +- When a module carries an **embedded, private** instance rather than consuming a + provider, how is that declared so the mesh knows it is module-internal — not a + provision, not published, not bindable by anything else — and can enforce it? +- What decides embed-vs-provision — is it the module's declaration alone, or may an + operator override at assignment (consolidate this one onto the node's provider)? From d66579c8f63fe3e60412c9e82d3a9b2ea202e6bd Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 20 Sep 2026 21:11:35 +0200 Subject: [PATCH 3/4] =?UTF-8?q?Issue=20068=20=E2=80=94=20secrets=20have=20?= =?UTF-8?q?no=20owning=20module?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Store and broker seats were made ordinary modules; secret-minting is still a privileged property of the controller that no module owns. Propose the vault become a module that provides a secret provision (generate/hold/rotate/backup/ audit), node-scoped like every other provider (issue 067), subsuming the three secret paths and giving local-secret rotation a home. Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx --- .../00-report.md | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 04-ISSUES/068-secrets-have-no-owning-module/00-report.md diff --git a/04-ISSUES/068-secrets-have-no-owning-module/00-report.md b/04-ISSUES/068-secrets-have-no-owning-module/00-report.md new file mode 100644 index 0000000..bc2fdbb --- /dev/null +++ b/04-ISSUES/068-secrets-have-no-owning-module/00-report.md @@ -0,0 +1,90 @@ +--- +status: open +opened: 2026-09-20 +located-in: [] +fixed-by: +amended-design: +--- + +# Secrets have no owning module + +## Symptom, as observed + +Every other piece of shared infrastructure in the mesh has been made an ordinary +module that claims a seat: the store seat and the broker seat are each filled by a +module that runs its own code, provisions for its consumers, and is built and +delivered like any other ([ADR 0047](../../02-DECISIONS/0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md), +[ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md), +issue 051). Secrets are the exception. There is **no module that owns a secret.** + +Secret handling is instead smeared across three built-in parts of the controller +and node runtime: + +- the **controller mints** — one password per consumer↔provider pair, sealed to + both node keys (ADR 0048); +- the **mesh database holds** — a secret is a database record, never a file in the + repository (as-is design, `06-configuration-and-secrets.md`); +- the **synchroniser injects** — a generated secret is written into a node's + generated env file and kept stable across regenerations. + +Nothing is the owner of "a secret" the way the store module is the owner of "a +database." The consequences are the gaps we already have written down separately: + +- **Generated local secrets exist but cannot be rotated by a mesh operation.** The + as-is design records the weakness verbatim — *"Rotation is not a mesh operation… + there is no mechanism that rotates one and informs everything holding it"*. A + `rotate ` command has since been added and proven for provisioned + provider↔consumer credentials, but it reaches **only** those pairs; a secret a + module generates for its own fully-local use (the password of a version-pinned + embedded database, for instance — see issue 067) is minted by the mesh and then + has no operation that can remake it. +- **Three secret paths, no single provider behind them.** Provisioned credentials + (ADR 0048), generated local secrets (a manifest's generated-secret env var), and + operator-delivered secrets (`secret accept`, sealed to a node) are three separate + mechanisms. Nothing unifies "the mesh has a secret and is responsible for its + whole life." + +## Why it matters beyond this instance + +- **It is the same de-specialisation the mesh already committed to, left half + done.** Making the store and broker seats ordinary modules was a deliberate + decision precisely so infrastructure would not be a privileged property of the + controller that no module owns, cannot be reasoned about as a module, and cannot + be built or replaced like one. Secret-minting is still exactly that privileged + controller property. Either the store/broker decision was right and this should + follow it, or it was wrong — but the mesh should not be half one and half the + other with no record of why. +- **Rotation, backup, audit and break-glass have nowhere to live.** These are + provider responsibilities everywhere else (the store provisioner rotates a DB + credential; a provider is where a resource's lifecycle lives). With no secrets + provider, each of these is either absent or a one-off in the controller. The mesh + being migrated onto has a working secrets subsystem to learn from — the source + mesh exposes locate / backup / verify / break-glass / preflight / generate + operations — none of which has an owner on the nox side. +- **It blocks the "assume every old secret leaked" step of the migration.** The + cutover plan ends with a full rotation of every credential on the assumption the + old ones are compromised. Today that step is only expressible for provisioned + pairs; app-internal and generated-local secrets must be rotated by hand, which is + the exact operation the design says has taken services down when done wrong. + +## Open questions + +- Is the vault a **module that claims a seat** (like store and broker), a provider + that offers a `secret` provision that other modules `require`, or both — a seat + whose claimant is also the provider of secrets to everyone else? +- Does a consumer request a secret the way it requests a database (`requires: + secret`, the vault mints and delivers it), so that a fully-local embedded + service's password is a provisioned secret rather than a magic generated env var? +- Does the vault **subsume** the three existing paths (provisioned credentials, + generated local secrets, operator-delivered secrets), or sit beside them owning + only rotation/backup/audit? Subsuming is cleaner but is a migration of every + provider that mints today. +- Is the vault **node-scoped** like every other provider (per issue 067) — each + node's own vault — or is there a case for a single mesh vault, and if so how does + that survive the same objections that made the store node-scoped? +- What does rotation of a secret mean when the holder is a local-only service the + mesh cannot reach as a consumer — does the vault restart the holder, or hand the + new value to the holding module's own runtime to apply? +- How does break-glass work — recovering a secret when the normal sealed path is + unavailable — without reintroducing a key some single place holds, which ADR 0048 + went out of its way to avoid? From fc4ab370d642a0e19db9061685b6f08f30891f7a Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 20 Sep 2026 21:19:31 +0200 Subject: [PATCH 4/4] Graduate issues 067 and 068 to decisions and to-be designs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR 0084 (extends 0027) — a provision is served by a node-scoped provider the consumer selects, defaulting to co-location; a module may instead carry a private embedded instance that is not a provision. Design: 01-to-be/23-choosing-a-provider. ADR 0085 (extends 0031) — a secret is a provision and the vault is the module that provides it; a module's own local secret becomes an ordinary pair credential that rotates through the existing machinery, while the controller's provisioning-credential mint (0048) is unchanged. Design: 01-to-be/24-the-secrets-vault; doc 13 amended to cross-link the non-pair secret. Issues 067/068 marked resolved with amended-design set. ADR index regenerated; records and index checks pass. Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx --- .../0084-which-provider-serves-a-consumer.md | 108 ++++++++++++++++++ 02-DECISIONS/0085-a-secret-is-a-provision.md | 107 +++++++++++++++++ 02-DECISIONS/README.md | 2 + .../13-credentials-and-their-rotation.md | 19 ++- 03-DESIGN/01-to-be/23-choosing-a-provider.md | 87 ++++++++++++++ 03-DESIGN/01-to-be/24-the-secrets-vault.md | 98 ++++++++++++++++ 03-DESIGN/01-to-be/README.md | 2 + .../00-report.md | 4 +- .../00-report.md | 4 +- 9 files changed, 426 insertions(+), 5 deletions(-) create mode 100644 02-DECISIONS/0084-which-provider-serves-a-consumer.md create mode 100644 02-DECISIONS/0085-a-secret-is-a-provision.md create mode 100644 03-DESIGN/01-to-be/23-choosing-a-provider.md create mode 100644 03-DESIGN/01-to-be/24-the-secrets-vault.md diff --git a/02-DECISIONS/0084-which-provider-serves-a-consumer.md b/02-DECISIONS/0084-which-provider-serves-a-consumer.md new file mode 100644 index 0000000..dacabb6 --- /dev/null +++ b/02-DECISIONS/0084-which-provider-serves-a-consumer.md @@ -0,0 +1,108 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-20 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md +--- + +# 84. Which provider serves a consumer, when the mesh runs more than one + +## Context + +[ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) settled what a provision +is *named* for — the thing the consumer's code is coupled to, so `postgres-database` and +`mssql-database` are different provisions and a wrong match is refused at resolution. It said one +thing more, in passing, and left it: *"Two providers of `postgres-database` — a container on this +node and a managed instance elsewhere — are interchangeable and should both match."* Naming was +decided; **which of several providers serves a given consumer was not.** + +The mesh assumes there is only one to choose. Provisions are mesh-scoped: the adopted store is a +single `mesh-store` a consumer on any node reaches, and `scope: "mesh"` is written into the +provision definitions. **That assumption is false on day one, and was always meant to be.** +Node-specific services delivered to the mesh is the plan, not an edge case: + +- Both control-capable nodes already run their **own** general-purpose relational store (the same + engine, one instance each), serving that node's own applications. +- Each runs its **own** SQL server, its **own** cache, its **own** object store. One node alone + runs six separate relational-store instances, each raised by the module that needed it. +- The one provision that is currently single — the identity provider, one instance on one node — + already authenticates applications whose home is a **different** node. + +So several providers of one provision name genuinely coexist, and they are **not** interchangeable +the way 0027's aside supposed. They differ by node, by the data they hold, and by locality. A +consumer bound to the wrong one reads the wrong database, or takes a cross-node hop it did not +need, or cannot be moved without silently rebinding. The model has no field in which to say which +one. This is 0027's own fault — *a match that resolves and is wrong* — one level up: 0027 refused +the wrong **dialect**; nothing refuses, or even asks about, the wrong **instance**. + +## Considered Options + +1. **Keep `scope: "mesh"` — one provider per provision, mesh-wide.** Rejected: it is false on day + one, and making it true would force every node's applications onto one node's server — the + exact opposite of node-specific services delivered to the mesh, and a single point of failure + the topology was built to avoid. +2. **Resolve to any provider of the name (0027's "both match").** Rejected: when providers hold + different data and live on different nodes they are not interchangeable, and picking one + arbitrarily is a wrong-instance match — the confidently-wrong answer 0027 exists to prevent, + restated at the level of the instance rather than the dialect. +3. **Always require the consumer to name the provider explicitly.** Rejected: needless ceremony in + the common case, where the consumer wants the provider on its own node; and a field every + manifest must carry is a field an author forgets, which then matches everything again — the + failure 0027 warned about for qualifiers. +4. **A provision is node-scoped; the consumer selects the provider, defaulting to co-location.** + Adopted. + +## Decision + +**A provision is served by a provider identified by its node, and the consumer selects which one.** +A provider is a (node, module) pair, not a mesh-wide singleton. A consumer's binding resolves to a +specific provider, and the selection is part of the assignment +([ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)), not the +manifest. + +**The default is co-location.** A consumer that names no provider is served by the provider of +that provision on **its own node**. This is the common case and needs nothing said. A mesh with +one provider of a kind is just the case where co-location and "the only one" coincide — expressed +as *there happens to be one*, not as a scope. + +**A consumer coupled to a provider's data names it.** Where two consumers must share one database, +or a consumer must reach a provider on another node, the assignment names that provider — because +that coupling is exactly what may not be guessed, and naming it is what makes a later move safe. + +**A module need not consume a provider at all.** It may carry its **own** instance inside its own +composition — on its own module network, publishing no host port, **not** declared as a provision — +when a genuine engine fork or a pinned server version makes the shared provider unusable. Such an +instance is invisible to resolution and can be bound by nothing else. The rule is *share by +default; embed only when a fork or a version forces it* — most of the per-module stores that exist +today are vanilla engines on stale pins that a consolidation onto the node's provider would absorb. + +## Consequences + +The mesh can carry its real topology **deliberately** rather than by the accident of which +provider happened to be the single one. A consumer's data-coupling becomes a stated fact, which is +what lets a provider be moved without a consumer silently following the wrong one — provided a +provider keeps its identity across a relocation, which is a follow-up this record opens rather than +closes. Rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)) addresses a +specific provider's holders, not "the provision's". + +What got harder: an assignment now may carry a provider selection, and a wrong one is a new way to +misconfigure. It is mitigated the way 0027 mitigated its own: the co-location default removes the +choice in the common case, and genuine ambiguity — several providers, none named, none co-located — +is refused with the candidates named, never resolved by picking. + +## References + +- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — the naming this extends; + its "both match" aside is the gap closed here. +- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is exactly such a + provider, and already serves consumers on another node. +- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md) — where the + selection lives. +- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the adopted store, whose mesh-wide + binding is the single-provider assumption this record replaces. +- [issue 067](../04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md) — the + gap, and the day-one evidence. +- [`03-DESIGN/01-to-be/23-choosing-a-provider.md`](../03-DESIGN/01-to-be/23-choosing-a-provider.md) + — the design. diff --git a/02-DECISIONS/0085-a-secret-is-a-provision.md b/02-DECISIONS/0085-a-secret-is-a-provision.md new file mode 100644 index 0000000..ca0c54e --- /dev/null +++ b/02-DECISIONS/0085-a-secret-is-a-provision.md @@ -0,0 +1,107 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-20 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md +--- + +# 85. A secret is a provision, and the vault is the module that provides it + +## Context + +[ADR 0031](0031-the-control-plane-authenticates-nobody.md) decided that identity *runs on the +mesh, not of it* — a module other modules require, rather than a privileged part of the +controller. [ADR 0078](0078-the-store-and-broker-are-modules.md) did the same for the store and +the broker: the twelve-module floor has no specialty left in it. **Secrets are the exception that +survived.** No module owns a secret. + +Secret handling is smeared across three built-in parts of the runtime: + +- the **controller mints** one credential per consumer↔provider pair and seals it to both node + keys ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)); +- the **mesh generates local secrets** — *"generated secrets are the mesh's, never authored"* + (as-is, `06-configuration-and-secrets.md`) — for a value a single module needs for its own use; +- the **synchroniser injects** both into a node's generated files. + +Nothing is the owner of "a secret" the way the store module is the owner of "a database", and the +cost is recorded rather than hypothetical. The as-is design names the weakness in its own words: +*"Rotation is not a mesh operation… there is no mechanism that rotates one and informs everything +holding it."* A `rotate` command has since been built and proven, but it reaches **only** the +provisioned pairs; a secret a module generates for its own fully-local use — the password of a +version-pinned embedded store ([ADR 0084](0084-which-provider-serves-a-consumer.md)), an internal +token — is minted by the mesh and then has no operation that can remake it. Three species of +secret, and only the first has an owner: + +| species | minted by | rotates? | +|---|---|---| +| a provisioned credential (a database login) | controller, sealed to nodes (0048) | yes — `rotate`, per pair | +| a module's own local secret | the mesh, as a generated value | **no owner, no rotation** | +| an operator-delivered secret (an external key) | a person, sealed in (`secret accept`) | no rotation, no audit | + +## Considered Options + +1. **Leave it a property of the controller.** Rejected: it is the smear above — no owner, local + secrets that cannot be rotated, operator secrets that cannot be audited — and it is exactly the + specialty 0078 removed for the store and broker, kept here for no reason anyone recorded. +2. **A dedicated vault built into the foundation, not a module.** Rejected: it reintroduces a + privileged built-in, the thing 0031 and 0078 went out of their way to remove, and a mesh that + wants none would still carry it. +3. **Fold all minting, the controller's provisioning credentials included, into the vault.** + Rejected: the controller must mint in order to **deliver** any provision — the vault's own + credential among them — so making the vault mint the credential of its own delivery is the + store/broker chicken-and-egg for no gain. The provisioned-pair credential already has an owner + and a rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)); this record + does not disturb it. +4. **A secret is a provision; the vault is an ordinary module that provides it.** Adopted. + +## Decision + +**Secret-holding is a module, parallel to identity.** A module that needs a secret **for its own +use** — a local service's password, an internal token, an external key it was handed — requires a +`secret` provision from a vault provider, exactly as it requires a database from the store. The +vault generates the value (or holds one it was given), and because the credential belongs to the +consumer↔vault pair it **rotates, backs up and is audited through the same per-pair machinery** +[13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md) already defines. That is what +gives the second and third species the rotation and audit they lack. A module's own secret stops +being a generated value that nothing owns and becomes an ordinary provision with a provider. + +**The controller's minting of provisioning-pair credentials is unchanged** ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)): +it is how every provision, the vault's own included, is delivered. The vault does not mint the +mesh's delivery credentials; it provides secrets to modules, and is itself provisioned the ordinary +way. + +**The vault is node-scoped like every provider** ([ADR 0084](0084-which-provider-serves-a-consumer.md)): +each node its own, selected the same way, so a module's own secret is held by the vault on the +module's node. **A mesh that wants no vault runs none** — a module requiring no secret needs +nothing, which is the same test 0031 applied to identity. + +## Consequences + +The gap that opened this — a generated local secret with no rotation — **closes without new +machinery**: rotating such a secret is the vault's provisioner remaking a pair credential, the +operation 13 already specifies. Backup, audit and break-glass gain an owner — the vault module — +and become things a design specifies rather than absences. The as-is sentence *"rotation is not a +mesh operation"* is already false for provisioned pairs and, once the vault ships, for local +secrets too; the as-is document is updated when it does, not before. + +What got harder: a break-glass path — recovering a secret when the sealed delivery path is +unavailable — must not reintroduce a key that one place holds, which is the property +[ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) was built to preserve; how +the vault offers recovery without it is left to the design as an open question. And a module that +today bakes a password into its own composition must instead require it from the vault — a +migration taken module by module, not a flag day. + +## References + +- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is a module; this is the + same move for secrets. +- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) — the provisioning-credential + path, left unchanged. +- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the de-specialisation this completes. +- [ADR 0084](0084-which-provider-serves-a-consumer.md) — the node-scoping the vault obeys. +- [issue 068](../04-ISSUES/068-secrets-have-no-owning-module/00-report.md) — the gap. +- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md) — + the design. The source mesh's `secret_locate` / `secret_backup` / `secret_verify` / + `secret_breakglass` subsystem is the prior art it draws on. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index d61a1e6..1207c69 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -136,6 +136,8 @@ python3 00-META/checks/index.py fail if stale - **0053** — [A scheduled step is a container run on a recurring schedule](0053-a-step-that-runs-on-a-schedule.md) - **0054** — [Model usage is a vendor-neutral record, produced by the adapter, at two grains](0054-model-usage-is-recorded-at-two-grains.md) - **0055** — [Model access is answered by a licence, or by a node that hosts the model](0055-model-access-is-answered-by-a-licence-or-a-node.md) +- **0084** — [Which provider serves a consumer, when the mesh runs more than one](0084-which-provider-serves-a-consumer.md) +- **0085** — [A secret is a provision, and the vault is the module that provides it](0085-a-secret-is-a-provision.md) ### How it is built diff --git a/03-DESIGN/01-to-be/13-credentials-and-their-rotation.md b/03-DESIGN/01-to-be/13-credentials-and-their-rotation.md index c2afcf1..47d1f33 100644 --- a/03-DESIGN/01-to-be/13-credentials-and-their-rotation.md +++ b/03-DESIGN/01-to-be/13-credentials-and-their-rotation.md @@ -5,10 +5,11 @@ code: - mesh-controller internal/inventory/secrets.go - mesh-controller cmd/mesh-controller/rotate.go - mesh-controller examples/postgres-provisioner -updated: 2026-09-01 +updated: 2026-09-20 decisions: - 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md - 02-DECISIONS/0009-modules-and-the-graph.md + - 02-DECISIONS/0085-a-secret-is-a-provision.md --- # 13 — Credentials, and moving them @@ -105,3 +106,19 @@ a provider that added a password and removed nothing. **Not over loopback.** `pg_hba` trusts anything there, so every password looks correct — a deliberately wrong one returned a row for an afternoon before that was noticed. + + +## The secret that is not a pair + +Everything on this page is about the credential *between a consumer and a provider* — the login +one module uses against another. A mesh also holds secrets that are not that: a value a single +module needs for **its own** use, and a value only an operator can supply. Those had no owner, and +so no rotation — the gap this page's machinery could not reach because there was no pair to rotate. + +[ADR 0085](../../02-DECISIONS/0085-a-secret-is-a-provision.md) gives them one. Such a secret is +provided by a **vault** module ([24](24-the-secrets-vault.md)): a module requires a `secret` +provision, and the credential of that consumer↔vault pair is an ordinary pair credential — so it +rotates, and is queried for who holds it, through exactly the machinery described above, unchanged. +The rotation a module's own secret lacks is not a second mechanism; it is this one, pointed at a +secret the vault provides. What this page proves for a database password holds, by construction, +for a secret from the vault. diff --git a/03-DESIGN/01-to-be/23-choosing-a-provider.md b/03-DESIGN/01-to-be/23-choosing-a-provider.md new file mode 100644 index 0000000..50b3ed3 --- /dev/null +++ b/03-DESIGN/01-to-be/23-choosing-a-provider.md @@ -0,0 +1,87 @@ +--- +layer: to-be +status: designed +code: [] +updated: 2026-09-20 +decisions: + - 02-DECISIONS/0084-which-provider-serves-a-consumer.md + - 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md +--- + +# 23 — Choosing a provider + +A provision is named for what the consumer's code is coupled to +([ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md)): +`postgres-database`, not `database`. That decides *what kind* of provider satisfies a requirement. +It does not decide *which* provider, and the mesh runs more than one of most kinds. + +## Why there is a choice at all + +Node-specific services delivered to the mesh is the design, not an exception. Every control-capable +node runs its own relational store, its own cache, its own object store; a single node may run +several relational stores, each raised by the module that needed a particular engine or version. +The one provision that is single today — the identity provider — already serves applications whose +home is another node. So for a given provision name there are usually several providers, one per +node, and they are **not** interchangeable: each holds different data and lives in a different +place. A consumer bound to the wrong one reads the wrong database or takes a network hop it did not +need. + +Naming the kind is therefore only half of "how a consumer gets what it needs". The other half is +which provider, and it has two shapes: **consume a provider**, or **carry your own**. + +## Consuming a provider + +A provider is not a mesh-wide singleton. It is identified by the node it runs on together with the +module that provides it — a (node, module) pair. A consumer's requirement resolves to one such +provider, and which one is part of the **assignment**, not the manifest +([ADR 0046](../../02-DECISIONS/0046-a-module-configuration-is-its-assignments-not-its-manifest.md)): +the same module, assigned twice, may be served by two different providers. + +**The default is co-location.** A consumer that names no provider is served by the provider of that +provision on its own node. This is the ordinary case and is meant to need nothing said — a module +that wants a database wants, almost always, the database on the machine it runs on. A mesh that +happens to run exactly one provider of a kind is simply the case where co-location and "the only +one there is" name the same thing; that is *there happens to be one*, not a mesh-wide scope written +into the provision. + +**Coupling to data is named.** The exception to co-location is a consumer coupled to a *particular +provider's contents*: two modules that must share one database, or a consumer that must reach a +provider on a different node. That coupling is exactly what may not be guessed, so the assignment +names the provider. Naming it is also what makes a later move safe — the mesh knows the binding is +to that provider and not to whichever one is nearest. + +**Ambiguity is refused, never resolved by picking.** If several providers of a kind exist, none is +named, and none is co-located, the requirement is unsatisfiable and is refused with the candidates +shown — the same stance +[ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md) took +against a confidently-wrong match, applied to the instance rather than the dialect. A wrong answer +delivered quietly costs more than a refusal. + +## Carrying your own + +A module need not consume a provider at all. It may carry its **own** instance of an engine inside +its own composition — reachable only on the module's own network, publishing no host port, and +**not** declared as a provision. Nothing else in the mesh can see it or bind to it, and it cannot +collide with anything on a well-known port. To resolution it does not exist; it is an internal part +of the module, like any other container the module runs. + +This is legitimate but it is the exception, and the design says when: **only when a genuine engine +fork or a pinned server version makes the shared provider unusable.** A module written against a +customised engine, or one that needs an extension the node's provider does not carry, has no choice +but to carry its own. A module that merely pins an old image of an ordinary engine does not — the +version on a compose file is the *server's*, and the application talks to a newer shared server +perfectly well once its data is migrated in. The rule is *share by default; embed only when a fork +or a version forces it*. Most of the per-module stores that exist in the mesh being migrated onto +are the first kind wearing the second's clothes, and consolidate onto the node's provider. + +The distinction is worth stating because the two cases look identical from outside — a module with +a database either way — and the mesh must be able to tell them apart to reason about either. A +consumed provider is a binding the mesh records, rotates and can move. An embedded instance is a +private detail the mesh does not manage and must not mistake for a provider. + +## What is not settled here + +A provider that moves between nodes must keep its identity, so that a consumer's recorded choice +does not silently rebind to a different provider that inherited its place. That is a property the +provider lifecycle must supply, and this document names it as a requirement rather than describing +its mechanism. diff --git a/03-DESIGN/01-to-be/24-the-secrets-vault.md b/03-DESIGN/01-to-be/24-the-secrets-vault.md new file mode 100644 index 0000000..020ac8c --- /dev/null +++ b/03-DESIGN/01-to-be/24-the-secrets-vault.md @@ -0,0 +1,98 @@ +--- +layer: to-be +status: designed +code: [] +updated: 2026-09-20 +decisions: + - 02-DECISIONS/0085-a-secret-is-a-provision.md + - 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md + - 02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md +--- + +# 24 — The secrets vault + +Identity runs on the mesh, not of it — a module other modules require, not a part of the +controller ([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)). The +store and the broker are ordinary modules too +([ADR 0078](../../02-DECISIONS/0078-the-store-and-broker-are-modules.md)). Secrets are the piece +that never got the same treatment: nothing owns a secret. This document describes the module that +does — a **vault** that provides a `secret` provision — and, as importantly, the boundary of what +it owns and what it deliberately does not. + +## Three kinds of secret, and which the vault owns + +A mesh handles three species of secret, and confusing them is how the current arrangement went +wrong. + +- **A provisioned credential** is the login one module uses against another — a database password, + a broker account. The mesh mints it per consumer↔provider pair and seals it to the machines that + must hold it ([ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md)), + and its provider makes it true. **This is not the vault's**, and the vault does not change it. It + already has an owner and a rotation + ([13](13-credentials-and-their-rotation.md)); disturbing it would buy nothing. +- **A module's own secret** is a value a single module needs for itself — the password of a store + it runs privately, an internal signing token. Today the mesh generates this as a value nothing + owns, and so it cannot be rotated. **This is the vault's.** +- **An operator-delivered secret** is a value only a person can supply — a credential for something + outside the mesh. Today it is sealed in and then held, un-audited and un-rotatable. **The vault + holds this**, and can hand it out and audit it, though it cannot generate it. + +The vault, then, is the provider a module turns to for a secret that is **not** the byproduct of +some other provision. It is the answer to *"this module needs a password, and there is no provider +whose job it is to give it one."* + +## A secret as a provision + +A module that needs a secret for its own use requires a `secret` provision, exactly as it requires +a database from the store. The vault generates the value — or takes custody of one an operator +delivered — and the credential belongs to the consumer↔vault pair. Because it is an ordinary pair +credential, **everything already built for pair credentials applies to it unchanged**: it rotates +with the one command that discards a credential and delivers both ends together, it is one secret +per holder so rotating one touches nothing else, and *who holds this* is a query rather than an +assumption ([13](13-credentials-and-their-rotation.md)). The rotation a module's own secret lacks +today is not new machinery; it is the machinery that already moves a database password, pointed at +a secret the vault provides. + +This is what replaces the "generated value that nothing owns". A module's own password stops being +a special kind of thing injected by the synchroniser and becomes a provision with a provider, a +holder, and a lifecycle — the same shape as everything else the mesh grants. + +## The vault is a module, and node-scoped + +The vault is an ordinary module. A mesh that wants one runs it; a mesh whose modules require no +secret of their own runs none — the same test identity meets +([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)). It is provisioned +the ordinary way, and it does **not** mint the mesh's delivery credentials — the controller does +that, because the controller must mint in order to deliver any provision, the vault's own included. +The vault mints secrets *for modules*, downstream of its own existence, never the credential that +delivers it. + +Like every provider it is node-scoped +([ADR 0084](../../02-DECISIONS/0084-which-provider-serves-a-consumer.md), [23](23-choosing-a-provider.md)): +each node may run its own vault, and a module's own secret is held by the vault on the module's +node, selected the same way any provider is. There is no single mesh vault holding everything, for +the same reason there is no single mesh store. + +## Beyond generate and hold + +Owning a secret means owning more than its creation. The mesh being migrated onto has a working +secrets subsystem whose surface names the operations a vault is responsible for — locating a secret, +backing it up, verifying it is what it should be, and a break-glass recovery for when the normal +path is unavailable. These become the vault module's, specified against it rather than scattered. + +One of them is left open on purpose. **Break-glass must not reintroduce a key that one place +holds.** The whole point of sealing a secret to the machine that needs it, asymmetrically, is that +no single place can open everything +([ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md)); a +recovery path that keeps a master key would undo exactly that. How the vault lets an operator +recover a secret without becoming the thing the sealing was designed to prevent is a question this +design opens and does not yet answer. + +## What this changes for a module + +A module that today writes a password into its own composition — an embedded store's login, an +internal token — instead requires it from the vault and reads it where the mesh puts it. The change +is taken module by module, not as a flag day, and it is the same change in each: a value the module +authored becomes a value the vault provides. When it is done, the guarantee the as-is design already +makes for generated secrets — *nothing in the repository contains a credential* — holds for a +module's own secrets not by convention but because there is a provider whose job it is to keep them. diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index 1b38359..5c10294 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -32,6 +32,8 @@ document is written and this one's status becomes `implemented`. | [`20-writing-a-module.md`](20-writing-a-module.md) | A worked guide: one module, four capabilities, four languages, and the packages it publishes | [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md), [ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md), [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md) | | [`21-the-installation-in-full.md`](21-the-installation-in-full.md) | Every step from a bare machine to a mesh that maintains itself, and what is not yet true | [ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md), [ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md), [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) | | [`22-the-work-ahead.md`](22-the-work-ahead.md) | Everything decided and not yet built, in dependency order, each phase ending at a run | [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md), [ADR 0075](../../02-DECISIONS/0075-two-stores-and-which-provides-what.md), [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) | +| [`23-choosing-a-provider.md`](23-choosing-a-provider.md) | Which of several providers of a kind serves a consumer, and when a module carries its own instead | [ADR 0084](../../02-DECISIONS/0084-which-provider-serves-a-consumer.md), [ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md) | +| [`24-the-secrets-vault.md`](24-the-secrets-vault.md) | The module that owns a secret — a `secret` provision, and the boundary of what it owns | [ADR 0085](../../02-DECISIONS/0085-a-secret-is-a-provision.md), [ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md), [ADR 0048](../../02-DECISIONS/0048-a-provider-creates-the-credential-the-mesh-minted.md) | ## Not yet written diff --git a/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md b/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md index 72b3108..411feae 100644 --- a/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md +++ b/04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md @@ -1,9 +1,9 @@ --- -status: open +status: resolved opened: 2026-09-20 located-in: [] fixed-by: -amended-design: +amended-design: 03-DESIGN/01-to-be/23-choosing-a-provider.md --- # A provision cannot name which provider serves it diff --git a/04-ISSUES/068-secrets-have-no-owning-module/00-report.md b/04-ISSUES/068-secrets-have-no-owning-module/00-report.md index bc2fdbb..6c63b96 100644 --- a/04-ISSUES/068-secrets-have-no-owning-module/00-report.md +++ b/04-ISSUES/068-secrets-have-no-owning-module/00-report.md @@ -1,9 +1,9 @@ --- -status: open +status: resolved opened: 2026-09-20 located-in: [] fixed-by: -amended-design: +amended-design: 03-DESIGN/01-to-be/24-the-secrets-vault.md --- # Secrets have no owning module