Files
mesh-catalog/modules/mesh-vault

mesh-vault

Holds the mesh-vault seat: the mesh's shared secrets, each kept sealed, handed out only on request and reply, sealed to whoever asked (novox/hq ADR 0085, ADR 0113, ADR 0349 §7 and §11). Go, one binary (cmd/mesh-vault), launched by its node's runtime and speaking MCP to it through the Go SDK.

Where the secrets are kept, and why there

In the vault's own store: one file per secret in ${dir:store} on the vault's own disk, mode 0600 in a 0700 directory. Not a database, as most Go modules keep their data, because:

  • the vault is installed before any other module built on the runtime base (ADR 0113 §14), and a database login is itself a shared secret the vault makes — the vault cannot need one to start;
  • every value in the store is ciphertext anyway, so a database would add nothing a file does not;
  • no secret is in a bucket, sealed or not (ADR 0201 §13), so the module declares no state.

Each record holds the value twice, never in the clear:

  • at rest, sealed to the vault's own key (the sealed box of ADR 0206, mesh-sdk/go/seal). The key is an X25519 pair the vault makes once in ${dir:state}/vault-key.json; its private half never leaves it. The key is made only for an empty store: with the key file missing and secrets in the store, the vault does not start and says so, rather than make a key that opens none of them.
  • for the operator, sealed to the operator key (ADR 0113 §22), so nothing exists only inside the vault. A secret is refused, loudly, while the vault has no operator key.

Every file is written whole, synced, renamed into place and its directory synced.

What the encryption at rest protects against, and what not

The vault's private key lives in plaintext in ${dir:state}, on the same disk as the store, readable by the vault's account alone. So:

  • It protects against a copy of the store directory alone — a store backup, a file sent somewhere, a listing of ${dir:store} by a tool that reads files — holding anything usable, and against a value being found by searching the disk for it.
  • It does not protect against anyone who can read the vault's state directory too: root on the vault's node, the vault's own account, or a backup that holds both state and store (both are class valuable, so the node's backup holds both). Against those, at-rest encryption adds nothing; the protection is the node's own security. Keeping the key elsewhere (another directory class, a key the operator unlocks) is for V3's review, before real secrets move in.

Where the operator key comes from

From the export the mesh writes to the vault's root directory (${dir:root}/export.json, the manifest's keeps): its operator-key is the public half operator key set recorded in the controller. The vault checks it every minute; a new key makes the vault seal every operator copy again to it — it can, because it opens its own copy. Each record names the key its copy is sealed to, so a failure partway is finished on the next minute's pass, and nothing is left sealed to an earlier key for good.

The vault takes an operator key only as SPKI PEM (SetOperatorKey). A PEM private key is refused, and so is a raw 32-byte key, because a raw private key looks the same. The one raw key it reads is the export's, which the controller writes from what it recorded; the vault wraps it as SPKI before taking it. It keeps the key raw in base64, the form of the operator's NaCl box. When the secrets move to the vault (design 57's V3), the operator key moves with them.

How the operator opens a copy

mesh-vault.secret_vault_export answers the vault's copies in the shape secret export writes. Saved to a file, one is opened with the operator's private key by

nox secret recover vault mesh-vault <name> --key <operator key file> --from-export <file> --out <file>

(--out is needed: a name holds /). TestTheOperatorRecoversAVaultSecretFromItsExport reads the file as that command does and opens it as it does. secret_export still answers the controller's export, as before; until V3 the controller's secrets are there.

The boxes and the key formats — settled

Two boxes on the mesh today, both over X25519:

format keys who uses it
NaCl anonymous box XSalsa20-Poly1305, box.SealAnonymous raw 32 bytes, base64 the controller sealing into declarations; the node-engine's sealing.key (mesh-host internal/identity/sealing.go); the operator key and secret recover --from-export
the sealed box (ADR 0206) X25519, HKDF-SHA256, AES-256-GCM, {v, eph, iv, tag, ct} PEM: SPKI public, PKCS #8 private the licence manager and claude-code; mesh-sdk/go/seal

What the vault uses:

  • Its answers to get are the ADR 0206 box, for every asker, a node included. The asker sends its public key as SPKI PEM. A node's provisioner client (V4) asks with a key of its own, made when it starts and kept in its memory alone: it runs under an account of its own and cannot read the node-engine's sealing key, which is root's. A node's sealing key would open an answer too, being X25519 (wrapped as SPKI and PKCS #8, TestANodesSealingKeyOpensTheAnswer), for the node-engine's own asks. A raw 32-byte key is refused: a raw private key looks the same, and the vault takes no private key.
  • Its operator copies are the NaCl anonymous box, to the operator key as it is made today, so the operator's recovery opens them with the key and the command it already has.
  • The at-rest copy is the ADR 0206 box, to the vault's own key.

The NaCl box stays on the node-engine only for what the controller seals into declarations, until the last module has moved (ADR 0349 §17). For the core's own credentials that travel in a declaration sealed by the vault (ADR 0349 §16), the node-engine is to open the ADR 0206 box with the same sealing key; that is not built yet (issue 647 leaves it to its own issue).

The seat's verb

mesh-vault.get — name, public_key (SPKI PEM). It answers {name, generation, changed-at, sealed}, sealed being the secret sealed to that key, and only to an asker allowed for that secret, the asker being the bus user the runtime proved — never an argument.

Every answer and every refusal is recorded in the vault's own store (handed.jsonl: who, which secret, which generation, when, and why a refusal), synced before the answer is given — as ADR 0277 §25 records asks; a secret whose hand-out cannot be recorded is not handed out. A refusal is recorded only for a caller of a bus user's shape and a well-formed secret name, so an arbitrary argument fills nothing. The record is bounded: rolled at 1 MiB into handed.jsonl.1 … .3, the oldest dropped, each roll said loudly in the journal and counted in vault_status. secret_handed reads it.

vault_status says what needs a person: operator copies still sealed to an earlier key, an operator key in the export the vault refused, and how often the record was rolled.

A failing re-seal is a condition (ADR 0366, issue 738). The vault re-seals its operator copies to the operator key once a minute. vault_health answers unhealthy once that has failed on three passes in a row, saying what failed and why, and healthy again after one pass succeeds. The manifest declares it as the health of the vault's own runtime (runtime-health), which the controller puts on mesh-vault.own-runtime; the node-engine asks it every minute and the controller raises module.mesh-vault.<machine>.unhealthy from it, cleared on the first healthy answer. An operator key in the export the vault refuses fails the pass too. How often the record was rolled still reaches a person through the journal and vault_status only.

Who may ask: a node's provisioner client, for what its plan names (ADR 0349 §11, design 57 V4, issue 647; askers.go). The caller is provisioner.<node>, the client's own account (never the tool runner's, which an agent on that machine shares), and the secret is own/<node>/<module>/<name> for that same machine, where the controller's assignment <node>.<module> (the assignments mirror, which the manifest reads) carries the provisioner's plan and the plan's secrets name it; or a value a provider's software issued for one holder, provision/<provider>/<provision>/<node>/<module>, where that plan's provisions name the provision from that provider with a credential's name (issue 757; an entry naming only the provider serving a provision, issue 760, hands nothing). The vault cannot read a node's declaration, so the plan the controller carries beside it is what the declaration names, as the vault reads it. A provision's one credential, provision/<provider>/<provision>, is handed to the machines the vault made it for (below) while the holder there is handed to the provisioner: the provider's module, or a consumer whose plan still names it from that provider. A module not handed to the provisioner, another machine's secret or holder value, and every other caller are refused, and each refusal is recorded with why. A module's own account (issue 569) is not answered yet.

The controller, for the core's own credentials, sealed only to their machine (ADR 0349 §16, issue 763; core.go). The bus and mesh-provisioner need their files before any client runs, so their credentials travel in the declaration sealed by the vault to their machine, which the controller carries and cannot open. The caller controller is answered own/<node>/<module>/<name> only for a module on the core's fixed list (nats, mesh-provisioner, mesh-provisioner-client, the controller's list too), and only when the key it names is that machine's sealing key as its own node-engine stated it in its own name, in move-in's first step (sealing_key). A machine whose engine stated none, any other key, any other module and any other name are refused and recorded.

Taking values in: accept, secret-ask, issued, rotate

Design 57's V2 (issue 675; ADR 0349 clauses 8 to 10, ADR 0277 §22–§27 and ADR 0305 with the vault in the controller's place). intake.go.

From the controller alone; the vault asks the desk and the provider itself. The controller alone asks these verbs, as the runtime proves its caller. Each value comes in sealed to an X25519 key the vault made for that one call (raw, base64 — the NaCl anonymous box the desk's prompt and a provider's tool already seal to), and opens with the private half, which lives in this process only and is forgotten once used or after two minutes. A value sealed to any other key is refused, and a call is answered once.

The vault's own account (issue 723). The vault runs as mesh-vault in a runtime of its own (runs-as, ADR 0259 §8), on a bus account of its own (<node>.mesh-vault, from its own secret broker). The bus lets that account call the desks' prompt (node-launcher.secret on any machine) and each issuing provider's tool in its own name, and nothing else outside its seat (mesh-controller's internal/broker grants, proven on a real server). So the prompt and the tool know the vault asks, and the controller carries neither the key nor the sealed value. The controller still asks them itself for a module filled the old way, until R1 (issue 682) removes that path; they answer both until then. Everything the vault keeps is its account's: its directories, the export it keeps, the grant files the controller still writes for it.

  • accept — two steps: node, module, name (a module's own secret) or provider, provision (a provision's one credential), and by; then call and sealed. The controller's terminal seals the value it read to the call's key before it leaves the terminal. Kept with origin accepted; an accepted provision credential is never remade (rotate refuses it); a value a provider's software issued is not taken by accept.
  • secret-ask — one step: node, module, name, desk and asked_by (who asked the controller, as the bus named its caller). The ask is recorded in asks.jsonl before anything else, and refused while another ask for the same secret is open (two minutes at most) or when three were made in the last hour. The vault then asks the desk's prompt itself, naming who asked as the record has it, with timeout_seconds 20. The answer is how it ended (given, dismissed, timed-out, empty, refused, failed) and who asked; how each ask ended is recorded. secret_asks reads the record. The hand-act log stays the controller's: it records the act once the vault has taken the value.
  • issued — one step: provision, provider (machine), module and tool. The vault asks that tool on the provider's machine itself; the value is kept as the provision's one credential with origin issued. The same value again writes nothing, says nothing and keeps the generation (ADR 0305 §5).
  • issued for each holder apart (issue 749; ADR 0349 clause 11, ADR 0365 clause 4: the forge's read token per agent account). The controller names the provision's holders from that provider (holders, each a node and a consuming module), and the vault asks the tool with them. The tool answers either one value (sealed, kept as above) or one per holder (holders: node, module and sealed each, all sealed to the call's key). Each is kept as provision/<provider node>/<provision>/<holder node>/<holder module>, origin issued, and the same value again for a holder writes nothing for it. The answer is taken whole or not at all: a holder not asked about, one named twice, a value not sealed to the call or an empty one keeps nothing. A holder asked about and not answered is said (not_answered) and keeps nothing. get hands a holder's value to that holder alone, its machine's provisioner client (provisioner.<node>) or the module under an account of its own (<node>.<module>), whatever the rule for who may ask allows; every other caller is refused and the refusal recorded. TestOnARealBusTwoHoldersOnTwoMachinesEachGetOnlyTheirOwnIssuedValueAndNoneCrossesInTheClear proves it on a NATS server with a stand-in forge.
  • rotate — a module's own secret or a provision's credential, made anew. Refused for an issued value (the software rotates it, ADR 0305 §9) and an accepted provision credential (clause 8). From the controller alone, which knows whether the module's secrets are filled from the vault yet, and with a why, said in the vault's log. Who asked is the by the controller names, beside the controller; both are kept (changed-by, changed-via). The controller's rotate <provision> forwards here for a provider filled the new way (issue 767): a shared value is made anew for every holder, never for one consumer alone; for a provider whose consumers keep a login each, consumer and consumer_module (--consumer, --module) name the one consumer whose own login is remade, and no other is touched.

Names in the store: own/<node>/<module>/<name>, provision/<provider node>/<provision> and, for a value issued for one holder, provision/<provider node>/<provision>/<holder node>/<holder module>. V3 copies the controller's rows in under these names.

Which modules reach the vault this way. The controller sends a module's verbs here only when its secrets are filled the new way (its machine's provisioner client asks the vault for them, V4); every other module keeps the controller's path, so no value lands where its machine never looks (ADR 0349 clause 17; the release lead's decision on issue 675).

What the issuing provider's test is here. TestAnIssuingProviderAnswersOnlyTheVaultsCallSealedToTheVault and TestTheVaultAsksTheIssuingToolItselfAndOnlyASealedValueCrosses use a stand-in provider that answers the vault's account (and, until R1, the controller) and no other caller, and only sealed to the call's key. plex's own tool answers the controller alone today; it must also answer <node>.mesh-vault when it moves with the media catalogue, after release 57.

Events

  • secret-changed — name, by, at, generation: which secret, who changed it, when. Never a value or a sealed copy.
  • provisioned, rotated — the custody ledger's, by fingerprint, as before.
  • provisioner.retirement (ADR 0230 §35) — waiting, settled, approved, rejected, retired, reenabled, deleted, naming holders, never a value.

The tools

secret_export, secret_holders, secret_verify answer as the TypeScript vault's did, and its custody ledger (${dir:ledger}, the same JSON files) is kept by a pass over the contributions file every minute. secret_handed and secret_vault_export read the vault's own store. None returns a value.

The ledger follows ADR 0230. A contributions file that does not say who is asked for (unreadable, not JSON, no given list or a null one, or for another requirement) is not "nobody": the pass records nothing, retires nobody and starts its count again. A holder no longer asked for is retired — a mark with when and why in its own ledger file, its history kept — only after the same set has gone unasked in five passes and ten minutes; more than three at once, or more than half of those held, waits for a person (provisioner_retire_approve, provisioner_retire_reject); asked for again it is re-enabled; it is deleted only by provisioner_delete, a person's act with a why. provisioner_retirement shows it all. The vault is mark-only: it holds no access to switch off.

The words are the controller's: the provisioner.retirement event is mesh-controller's link.Retirement (provider-node, consumers as objects) and the provisioner_retirement answer its link.RetirementState (held, bound, retires_by: mark-only), so retire approve, retire reject and cleanup list read the vault and a set waiting for a person raises the controller's urgent condition. The acts take what the SDK's harness takes: approve and reject repeat the set in consumers and refuse any other set; delete repeats the name in confirm; each records the by and via the controller sends. testdata/retirement-*.json are those types marshalled by the controller (feat/57 at dc36536), and the tests hold the vault's words to them.

What the TypeScript vault did and this one does not: it consumed its own events only to print an audit line, which the pass now prints itself. Its retirement ran on the SDK's harness, which this one follows: mark-only, and a holder's ledger file removed only through a person's delete, as here. The difference is where the mark lives: the harness kept it in its process, so a restart forgot it (ADR 0230 §13); here it is in the ledger file.

The credential a provider shares with its consumers (ADR 0349 clause 8, ADR 0158 §4–§6, design 57 V5, issue 677; pair.go). One value per provider assignment and provision, provision/<provider>/<provision>. Once a minute the vault reads every plan in the assignments mirror; each plan's provisions names the provider serving each provision its module requires (issue 760). A consumer binds when an entry appears and unbinds when it goes or its assignment goes; on either, the vault makes one new value for every holder together and says secret-changed, and each holder's client asks again. What the move brought in (V3) is adopted when every pair row of a provider and provision shares one generation stamp and one value (ADR 0158's shared form), with its origin, and nothing is remade. Rows without a stamp (ADR 0048, a login per consumer) are never made one value (ADR 0158 §11): rotate of the provision remakes each consumer's own apart, and each consumer's machine is handed only its own row, pair/<provider>/<provision>/<consumer>/<module>[/<local>] (the provider's machine all of them). A provider the vault holds no row of gets its first value in the form each plan entry names (credential-form, from the provider's offer, issue 768): shared, one value for every holder; pair, a login of its own for each consumer bound; issued or given are never made. No form named, plans that disagree, or a form it does not know: nothing is made. An accepted or issued value is never remade, only its holders recorded. A provision whose plans name a credential per holder (issue 757) has no shared value. No value is made that a holder would never get: while the provider's assignment, or a consumer the move brought in, is still filled the old way (no plan), every make, remake and rotation is held back and the reason said in the log. Who holds each value is kept in the store's shared/ directory, never a value.

What comes later

Taking values in is built (above); the controller's secrets move in with V3; nodes' provisioner clients ask with V4, for each module handed to the provisioner; providers answer their consumers with B1–B3.

Tests

go test ./...

vault_test.go is ADR 0349's vault test list as far as this package goes: no bucket holds a secret; no private key is accepted; every secret has an operator-sealed copy; get answers only an allowed asker, sealed to its key; secret-changed carries no value. intake_test.go is ADR 0349's secret-ask test as far as the vault answers it (recorded before the prompt, one open ask per secret and three an hour, the asker named as the bus named it, the typed value opened only here), the issuing provider's test with a stand-in provider (ADR 0305 §13's checks that belong to the vault: nothing rewritten for a value held, rotate refused, no other way in), and accept and rotate. followups_test.go holds V1's review follow-ups.