mesh-vault: tests for the review's fixes, and the README saying what at-rest encryption protects (review 250)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

modules/mesh-vault/README.md

@@ -18,16 +18,51 @@ Each record holds the value twice, never in the clear:
 
 - **at rest**, sealed to the vault's own key (the sealed box of ADR 0206, `mesh-sdk/go/seal`). The key is
   an X25519 pair the vault makes once in `${dir:state}/vault-key.json`; its private half never leaves it.
+  The key is made only for an empty store: with the key file missing and secrets in the store, the vault
+  does not start and says so, rather than make a key that opens none of them.
 - **for the operator**, sealed to the operator key (ADR 0113 §22), so nothing exists only inside the
   vault. A secret is refused, loudly, while the vault has no operator key.
 
+Every file is written whole, synced, renamed into place and its directory synced.
+
+### What the encryption at rest protects against, and what not
+
+The vault's private key lives in plaintext in `${dir:state}`, on the same disk as the store, readable by the
+vault's account alone. So:
+
+- **It protects** against a copy of the store directory alone — a `store` backup, a file sent somewhere,
+  a listing of `${dir:store}` by a tool that reads files — holding anything usable, and against a value
+  being found by searching the disk for it.
+- **It does not protect** against anyone who can read the vault's state directory too: root on the
+  vault's node, the vault's own account, or a backup that holds both `state` and `store` (both are
+  class `valuable`, so the node's backup holds both). Against those, at-rest encryption adds nothing; the
+  protection is the node's own security. Keeping the key elsewhere (another directory class, a key the
+  operator unlocks) is for V3's review, before real secrets move in.
+
 ## Where the operator key comes from
 
 From the export the mesh writes to the vault's root directory (`${dir:root}/export.json`, the manifest's
 `keeps`): its `operator-key` is the public half `operator key set` recorded in the controller. The vault
 checks it every minute; a new key makes the vault seal every operator copy again to it — it can, because
-it opens its own copy. The operator key's private half never enters the mesh, and the vault refuses one
-(ADR 0113 §2). When the secrets move to the vault (design 57's V3), the operator key moves with them.
+it opens its own copy. Each record names the key its copy is sealed to, so a failure partway is finished on
+the next minute's pass, and nothing is left sealed to an earlier key for good.
+
+The vault takes an operator key only as SPKI PEM (`SetOperatorKey`). A PEM private key is refused, and so
+is a raw 32-byte key, because a raw private key looks the same. The one raw key it reads is the export's,
+which the controller writes from what it recorded; the vault wraps it as SPKI before taking it. It keeps
+the key raw in base64, the form of the operator's NaCl box. When the secrets move to the vault (design 57's
+V3), the operator key moves with them.
+
+### How the operator opens a copy
+
+`mesh-vault.secret_vault_export` answers the vault's copies in the shape `secret export` writes. Saved to a
+file, one is opened with the operator's private key by
+
+    nox secret recover vault mesh-vault <name> --key <operator key file> --from-export <file> --out <file>
+
+(`--out` is needed: a name holds `/`). `TestTheOperatorRecoversAVaultSecretFromItsExport` reads the file
+as that command does and opens it as it does. `secret_export` still answers the controller's export, as
+before; until V3 the controller's secrets are there.
 
 ## The boxes and the key formats — settled
 
@@ -59,6 +94,10 @@ vault (ADR 0349 §16), V4 adds opening the ADR 0206 box with the same sealing ke
 `sealed` being the secret sealed to that key, and only to an asker allowed for that secret, the asker being
 the bus user the runtime proved — never an argument.
 
+Every answer and every refusal is recorded in the vault's own store (`handed.jsonl`: who, which secret,
+which generation, when, and why a refusal), synced before the answer is given — as ADR 0277 §25 records
+asks. `secret_handed` reads it.
+
 **Who may ask, today: nobody.** The rule (ADR 0349 §11) is a node's provisioner client under a bus account
 of its own, for what its node's declaration names, and a module under its own account (issue 569). Neither
 exists yet, so the running vault refuses every asker; the tests set a stand-in list. The real rule goes
@@ -68,16 +107,28 @@ behind the `Askers` interface in `askers.go` (design 57's V4 and P3).
 
 - `secret-changed` — `name`, `by`, `at`, `generation`: which secret, who changed it, when. Never a value
   or a sealed copy.
-- `provisioned`, `rotated`, `deprovisioned` — the custody ledger's, by fingerprint, as before.
+- `provisioned`, `rotated` — the custody ledger's, by fingerprint, as before.
+- `provisioner.retirement` (ADR 0230 §35) — `waiting`, `settled`, `approved`, `rejected`, `retired`,
+  `reenabled`, `deleted`, naming holders, never a value.
 
 ## The tools
 
 `secret_export`, `secret_holders`, `secret_verify` answer as the TypeScript vault's did, and its custody
 ledger (`${dir:ledger}`, the same JSON files) is kept by a pass over the contributions file every minute.
-None returns a value.
+`secret_handed` and `secret_vault_export` read the vault's own store. None returns a value.
 
-What the TypeScript vault did and this one does not: it consumed its own three events only to print an
-audit line; the pass now prints that line itself.
+**The ledger follows ADR 0230.** A contributions file that does not say who is asked for (unreadable, not
+JSON, no `given` list or a null one, or for another requirement) is not "nobody": the pass records nothing,
+retires nobody and starts its count again. A holder no longer asked for is retired — a mark with when and
+why in its own ledger file, its history kept — only after the same set has gone unasked in five passes and
+ten minutes; more than three at once, or more than half of those held, waits for a person
+(`provisioner_retire_approve`, `provisioner_retire_reject`); asked for again it is re-enabled; it is deleted
+only by `provisioner_delete`, a person's act with a why. `provisioner_retirement` shows it all. The vault is
+mark-only: it holds no access to switch off.
+
+What the TypeScript vault did and this one does not: it consumed its own events only to print an audit
+line, which the pass now prints itself; and it withdrew a holder on the first pass that missed it, with its
+history, and said `deprovisioned` — that is gone for ADR 0230's retirement.
 
 ## What comes later
 

modules/mesh-vault/cmd/mesh-vault/vault_test.go

@@ -53,6 +53,16 @@ func operatorKey(t *testing.T) (public, private string) {
 	return base64.StdEncoding.EncodeToString(pub[:]), base64.StdEncoding.EncodeToString(priv[:])
 }
 
+// asPEM is a raw operator key as the SPKI PEM the vault takes.
+func asPEM(t *testing.T, raw string) string {
+	t.Helper()
+	p, err := publicPEM(raw)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
 // openOperator opens an operator copy the way `secret recover --from-export` does.
 func openOperator(t *testing.T, public, private, sealed string) string {
 	t.Helper()
@@ -87,7 +97,7 @@ func newVault(t *testing.T, askers Askers) testVault {
 	if err != nil {
 		t.Fatal(err)
 	}
-	own, err := OwnKey(filepath.Join(dir, "state"))
+	own, err := OwnKey(filepath.Join(dir, "state"), store)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -96,7 +106,7 @@ func newVault(t *testing.T, askers Askers) testVault {
 	v := &Vault{Store: store, Own: own, Askers: askers, Emit: bus.emit,
 		Now: func() time.Time { clock = clock.Add(time.Second); return clock }, Log: t.Logf}
 	pub, priv := operatorKey(t)
-	if _, err := v.SetOperatorKey(pub, "the test"); err != nil {
+	if _, err := v.SetOperatorKey(asPEM(t, pub), "the test"); err != nil {
 		t.Fatal(err)
 	}
 	return testVault{Vault: v, dir: dir, bus: bus, operatorPublic: pub, operatorPrivate: priv}
@@ -227,9 +237,16 @@ func TestNoPrivateKeyIsAccepted(t *testing.T) {
 		}
 	}
 
-	// Not as the operator key.
-	if _, err := v.SetOperatorKey(keys.PrivateKey, "the test"); err == nil {
-		t.Error("the vault took a private key as the operator key")
+	// Not as the operator key: neither a PEM private key nor a raw key, which looks the same as a raw
+	// private one.
+	rawPub, rawPriv := operatorKey(t)
+	for _, key := range []string{keys.PrivateKey, rawPriv, rawPub} {
+		if _, err := v.SetOperatorKey(key, "the test"); err == nil {
+			t.Errorf("the vault took %.20q as the operator key", key)
+		}
+	}
+	if k, _ := v.Store.OperatorKey(); k != v.operatorPublic {
+		t.Error("a refused operator key replaced the operator key")
 	}
 
 	// And nothing it publishes or writes names one that it refused.
@@ -278,7 +295,7 @@ func TestEverySecretHasAnOperatorSealedCopy(t *testing.T) {
 
 	// A new operator key: every copy is sealed again to it, and none stays sealed to the old one.
 	pub, priv := operatorKey(t)
-	n, err := v.SetOperatorKey(pub, "the test")
+	n, err := v.SetOperatorKey(asPEM(t, pub), "the test")
 	if err != nil || n != 3 {
 		t.Fatalf("a new operator key sealed %d copies again (%v), not 3", n, err)
 	}
@@ -296,7 +313,7 @@ func TestEverySecretHasAnOperatorSealedCopy(t *testing.T) {
 func TestNoOperatorKeyKeepsNothing(t *testing.T) {
 	dir := t.TempDir()
 	store, _ := OpenStore(filepath.Join(dir, "store"))
-	own, _ := OwnKey(filepath.Join(dir, "state"))
+	own, _ := OwnKey(filepath.Join(dir, "state"), store)
 	v := &Vault{Store: store, Own: own, Emit: (&published{}).emit, Now: time.Now, Log: t.Logf}
 	if _, err := v.Put("own/g14/dunst/token", "x", "the test"); err == nil || !strings.Contains(err.Error(), "operator key") {
 		t.Fatalf("a secret was kept with no operator key to seal its copy to: %v", err)
@@ -485,11 +502,11 @@ func TestASecretsNameIsNotAPath(t *testing.T) {
 // The vault's own key is made once, kept beside it, and the same on the next start.
 func TestTheVaultsOwnKeyIsMadeOnce(t *testing.T) {
 	dir := t.TempDir()
-	a, err := OwnKey(dir)
+	a, err := OwnKey(dir, nil)
 	if err != nil {
 		t.Fatal(err)
 	}
-	b, err := OwnKey(dir)
+	b, err := OwnKey(dir, nil)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -501,3 +518,136 @@ func TestTheVaultsOwnKeyIsMadeOnce(t *testing.T) {
 		t.Errorf("the vault's key is %v", info.Mode())
 	}
 }
+
+// The vault's key gone while its store holds secrets: no new key is made, and the vault does not start.
+func TestAMissingKeyBesideSecretsIsRefused(t *testing.T) {
+	v := newVault(t, nil)
+	if _, err := v.Put("pair/postgres/gitea", "x", "the test"); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.Remove(filepath.Join(v.dir, "state", OwnKeyFile)); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := OwnKey(filepath.Join(v.dir, "state"), v.Store); err == nil || !strings.Contains(err.Error(), "MISSING") {
+		t.Fatalf("a new key was made beside secrets sealed to the lost one: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(v.dir, "state", OwnKeyFile)); err == nil {
+		t.Error("a key file was written all the same")
+	}
+}
+
+// A new operator key that fails partway leaves no copy sealed to the earlier key for good: the next call
+// with the same key — every minute, from the mesh's export — finishes the rest.
+func TestANewOperatorKeyFailingPartwayIsFinished(t *testing.T) {
+	v := newVault(t, nil)
+	for _, name := range []string{"a/one", "b/two", "c/three"} {
+		if _, err := v.Mint(name, "the test"); err != nil {
+			t.Fatal(err)
+		}
+	}
+	pub, priv := operatorKey(t)
+	v.Store.failWrite = func(r Record) error {
+		if r.Name == "b/two" {
+			return os.ErrPermission
+		}
+		return nil
+	}
+	n, err := v.SetOperatorKey(asPEM(t, pub), "the test")
+	if err == nil || n != 2 {
+		t.Fatalf("a failure partway said %d sealed again, err %v", n, err)
+	}
+	if left, _ := v.StillSealedToAnother(); left != 1 {
+		t.Fatalf("%d copies left on the earlier key, not 1", left)
+	}
+	v.Store.failWrite = nil
+	if n, err := v.Reseal(); err != nil || n != 1 {
+		t.Fatalf("the retry sealed %d again (%v), not the 1 left", n, err)
+	}
+	if n, err := v.SetOperatorKey(asPEM(t, pub), "the test"); err != nil || n != 0 {
+		t.Fatalf("the same key again sealed %d (%v)", n, err)
+	}
+	all, _ := v.Store.List()
+	for _, rec := range all {
+		if rec.Operator.Key != pub {
+			t.Errorf("%s is still sealed to an earlier key", rec.Name)
+		}
+		openOperator(t, pub, priv, rec.Operator.Sealed)
+	}
+}
+
+// Who was handed which secret, and when, and who was refused, is in the vault's own store.
+func TestWhoWasHandedWhatIsRecordedInTheStore(t *testing.T) {
+	v := newVault(t, AllowList{"g14.node-provisioner": {"pair/postgres/gitea"}})
+	if _, err := v.Put("pair/postgres/gitea", "the-value", "the test"); err != nil {
+		t.Fatal(err)
+	}
+	asker, _ := seal.GenerateKeyPair()
+	if _, err := v.Get("g14.node-provisioner", "pair/postgres/gitea", asker.PublicKey); err != nil {
+		t.Fatal(err)
+	}
+	_, _ = v.Get("ace.node-provisioner", "pair/postgres/gitea", asker.PublicKey)
+	handed, err := v.Store.Handed(10)
+	if err != nil || len(handed) != 2 {
+		t.Fatalf("%v %v", handed, err)
+	}
+	if handed[1].To != "g14.node-provisioner" || handed[1].Outcome != "handed" || handed[1].Generation != 1 || handed[1].At.IsZero() {
+		t.Errorf("the hand-out is recorded as %+v", handed[1])
+	}
+	if handed[0].To != "ace.node-provisioner" || handed[0].Outcome != "refused" || handed[0].Why == "" {
+		t.Errorf("the refusal is recorded as %+v", handed[0])
+	}
+	raw, _ := os.ReadFile(filepath.Join(v.dir, "store", "handed.jsonl"))
+	if strings.Contains(string(raw), "the-value") {
+		t.Error("the record of hand-outs holds a value")
+	}
+}
+
+// The vault's operator copies reach the operator's recovery: secret_vault_export is a document
+// `secret recover --from-export` reads, found by node "vault", module "mesh-vault" and the secret's name,
+// and opened with the operator's private key as that command opens it (mesh-controller
+// cmd/mesh-controller/secret.go keptFromExport, internal/secrets Open).
+func TestTheOperatorRecoversAVaultSecretFromItsExport(t *testing.T) {
+	v := newVault(t, nil)
+	if _, err := v.Put("pair/postgres/gitea", "recover-me", "the test"); err != nil {
+		t.Fatal(err)
+	}
+	answer, err := toolNamed(t, definitions(&Env{Vault: v.Vault}), "secret_vault_export").Run(map[string]any{})
+	if err != nil {
+		t.Fatal(err)
+	}
+	file := filepath.Join(t.TempDir(), "vault-export.json")
+	raw, _ := json.Marshal(answer)
+	_ = os.WriteFile(file, raw, 0o600)
+
+	// What `secret recover vault mesh-vault pair/postgres/gitea --from-export <file>` reads.
+	var e struct {
+		Kept []struct {
+			Node, Module, Name, Sealed, Key string
+		} `json:"kept"`
+	}
+	read, _ := os.ReadFile(file)
+	if err := json.Unmarshal(read, &e); err != nil {
+		t.Fatal(err)
+	}
+	var sealed string
+	for _, k := range e.Kept {
+		if k.Node == "vault" && k.Module == "mesh-vault" && k.Name == "pair/postgres/gitea" {
+			sealed = k.Sealed
+			if k.Key != v.operatorPublic {
+				t.Errorf("the export names key %s, not the operator's", k.Key)
+			}
+		}
+	}
+	if sealed == "" {
+		t.Fatalf("the export holds no copy of the secret: %s", read)
+	}
+	// secrets.Open: the public half derived from the private key, then the anonymous box.
+	privRaw, _ := base64.StdEncoding.DecodeString(v.operatorPrivate)
+	key, _ := ecdh.X25519().NewPrivateKey(privRaw)
+	if got := openOperator(t, base64.StdEncoding.EncodeToString(key.PublicKey().Bytes()), v.operatorPrivate, sealed); got != "recover-me" {
+		t.Fatalf("recovered %q", got)
+	}
+	if strings.Contains(string(read), "recover-me") {
+		t.Error("the export holds the value in the clear")
+	}
+}