Credentials the mesh delivers and cannot read

HAL keeps env vars in the registry, encrypted at rest. Its own tooling
records what that bought and what it did not. `secret_locate` matches by
value rather than by name — because the same password sits in
mesh_provisions, in module_env, in each node's .env in plain text, and
inside every connection string composed from it, and its documentation
says those URL copies "are often the only copies actually in use". And a
query against the encrypted column returns zero rows and proves nothing,
so auditing moved to the decrypted copies on the nodes.

Two faults there, and encryption at rest addresses neither: the control
plane can read what it stores, so a copy of the database is a copy of
every credential; and one secret has many homes with nothing tracking
them.

So here the mesh generates a password, seals it to each end with keys
those nodes generated, stores both blobs, and discards the plaintext. It
cannot read what it holds. Neither can the broker relaying it. And
nothing is composed centrally — a connection string is assembled on the
machine that needs one — so no copy is ever minted in a shape nothing
tracks. `Compromise of a node is compromise of that node` (ADR 0004) is
now true of secrets, not only of identity.

Two files rather than one, because the mesh cannot compose a document
containing a value it discarded: `binds` carries the readable facts,
`secrets` carries the credential alone. The readable half stays readable
in the declaration; the secret half changes only when the secret does,
which makes restart-on precise. The provider gets a directory, one file
per consumer, for the same reason.

It is made once and kept — regenerating per declaration would restart
both ends on every push, and the password a provider was told to create
would never be the one its consumer was given. It is remade when either
end's sealing key changes, and both ends learn the new one in the same
push, so there is no window where half the mesh holds a dead credential.

Two tests found passing for the wrong reason, both caught because their
injection came back clean:

- the provider's copy was asserted non-empty, which reads the same
  whichever column is selected. It now opens the blob with the
  provider's own key.
- RotateSecret deleted and re-created; the re-create was dead, because
  the next read makes one anyway. Removed, and a second path to the same
  act is how two ends come to disagree.

And one real fault: three places built a declaration, and the one behind
`--json` predated credentials, so it silently produced a declaration
missing them — a difference between what `plan` showed and what anything
reading `--json` got. There is one path now.

internal/inventory/migrations/0009-secrets.sql

@@ -0,0 +1,44 @@
+-- The key a node's secrets are sealed to, and the sealed secrets themselves.
+--
+-- The arrangement is the opposite of encrypting a credential column. There, the control plane can
+-- read every secret it stores, so a copy of its database is a copy of every credential in the
+-- mesh, and encryption at rest only means somebody needs the process rather than the file. Here
+-- the value is sealed to the node that will use it before it is written, so **this table holds
+-- nothing usable** -- which is what makes novox/hq ADR 0004's "compromise of a node is compromise
+-- of that node" true of secrets and not only of identity.
+--
+-- It also costs the ability to audit by value, and that is the right trade rather than an
+-- oversight: a `where value like ...` over an encrypted column returns zero rows and proves
+-- nothing, so the audit was never real. What is answerable here is which node holds what, which
+-- is the question rotation actually asks.
+
+alter table node add column sealing_key text;
+
+create table secret (
+    -- What it is for. The provision as required -- `database` -- not the module answering it.
+    name          text        not null,
+    consumer      uuid        not null references node(id) on delete cascade,
+    provider      uuid        not null references node(id) on delete cascade,
+
+    -- The same value, sealed twice: once to each end. Two blobs rather than one shared key,
+    -- because a key both ends hold is a key the mesh must also hold to distribute.
+    --
+    -- The plaintext is never written. It exists for the length of one function call, is sealed to
+    -- both recipients, and is discarded -- so rotation means generating a new one rather than
+    -- reading the old one back, which is the only version of rotation that is honest about what
+    -- the mesh knows.
+    for_consumer  text        not null,
+    for_provider  text        not null,
+
+    -- Which key each was sealed to. A node that regenerates its sealing key can no longer open
+    -- what was sealed to the old one, and this is what lets that be reported rather than
+    -- discovered as a service that will not start.
+    consumer_key  text        not null,
+    provider_key  text        not null,
+
+    created_at    timestamptz not null default now(),
+
+    primary key (name, consumer, provider)
+);
+
+create index secret_by_provider on secret (provider);

internal/inventory/nodes.go

@@ -322,6 +322,34 @@ type Overlay struct {
 // closed firewall (novox/hq ADR 0007).
 func (o Overlay) Reachable() bool { return strings.TrimSpace(o.Endpoint) != "" }
 
+// RecordSealingKey keeps the public half of the key this node's secrets are sealed to.
+//
+// Replacing whatever was there. A node that rejoins has generated a new one, and everything
+// sealed to the old key is unreadable to it -- which is why this does not merge and why what it
+// invalidates is reported rather than repaired silently.
+func (i *Inventory) RecordSealingKey(ctx context.Context, node, key string) error {
+	if key == "" {
+		return nil
+	}
+	_, err := i.store.Pool().Exec(ctx,
+		`update node set sealing_key = $2 where id = $1`, node, key)
+	return err
+}
+
+// SealingKeyOf is the key to seal something to for a node, empty if it has none.
+func (i *Inventory) SealingKeyOf(ctx context.Context, name string) (string, error) {
+	var key *string
+	err := i.store.Pool().QueryRow(ctx,
+		`select sealing_key from node where name = $1`, name).Scan(&key)
+	if err != nil {
+		return "", err
+	}
+	if key == nil {
+		return "", nil
+	}
+	return *key, nil
+}
+
 // RecordOverlayKey keeps the public half a node generated.
 func (i *Inventory) RecordOverlayKey(ctx context.Context, node, key string) error {
 	if strings.TrimSpace(key) == "" {

internal/inventory/secrets.go

@@ -0,0 +1,135 @@
+package inventory
+
+import (
+	"context"
+
+	"github.com/novox/mesh-control/internal/secrets"
+)
+
+// Where sealed secrets live.
+//
+// The table holds nothing usable — see the migration and internal/secrets for why that is the
+// design rather than an inconvenience.
+
+// Secret is one provision's credential, sealed to each end.
+type Secret struct {
+	Name        string
+	Consumer    string
+	Provider    string
+	ForConsumer string
+	ForProvider string
+	ConsumerKey string
+	ProviderKey string
+}
+
+// SecretFor is the credential for one provision between two nodes, making one the first time.
+//
+// **Made once and kept**, rather than regenerated whenever it is asked for. A secret that changed
+// on every declaration would restart both ends on every push and would mean the password a
+// provider was told to create never matches the one a consumer was given — which is a mesh that
+// reports success and cannot connect.
+//
+// **Remade when either end's sealing key changes.** A node that rejoined generated a new key and
+// can no longer open what was sealed to the old one, so keeping the blob would deliver something
+// unreadable for ever. The new secret reaches both ends in the same push, which is the only
+// moment they can be changed together.
+func (i *Inventory) SecretFor(ctx context.Context, name, consumer, provider string) (Secret, error) {
+	consumerKey, err := i.SealingKeyOf(ctx, consumer)
+	if err != nil {
+		return Secret{}, err
+	}
+	providerKey, err := i.SealingKeyOf(ctx, provider)
+	if err != nil {
+		return Secret{}, err
+	}
+
+	consumerNode, err := i.NodeByName(ctx, consumer)
+	if err != nil {
+		return Secret{}, err
+	}
+	providerNode, err := i.NodeByName(ctx, provider)
+	if err != nil {
+		return Secret{}, err
+	}
+
+	var held Secret
+	err = i.store.Pool().QueryRow(ctx,
+		`select for_consumer, for_provider, consumer_key, provider_key from secret
+		 where name = $1 and consumer = $2 and provider = $3`,
+		name, consumerNode.ID, providerNode.ID).
+		Scan(&held.ForConsumer, &held.ForProvider, &held.ConsumerKey, &held.ProviderKey)
+	if err == nil && held.ConsumerKey == consumerKey && held.ProviderKey == providerKey {
+		held.Name, held.Consumer, held.Provider = name, consumer, provider
+		return held, nil
+	}
+
+	made, err := secrets.Make(consumerKey, providerKey)
+	if err != nil {
+		return Secret{}, err
+	}
+	_, err = i.store.Pool().Exec(ctx,
+		`insert into secret (name, consumer, provider, for_consumer, for_provider,
+		                     consumer_key, provider_key)
+		 values ($1, $2, $3, $4, $5, $6, $7)
+		 on conflict (name, consumer, provider) do update set
+		   for_consumer = excluded.for_consumer, for_provider = excluded.for_provider,
+		   consumer_key = excluded.consumer_key, provider_key = excluded.provider_key,
+		   created_at = now()`,
+		name, consumerNode.ID, providerNode.ID,
+		made.ForConsumer, made.ForProvider, made.ConsumerKey, made.ProviderKey)
+	if err != nil {
+		return Secret{}, err
+	}
+	return Secret{Name: name, Consumer: consumer, Provider: provider,
+		ForConsumer: made.ForConsumer, ForProvider: made.ForProvider,
+		ConsumerKey: made.ConsumerKey, ProviderKey: made.ProviderKey}, nil
+}
+
+// RotateSecret discards what was there, so the next declaration carries a new one.
+//
+// Only a delete. Nothing reads the old value first, because nothing can — and making the
+// replacement here rather than on the next read would be a second path to the same act, which is
+// how two ends come to hold different passwords.
+//
+// The new secret then reaches both ends on the same push, together, which is what makes rotation
+// a single event rather than a fanout with a window where half the mesh holds a dead credential.
+func (i *Inventory) RotateSecret(ctx context.Context, name, consumer, provider string) error {
+	consumerNode, err := i.NodeByName(ctx, consumer)
+	if err != nil {
+		return err
+	}
+	providerNode, err := i.NodeByName(ctx, provider)
+	if err != nil {
+		return err
+	}
+	_, err = i.store.Pool().Exec(ctx,
+		`delete from secret where name = $1 and consumer = $2 and provider = $3`,
+		name, consumerNode.ID, providerNode.ID)
+	return err
+}
+
+// SecretsFrom is every credential a provider node was issued, so it can be told what to create.
+func (i *Inventory) SecretsFrom(ctx context.Context, provider string) ([]Secret, error) {
+	providerNode, err := i.NodeByName(ctx, provider)
+	if err != nil {
+		return nil, err
+	}
+	rows, err := i.store.Pool().Query(ctx,
+		`select s.name, c.name, s.for_provider from secret s
+		 join node c on c.id = s.consumer
+		 where s.provider = $1 order by s.name, c.name`, providerNode.ID)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	var out []Secret
+	for rows.Next() {
+		s := Secret{Provider: provider}
+		if err := rows.Scan(&s.Name, &s.Consumer, &s.ForProvider); err != nil {
+			return nil, err
+		}
+		out = append(out, s)
+	}
+	return out, rows.Err()
+}

internal/inventory/secrets_test.go

@@ -0,0 +1,232 @@
+package inventory
+
+import (
+	"context"
+	"crypto/ecdh"
+	"crypto/rand"
+	"encoding/base64"
+	"strings"
+	"testing"
+
+	"golang.org/x/crypto/nacl/box"
+)
+
+// aSealingKey is a node's key, keeping the private half so a test can open what was sealed — the
+// only assertion that actually distinguishes "the right blob" from "a blob".
+func aSealingKey(t *testing.T) (string, func(string) ([]byte, bool)) {
+	t.Helper()
+	k, err := ecdh.X25519().GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var pub, priv [32]byte
+	copy(pub[:], k.PublicKey().Bytes())
+	copy(priv[:], k.Bytes())
+	return base64.StdEncoding.EncodeToString(k.PublicKey().Bytes()),
+		func(sealed string) ([]byte, bool) {
+			blob, err := base64.StdEncoding.DecodeString(sealed)
+			if err != nil {
+				return nil, false
+			}
+			return box.OpenAnonymous(nil, blob, &pub, &priv)
+		}
+}
+
+func twoNodesWithKeys(t *testing.T) (*Inventory, context.Context) {
+	t.Helper()
+	inv := fresh(t)
+	ctx := context.Background()
+	for _, n := range []string{"consumer", "provider"} {
+		node, err := inv.AddNode(ctx, n)
+		if err != nil {
+			t.Fatal(err)
+		}
+		key, _ := aSealingKey(t)
+		if err := inv.RecordSealingKey(ctx, node.ID, key); err != nil {
+			t.Fatal(err)
+		}
+	}
+	return inv, ctx
+}
+
+func TestASecretIsMadeOnceAndKept(t *testing.T) {
+	// Regenerating on every declaration would restart both ends on every push, and — worse — the
+	// password a provider was told to create would never be the one its consumer was given.
+	inv, ctx := twoNodesWithKeys(t)
+	first, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	second, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if first.ForConsumer != second.ForConsumer || first.ForProvider != second.ForProvider {
+		t.Fatal("asking twice produced two different credentials")
+	}
+}
+
+func TestTheStoredSecretIsNotTheSecret(t *testing.T) {
+	// The whole point. A copy of this database is not a copy of the mesh's credentials — which is
+	// what an encrypted column does not achieve, because whoever runs the control plane can read
+	// through it.
+	inv, ctx := twoNodesWithKeys(t)
+	got, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var columns []string
+	rows, err := inv.store.Pool().Query(ctx,
+		`select column_name from information_schema.columns where table_name = 'secret'`)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var c string
+		if err := rows.Scan(&c); err != nil {
+			t.Fatal(err)
+		}
+		columns = append(columns, c)
+	}
+	for _, c := range columns {
+		if strings.Contains(c, "password") || strings.Contains(c, "value") ||
+			strings.Contains(c, "plain") {
+			t.Fatalf("the table has a column called %q, which suggests it holds the thing", c)
+		}
+	}
+	if got.ForConsumer == got.ForProvider {
+		t.Fatal("both ends were given the identical blob, so the storage reveals they match")
+	}
+}
+
+func TestANewSealingKeyMeansANewSecret(t *testing.T) {
+	// A node that rejoined generated a new key and can no longer open what was sealed to the old
+	// one. Keeping the blob would deliver something unreadable for ever, reported as configured.
+	inv, ctx := twoNodesWithKeys(t)
+	before, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	node, err := inv.NodeByName(ctx, "consumer")
+	if err != nil {
+		t.Fatal(err)
+	}
+	fresh, _ := aSealingKey(t)
+	if err := inv.RecordSealingKey(ctx, node.ID, fresh); err != nil {
+		t.Fatal(err)
+	}
+	after, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if after.ForConsumer == before.ForConsumer {
+		t.Fatal("the node was handed a credential sealed to a key it no longer has")
+	}
+	// And the provider's copy changed too, in the same breath. Otherwise the two ends hold
+	// different passwords — which is the fanout window that makes rotation dangerous elsewhere.
+	if after.ForProvider == before.ForProvider {
+		t.Fatal("only one end was rotated, so the two now disagree")
+	}
+}
+
+func TestRotatingReachesBothEnds(t *testing.T) {
+	inv, ctx := twoNodesWithKeys(t)
+	before, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := inv.RotateSecret(ctx, "database", "consumer", "provider"); err != nil {
+		t.Fatal(err)
+	}
+	after, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if after.ForConsumer == before.ForConsumer || after.ForProvider == before.ForProvider {
+		t.Fatal("rotation left one of the ends holding what it had")
+	}
+}
+
+func TestAProviderIsToldEveryCredentialItMustCreate(t *testing.T) {
+	// The half that makes a credential real. A password nothing was told to create authenticates
+	// nowhere, and the mesh cannot tell the provider what it is in any other way — it cannot read
+	// it either.
+	inv, ctx := twoNodesWithKeys(t)
+	// The provider's own key, kept, so this asserts it can *open* what it was handed rather than
+	// that the field is non-empty. Without that, selecting the wrong column reads the same both
+	// ways and the test proves nothing — which it did, until the check was removed and it passed.
+	providerKey, openProvider := aSealingKey(t)
+	provider, err := inv.NodeByName(ctx, "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := inv.RecordSealingKey(ctx, provider.ID, providerKey); err != nil {
+		t.Fatal(err)
+	}
+
+	other, err := inv.AddNode(ctx, "second-consumer")
+	if err != nil {
+		t.Fatal(err)
+	}
+	secondKey, _ := aSealingKey(t)
+	if err := inv.RecordSealingKey(ctx, other.ID, secondKey); err != nil {
+		t.Fatal(err)
+	}
+	for _, who := range []string{"consumer", "second-consumer"} {
+		if _, err := inv.SecretFor(ctx, "database", who, "provider"); err != nil {
+			t.Fatal(err)
+		}
+	}
+	issued, err := inv.SecretsFrom(ctx, "provider")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(issued) != 2 {
+		t.Fatalf("the provider was told about %d of 2", len(issued))
+	}
+	for _, s := range issued {
+		if _, ok := openProvider(s.ForProvider); !ok {
+			t.Fatalf("the provider cannot open the credential it was given for %s", s.Consumer)
+		}
+		if s.ForConsumer != "" {
+			// It has no business holding the other end's copy, and handing it out would put a
+			// second readable-by-someone-else copy into circulation.
+			t.Fatalf("the provider was handed the consumer's own copy of %s", s.Name)
+		}
+	}
+}
+
+func TestANodeWithNoSealingKeyCannotBeGivenASecret(t *testing.T) {
+	inv := fresh(t)
+	ctx := context.Background()
+	for _, n := range []string{"consumer", "provider"} {
+		if _, err := inv.AddNode(ctx, n); err != nil {
+			t.Fatal(err)
+		}
+	}
+	_, err := inv.SecretFor(ctx, "database", "consumer", "provider")
+	if err == nil {
+		t.Fatal("a credential was made for nodes that cannot open one")
+	}
+	if !strings.Contains(err.Error(), "sealing key") {
+		t.Fatalf("the refusal does not say what is missing: %v", err)
+	}
+}
+
+func TestSecretsGoWhenANodeLeaves(t *testing.T) {
+	inv, ctx := twoNodesWithKeys(t)
+	if _, err := inv.SecretFor(ctx, "database", "consumer", "provider"); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := inv.store.Pool().Exec(ctx, `delete from node where name = 'consumer'`); err != nil {
+		t.Fatal(err)
+	}
+	var left int
+	if err := inv.store.Pool().QueryRow(ctx, `select count(*) from secret`).Scan(&left); err != nil {
+		t.Fatal(err)
+	}
+	if left != 0 {
+		t.Fatalf("%d credential(s) outlived the machine they were for", left)
+	}
+}