A node is known by a key it generated

The thing I had been calling blocked for weeks, built in an afternoon once it
was pointed out that it was already decided. 08-connectivity says of the
overlay keys: each node generates its own keypair, the private half never
leaves the machine, the public half is published -- and says outright this IS
ADR 0004's "a node holds its own identity". Nobody had applied it to node
identity itself.

identity now holds the public half of each node's key. Only the public half,
which is the property worth having: a copy of this database is a list of who to
believe, not a set of credentials, so compromise of a node really is compromise
of only that node.

Exactly one key is live per node, and re-enrolment revokes the one it replaced
in the same transaction -- two live identities is the stolen-laptop case with
the replaced machine still believed.

Fault injection was worth the time here. Three findings. The unique index was
defended by no test at all: sequential enrolment is already safe because the
code revokes before inserting, so removing the constraint changed nothing. The
constraint only matters when two enrolments race, and there is now a test that
runs six at once and fails without it.

My injection harness also lied to me. One injection matched nothing, changed no
file, and reported NO BITE identically to a real one -- so a test that defends
nothing and an injection that does nothing look the same. The harness now
checksums the files and says NO-OP when they did not change.

And one honest NO BITE left standing: making the key lookup return a zero key
for an unknown node does not fail the test, because the signature check refuses
it a line later. Two independent mechanisms, not a placebo.

61 tests, none skipped.

README.md

@@ -26,7 +26,7 @@ argument that is not settled there.
 | | |
 |---|---|
 | `inventory` | node records and enrolment tokens — **built, as far as identity** |
-| `identity` | the control plane's own signing key — **built, and no further** |
+| `identity` | the control plane's signing key, and the keys nodes are known by — **built** |
 | `config`, `connectivity`, `provisioning`, `delivery`, `observability` | not built |
 | the interface every surface speaks to | not built; its shape is not decided |
 
@@ -87,16 +87,24 @@ for one. Run `migrate` with only one and it stops, naming the grant it does not
 A token needs a node record from one and a signing key from the other. Neither reads the other's
 store — the process holding both grants asks each for its part.
 
-### Where this stops, and why there
+### How a node is known
 
-At **identity**. A node's own identity is the next thing needed and its cryptographic form is not
+**The node generates a keypair; the mesh records the public half.** The same principle as SSH, and
-decided anywhere: whether a node holds a keypair whose public half the mesh keeps, or something
+the same rule `08-connectivity` already applies to the overlay keys — which is where this was
-else. Modelling it would have meant guessing, in a migration — which is the most expensive place
+settled all along, though it took being asked directly to notice.
-in this system to guess, because a schema that ran is finished and the only way back is another
-migration.
 
-So the node table holds what a node record *is* — a name, when it was made, what the machine last
+Only the public half is ever stored, and that is the property worth having: **a copy of this
-reported about itself, when it was last heard from — and stops before what a node *presents*.
+database grants nothing.** It is a list of who to believe, not a set of credentials, which is what
+makes *compromise of a node is compromise of that node* literally true.
+
+Exactly one key is live per node. Re-enrolment revokes the one it replaced, in the same
+transaction — two live identities for one node record is the stolen-laptop case, with the replaced
+machine still believed. The database enforces it as well as the code, and there is a test running
+six concurrent enrolments that fails when the constraint is removed.
+
+**Not the machine's SSH host key**, though that was the obvious economy. Host keys are regenerated
+by reinstalls and image clones, which would silently un-enrol a node; their lifecycle belongs to
+sshd rather than the mesh; and a partial host has no SSH daemon at all.
 
 ## Reaching a store
 

internal/identity/identity.go

@@ -150,3 +150,90 @@ func (i *Identity) Sign(ctx context.Context, message []byte) ([]byte, error) {
 func Verify(public ed25519.PublicKey, message, signature []byte) bool {
 	return ed25519.Verify(public, message, signature)
 }
+
+// NodeKey is the public half of a node's own keypair, as the mesh holds it.
+type NodeKey struct {
+	ID     string
+	Node   string
+	Public ed25519.PublicKey
+	Issued time.Time
+}
+
+// ErrNotThisNode is what verification returns when a key is not the live one for a node.
+//
+// One error whether the key is unknown, revoked, or belongs to a different node. Whoever is
+// presenting a key that does not work is either a machine whose operator can be told out of band,
+// or something probing, and the second must not learn which.
+var ErrNotThisNode = errors.New("that key does not identify that node")
+
+// RecordNodeKey writes down the public key the mesh will believe for a node.
+//
+// Any previous key for the node is revoked in the same transaction. Two live identities for one
+// node record is novox/hq ADR 0004's stolen-laptop case — the machine that was replaced going on
+// being believed — and the window between two statements is exactly when it would exist.
+func (i *Identity) RecordNodeKey(ctx context.Context, node string, public ed25519.PublicKey) (NodeKey, error) {
+	if len(public) != ed25519.PublicKeySize {
+		return NodeKey{}, fmt.Errorf(
+			"a node key is %d bytes and this is %d: a node presents an Ed25519 public key",
+			ed25519.PublicKeySize, len(public))
+	}
+
+	tx, err := i.store.Pool().Begin(ctx)
+	if err != nil {
+		return NodeKey{}, err
+	}
+	defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
+
+	if _, err := tx.Exec(ctx,
+		`update node_key set revoked = now() where node = $1 and revoked is null`, node); err != nil {
+		return NodeKey{}, err
+	}
+
+	var k NodeKey
+	var stored []byte
+	err = tx.QueryRow(ctx,
+		`insert into node_key (node, public) values ($1, $2) returning id, node, public, issued`,
+		node, []byte(public)).Scan(&k.ID, &k.Node, &stored, &k.Issued)
+	if err != nil {
+		return NodeKey{}, err
+	}
+	k.Public = stored
+
+	if err := tx.Commit(ctx); err != nil {
+		return NodeKey{}, err
+	}
+	return k, nil
+}
+
+// LiveKey is the key currently identifying a node.
+func (i *Identity) LiveKey(ctx context.Context, node string) (NodeKey, error) {
+	var k NodeKey
+	var public []byte
+	err := i.store.Pool().QueryRow(ctx,
+		`select id, node, public, issued from node_key where node = $1 and revoked is null`,
+		node).Scan(&k.ID, &k.Node, &public, &k.Issued)
+	if errors.Is(err, pgx.ErrNoRows) {
+		return NodeKey{}, ErrNotThisNode
+	}
+	if err != nil {
+		return NodeKey{}, err
+	}
+	k.Public = public
+	return k, nil
+}
+
+// VerifyNode checks that something signed a challenge with the live key for a node.
+//
+// This is the whole of proving a node is that node, and it is the same operation the node performs
+// in the other direction on every declaration it receives. Nothing here is stored that could be
+// replayed: the mesh holds a public key, so a copy of this database proves nothing to anybody.
+func (i *Identity) VerifyNode(ctx context.Context, node string, challenge, signature []byte) error {
+	key, err := i.LiveKey(ctx, node)
+	if err != nil {
+		return err
+	}
+	if !ed25519.Verify(key.Public, challenge, signature) {
+		return ErrNotThisNode
+	}
+	return nil
+}

internal/identity/identity_test.go

@@ -1,7 +1,9 @@
 package identity
 
 import (
+	"bytes"
 	"context"
+	"crypto/ed25519"
 	"errors"
 	"fmt"
 	"os"
@@ -203,3 +205,213 @@ func TestTheFingerprintIsOfThePublicHalf(t *testing.T) {
 		t.Error("the fingerprint does not depend on the key")
 	}
 }
+
+func TestANodeIsVerifiedByAKeyItGenerated(t *testing.T) {
+	// The whole of proving a node is that node. The mesh holds only the public half, so this
+	// verification is the same operation the node performs on every declaration, in reverse.
+	ident := fresh(t)
+	public, private, err := ed25519.GenerateKey(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	node := uuid(t)
+
+	if _, err := ident.RecordNodeKey(t.Context(), node, public); err != nil {
+		t.Fatal(err)
+	}
+	challenge := []byte("prove you are that node")
+	if err := ident.VerifyNode(t.Context(), node, challenge, ed25519.Sign(private, challenge)); err != nil {
+		t.Fatalf("a node signing with its own key was refused: %v", err)
+	}
+}
+
+func TestAnotherMachinesKeyDoesNotIdentifyThisNode(t *testing.T) {
+	ident := fresh(t)
+	mine, _, err := ed25519.GenerateKey(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	_, theirPrivate, err := ed25519.GenerateKey(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	node := uuid(t)
+	if _, err := ident.RecordNodeKey(t.Context(), node, mine); err != nil {
+		t.Fatal(err)
+	}
+
+	challenge := []byte("prove you are that node")
+	err = ident.VerifyNode(t.Context(), node, challenge, ed25519.Sign(theirPrivate, challenge))
+	if !errors.Is(err, ErrNotThisNode) {
+		t.Fatalf("a different machine's signature was accepted: %v", err)
+	}
+}
+
+func TestReEnrolmentRevokesTheMachineItReplaced(t *testing.T) {
+	// novox/hq ADR 0004's stolen-laptop case. Two live identities for one node record means the
+	// machine that was replaced goes on being believed, which is the thing revocation exists for.
+	ident := fresh(t)
+	node := uuid(t)
+	oldPublic, oldPrivate, err := ed25519.GenerateKey(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := ident.RecordNodeKey(t.Context(), node, oldPublic); err != nil {
+		t.Fatal(err)
+	}
+
+	newPublic, newPrivate, err := ed25519.GenerateKey(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := ident.RecordNodeKey(t.Context(), node, newPublic); err != nil {
+		t.Fatal(err)
+	}
+
+	challenge := []byte("still me?")
+	if err := ident.VerifyNode(t.Context(), node, challenge, ed25519.Sign(oldPrivate, challenge)); !errors.Is(err, ErrNotThisNode) {
+		t.Error("the replaced machine is still believed")
+	}
+	if err := ident.VerifyNode(t.Context(), node, challenge, ed25519.Sign(newPrivate, challenge)); err != nil {
+		t.Errorf("the new machine was refused: %v", err)
+	}
+}
+
+func TestOnlyOneKeyIsEverLiveForANode(t *testing.T) {
+	// Enforced by the database rather than by the order of two statements, because the window
+	// between them is exactly when two live identities would exist.
+	ident := fresh(t)
+	node := uuid(t)
+	for i := 0; i < 4; i++ {
+		public, _, err := ed25519.GenerateKey(nil)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if _, err := ident.RecordNodeKey(t.Context(), node, public); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	var live int
+	if err := ident.store.Pool().QueryRow(t.Context(),
+		`select count(*) from node_key where node = $1 and revoked is null`, node).Scan(&live); err != nil {
+		t.Fatal(err)
+	}
+	if live != 1 {
+		t.Errorf("%d live keys for one node; exactly one may be", live)
+	}
+}
+
+func TestOnlyThePublicHalfIsEverStored(t *testing.T) {
+	// The property that makes a copy of this database worthless to whoever takes it: it is a list
+	// of who to believe, not a set of credentials.
+	ident := fresh(t)
+	public, private, err := ed25519.GenerateKey(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	node := uuid(t)
+	if _, err := ident.RecordNodeKey(t.Context(), node, public); err != nil {
+		t.Fatal(err)
+	}
+
+	var stored []byte
+	if err := ident.store.Pool().QueryRow(t.Context(),
+		`select public from node_key where node = $1`, node).Scan(&stored); err != nil {
+		t.Fatal(err)
+	}
+	if len(stored) != ed25519.PublicKeySize {
+		t.Errorf("stored %d bytes for a node key; a public key is %d and a private key is %d",
+			len(stored), ed25519.PublicKeySize, ed25519.PrivateKeySize)
+	}
+	if bytes.Contains(private, stored) && bytes.Contains(stored, private[:32]) {
+		t.Error("what is stored looks like part of the private key")
+	}
+}
+
+func TestAnUnknownNodeAndARevokedKeyAreRefusedAlike(t *testing.T) {
+	ident := fresh(t)
+	_, private, err := ed25519.GenerateKey(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	challenge := []byte("hello")
+	err = ident.VerifyNode(t.Context(), uuid(t), challenge, ed25519.Sign(private, challenge))
+	if !errors.Is(err, ErrNotThisNode) {
+		t.Fatalf("an unknown node gave %v", err)
+	}
+}
+
+func TestSomethingThatIsNotAKeyIsRefused(t *testing.T) {
+	ident := fresh(t)
+	if _, err := ident.RecordNodeKey(t.Context(), uuid(t), []byte("far too short")); err == nil {
+		t.Fatal("a truncated key was recorded as a node's identity")
+	}
+}
+
+// uuid gives each test its own node id. The node records live in another context's database
+// (novox/hq ADR 0008), so there is nothing here to reference and nothing to create.
+func uuid(t *testing.T) string {
+	t.Helper()
+	var id string
+	if err := freshConn(t).QueryRow(t.Context(), `select gen_random_uuid()`).Scan(&id); err != nil {
+		t.Fatal(err)
+	}
+	return id
+}
+
+func freshConn(t *testing.T) *pgx.Conn {
+	t.Helper()
+	conn, err := pgx.Connect(t.Context(), os.Getenv("MESH_TEST_POSTGRES"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { conn.Close(context.Background()) })
+	return conn
+}
+
+func TestTwoEnrolmentsAtOnceStillLeaveOneLiveKey(t *testing.T) {
+	// The case the database constraint is for, and the only one: sequential calls are already
+	// safe because RecordNodeKey revokes before it inserts. Two at once both revoke what they
+	// found and both insert, and without the partial unique index the node ends with two live
+	// identities — the replaced machine still believed, which is the whole thing revocation
+	// exists to prevent.
+	//
+	// Some of these are expected to fail. What must not happen is two of them succeeding.
+	ident := fresh(t)
+	node := uuid(t)
+
+	var wg sync.WaitGroup
+	errs := make([]error, 6)
+	for i := range errs {
+		public, _, err := ed25519.GenerateKey(nil)
+		if err != nil {
+			t.Fatal(err)
+		}
+		wg.Add(1)
+		go func(i int, public ed25519.PublicKey) {
+			defer wg.Done()
+			_, errs[i] = ident.RecordNodeKey(context.Background(), node, public)
+		}(i, public)
+	}
+	wg.Wait()
+
+	var live int
+	if err := ident.store.Pool().QueryRow(t.Context(),
+		`select count(*) from node_key where node = $1 and revoked is null`, node).Scan(&live); err != nil {
+		t.Fatal(err)
+	}
+	if live != 1 {
+		t.Errorf("%d live keys after six concurrent enrolments; exactly one may be live", live)
+	}
+
+	succeeded := 0
+	for _, err := range errs {
+		if err == nil {
+			succeeded++
+		}
+	}
+	if succeeded == 0 {
+		t.Error("every concurrent enrolment failed; at least one must win")
+	}
+}

internal/identity/migrations/0002-node-keys.sql

@@ -0,0 +1,38 @@
+-- What a node presents to prove it is that node.
+--
+-- novox/hq ADR 0004: the node generates a keypair, the private half never leaves the machine, and
+-- the mesh records the public half. The same rule 08-connectivity already applies to the overlay
+-- keys, applied to the thing 0004 is about.
+--
+-- Only the public half is here, and that is the property worth having: a copy of this database
+-- grants nothing. It is a list of who to believe, not a set of credentials -- which is what makes
+-- "compromise of a node is compromise of that node" literally true.
+
+create table node_key (
+    id      uuid        primary key default gen_random_uuid(),
+
+    -- The node record this key speaks for. Held as an id rather than a foreign key: the node
+    -- records live in `inventory`, which is a different context and a different database
+    -- (novox/hq ADR 0008). There is deliberately no join to be had -- the process holding both
+    -- grants asks each for its part.
+    node    uuid        not null,
+
+    public  bytea       not null check (octet_length(public) = 32),
+
+    issued  timestamptz not null default now(),
+
+    -- Issuing a re-enrolment token revokes the previous identity for that node, and that is not
+    -- housekeeping: two live identities for one node record is the stolen-laptop case with the
+    -- thief's credentials still valid.
+    revoked timestamptz
+);
+
+-- One live key per node. The constraint is what makes revocation mean something -- without it a
+-- second enrolment would add a key rather than replace one, and the old machine would go on being
+-- believed.
+create unique index node_key_one_live on node_key (node) where revoked is null;
+
+-- Redemption looks a node up by the key it presented, so that is the other direction. Not unique:
+-- a revoked key stays, and a machine re-enrolling after a rebuild may legitimately present the
+-- same one it had before.
+create index node_key_public on node_key (public);