Rotate a credential and move both ends together

The invariant novox/hq ADR 0001 records as unowned, and it was measurably
false in HAL: a provision documented as never rotating minted a new password on
every adoption and updated only the provider's row. Consumers on three nodes
held dead credentials for two days while the mesh reported success. Nothing
enumerated who held the old one.

Three things make that impossible here. The holders are a set the mesh can name
— each pair has its own credential, so rotating one consumer touches one role
and the affected list is a query rather than an assumption. Both ends are
pushed by this command rather than a later one, because leaving the sending to
whoever remembered is the fault exactly. And it is all-or-nothing: if any
affected machine cannot be resolved, nothing is sent and the old credential
keeps working, which is a mesh that has not rotated rather than one that has
half-rotated.

The window is stated rather than hidden: a role's password changes on the
provider and the file changes on the consumer, and they cannot be simultaneous.

The provisioner now takes its superuser password from the file the mesh wrote,
which is how the mesh delivers one. Passing it through the environment needed a
person in the middle of the one path that exists so there is not one — and put
a superuser password where `docker inspect` prints it.

examples/postgres-provisioner/where_test.go

@@ -0,0 +1,69 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// The password comes from a file, because that is how the mesh delivers one.
+//
+// A provisioner told to take a superuser password from an environment variable needs somebody to
+// read the sealed file and pass it in — a person in the middle of the one path that exists so
+// there is not one. It is also the difference between a credential in a file and one in a process
+// listing: `docker inspect` prints environment.
+func TestTheSuperuserPasswordComesFromTheFileTheMeshWrote(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "superuser")
+	if err := os.WriteFile(path, []byte("the-sealed-one\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("MESH_PROVISION_POSTGRES", "postgres://postgres@127.0.0.1:5433/postgres?sslmode=disable")
+	t.Setenv("MESH_PROVISION_PASSWORD_FILE", path)
+
+	where, err := connectionString()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(where, "the-sealed-one") {
+		t.Fatalf("the password the mesh wrote is not in the connection: %s", where)
+	}
+	if !strings.Contains(where, "127.0.0.1:5433") || !strings.Contains(where, "sslmode=disable") {
+		t.Fatalf("the rest of the connection was lost: %s", where)
+	}
+}
+
+// An empty file connects as nobody and is refused by the database three layers away, as an
+// authentication problem with no cause anybody changed.
+func TestAnEmptyPasswordFileIsRefusedHere(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "superuser")
+	if err := os.WriteFile(path, []byte("\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("MESH_PROVISION_POSTGRES", "postgres://postgres@127.0.0.1:5433/postgres")
+	t.Setenv("MESH_PROVISION_PASSWORD_FILE", path)
+	if _, err := connectionString(); err == nil {
+		t.Fatal("a provisioner with no password reported one")
+	}
+}
+
+// And a provisioner somebody runs by hand still works with the URL alone.
+func TestAConnectionWithNoPasswordFileIsLeftAlone(t *testing.T) {
+	t.Setenv("MESH_PROVISION_POSTGRES", "postgres://postgres:typed@127.0.0.1:5433/postgres")
+	t.Setenv("MESH_PROVISION_PASSWORD_FILE", "")
+	where, err := connectionString()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if where != "postgres://postgres:typed@127.0.0.1:5433/postgres" {
+		t.Fatalf("the connection was rewritten when it should have been left alone: %s", where)
+	}
+}
+
+func TestAProvisionerWithNoDatabaseSaysSo(t *testing.T) {
+	t.Setenv("MESH_PROVISION_POSTGRES", "")
+	t.Setenv("MESH_PROVISION_PASSWORD_FILE", "")
+	if _, err := connectionString(); err == nil {
+		t.Fatal("a provisioner that does not know which database it owns reported one")
+	}
+}