A module may need a secret of its own, and the provisioner watches

Two things, both found by trying to write a real postgres module and
discovering it could not be said.

A database has a superuser password, a broker an administrator, a
registry an account. None of them is *for* anybody — they are not the
credential a consumer is given, and the mechanism that hands those out
has a consumer in the middle of it. So a module may declare what it needs
and where to put it, and the mesh generates one per node, seals it, and
reads it no more than it reads any other.

Per node, deliberately: a module running on three machines has three
passwords. One in the manifest instead would put the same secret on every
machine that ever runs it, in a file anybody can read, for ever. Made
once and kept, or a running database would be handed a password it was
not started with; remade when the machine's sealing key changes, like
everything else sealed here.

A need declared and not made is refused rather than skipped, because a
module whose own credential is silently absent starts, fails to
authenticate, and the reason is three layers from the machine reporting
it.

And the provisioner can watch. That is what lets it be a module rather
than a binary somebody places: run once, it needs invoking after every
declaration by a timer or a unit wired to a file; watching, it is an
ordinary long-running service the host already supervises. It polls
rather than watching the filesystem, because the host writes atomically —
the file is replaced, so a watch on the path stops seeing anything after
the first replacement, and a watcher that silently stops working is worse
than a poll. Credentials are compared by digest and never held: this runs
for as long as the machine is up.

examples/README.md

@@ -10,3 +10,16 @@ with the module that ships the software it configures.
 | | |
 |---|---|
 | `postgres-provisioner` | the last step of a credential: reads what the mesh delivered and makes PostgreSQL accept it |
+
+## Running the provisioner
+
+`--watch` reconciles now and again whenever what the mesh delivered changes. That is what lets it
+be a module: an ordinary long-running service the host supervises, rather than something that has
+to be invoked after every declaration by a timer or a unit wired to a file.
+
+It polls rather than watching the filesystem, because the host writes atomically — the file is
+replaced, so a watch on the path stops seeing anything after the first replacement. A watcher that
+silently stops working is worse than a poll.
+
+Credentials are compared by digest and never by content. This runs for as long as the machine is
+up, and a secret does not belong in a long-lived variable when a hash answers the same question.

examples/postgres-provisioner/Dockerfile

@@ -0,0 +1,18 @@
+# The provisioner, as a module ships one.
+#
+# Built here so a machine can be given it by the mesh rather than by somebody putting a binary on
+# it. Static and FROM scratch for the same reason the control plane's image is: it is fetched by
+# digest and run on a machine, and everything in it is something a person would have to audit.
+FROM golang:1.25-alpine AS build
+WORKDIR /src
+COPY go.mod go.sum ./
+RUN go mod download
+COPY . .
+RUN CGO_ENABLED=0 go build -trimpath -ldflags '-s -w' \
+    -o /mesh-provision-postgres ./examples/postgres-provisioner
+
+FROM scratch
+COPY --from=build /mesh-provision-postgres /mesh-provision-postgres
+# Watching by default, because that is what makes it a module: an ordinary long-running service
+# the host supervises, rather than something invoked after every declaration.
+ENTRYPOINT ["/mesh-provision-postgres", "--watch"]

examples/postgres-provisioner/main.go

@@ -20,12 +20,17 @@ package main
 
 import (
 	"context"
+	"crypto/sha256"
+	"encoding/hex"
 	"encoding/json"
 	"fmt"
 	"os"
+	"os/signal"
 	"path/filepath"
 	"sort"
 	"strings"
+	"syscall"
+	"time"
 
 	"github.com/jackc/pgx/v5"
 )
@@ -53,12 +58,99 @@ type manifest struct {
 }
 
 func main() {
-	if err := run(context.Background()); err != nil {
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+
+	// Once, or whenever what it was given changes.
+	//
+	// **Watching is what lets this be a module.** Run once, it has to be invoked by something
+	// after every declaration — a timer that runs it when nothing changed, or a unit wired to
+	// restart on a file. Watching, it is an ordinary long-running service, which is a shape the
+	// mesh already delivers and the host already supervises.
+	//
+	// The file it watches is the manifest the mesh writes. A credential changing rewrites the
+	// file beside it and not the manifest, so the manifest is stamped whenever either is written
+	// — which is why this compares content rather than modification time.
+	if len(os.Args) > 1 && os.Args[1] == "--watch" {
+		if err := watch(ctx); err != nil {
+			fmt.Fprintf(os.Stderr, "mesh-provision-postgres: %v\n", err)
+			os.Exit(1)
+		}
+		return
+	}
+	if err := run(ctx); err != nil {
 		fmt.Fprintf(os.Stderr, "mesh-provision-postgres: %v\n", err)
 		os.Exit(1)
 	}
 }
 
+// watch reconciles now, and again whenever what the mesh delivered changes.
+//
+// By polling rather than by watching the filesystem, because the file is replaced rather than
+// written in place — the host writes atomically, so an inotify watch on the path stops seeing
+// anything after the first replacement, which is a watcher that silently stops working.
+func watch(ctx context.Context) error {
+	const every = 10 * time.Second
+	var last string
+
+	for {
+		state, err := given()
+		switch {
+		case err != nil:
+			// Said and retried. A provisioner that exits because the mesh has not written
+			// anything yet is one that has to be restarted by hand after the first push.
+			fmt.Fprintf(os.Stderr, "cannot read what was granted: %v\n", err)
+		case state != last:
+			if err := run(ctx); err != nil {
+				// Reported and retried. The usual reason is that the database has not finished
+				// starting, and giving up would mean a module that works only if the two
+				// containers happen to come up in the right order.
+				fmt.Fprintf(os.Stderr, "%v\n", err)
+			} else {
+				last = state
+			}
+		}
+
+		select {
+		case <-ctx.Done():
+			return nil
+		case <-time.After(every):
+		}
+	}
+}
+
+// given is everything the mesh has delivered, as one string, so a change of any of it is one
+// comparison.
+//
+// The credentials are included by their **digest**, never their content: this is compared, logged
+// on nothing, and held in memory for as long as the process runs, and a secret does not belong in
+// any of that when a hash answers the same question.
+func given() (string, error) {
+	grants := os.Getenv("GRANTS")
+	if grants == "" {
+		grants = "/var/lib/postgres/grants"
+	}
+	entries, err := os.ReadDir(grants)
+	if err != nil {
+		return "", err
+	}
+	var names []string
+	for _, e := range entries {
+		names = append(names, e.Name())
+	}
+	sort.Strings(names)
+
+	sum := sha256.New()
+	for _, name := range names {
+		body, err := os.ReadFile(filepath.Join(grants, name))
+		if err != nil {
+			return "", err
+		}
+		fmt.Fprintf(sum, "%s:%x\n", name, sha256.Sum256(body))
+	}
+	return hex.EncodeToString(sum.Sum(nil)), nil
+}
+
 func run(ctx context.Context) error {
 	grants := os.Getenv("GRANTS")
 	if grants == "" {