Where the answer to a requirement is allowed to live

Two different things were both written `requires`. A shell, a display
server and a private network have to be on the machine that needs them.
A database does not — it runs somewhere and is reached over the network.
Both were answered the same way, so requiring a database installed
PostgreSQL on every machine that ran a web application.

What a module provides now carries a scope, the same idea claims already
use, written short in the ordinary case:

  "provides": ["shell"]
  "provides": [{"name": "database", "scope": "mesh"}]

A mesh-scoped requirement is answered by finding the node already running
it — never by installing it here. Choosing a machine to put a database on
is a decision with consequences, and nothing resolving a web application
should make it silently. With nothing anywhere it refuses and says which
module to assign; with two it refuses and says how to choose.

Choosing is `pin <node> <provision> <from>`, kept per node because that
is the granularity the choice has. A pin at a machine that does not
provide it refuses rather than falling back — a fallback would quietly
move somebody's data. One provider does not overrule a pin either.

Resolving a node now needs to know what the others offer, and working
that out needs them resolved, so it is two passes: the first answers only
what each node offers, the second answers everything. Nothing is ever
declared from the first.

A node's plan says what it takes from elsewhere. It is the only part of a
set that stops working when a different machine goes away, and nothing
else in that output would have said so. It is also where a credential
will hang once there is a mechanism for handing one back.

One test found passing for the wrong reason: it read pins through a join
on the provider, which hides a dangling row whether or not it was cleaned
up. It counts rows now, and bites when the cascade is removed.

internal/inventory/catalogue.go

@@ -436,3 +436,82 @@ func (i *Inventory) SettingsFor(ctx context.Context, nodeName, module string) ([
 	}
 	return layers, rows.Err()
 }
+
+// PinProvision records which node a machine gets a provision from.
+//
+// Only needed when more than one node could answer. Recordable before that, because a mesh with
+// one database should not change where an existing machine gets its data the day a second
+// arrives.
+func (i *Inventory) PinProvision(ctx context.Context, nodeName, provision, provider string) error {
+	node, err := i.NodeByName(ctx, nodeName)
+	if err != nil {
+		return err
+	}
+	from, err := i.NodeByName(ctx, provider)
+	if err != nil {
+		return err
+	}
+	_, err = i.store.Pool().Exec(ctx,
+		`insert into provision_pin (node, name, provider) values ($1, $2, $3)
+		 on conflict (node, name) do update set provider = excluded.provider, pinned_at = now()`,
+		node.ID, provision, from.ID)
+	return err
+}
+
+// UnpinProvision removes a choice, putting the question back.
+func (i *Inventory) UnpinProvision(ctx context.Context, nodeName, provision string) error {
+	node, err := i.NodeByName(ctx, nodeName)
+	if err != nil {
+		return err
+	}
+	tag, err := i.store.Pool().Exec(ctx,
+		`delete from provision_pin where node = $1 and name = $2`, node.ID, provision)
+	if err != nil {
+		return err
+	}
+	if tag.RowsAffected() == 0 {
+		return fmt.Errorf("%s was not told where to get %q from", nodeName, provision)
+	}
+	return nil
+}
+
+// PinsFor is what a node was told about where its provisions come from.
+func (i *Inventory) PinsFor(ctx context.Context, nodeName string) (map[string]string, error) {
+	node, err := i.NodeByName(ctx, nodeName)
+	if err != nil {
+		return nil, err
+	}
+	rows, err := i.store.Pool().Query(ctx,
+		`select p.name, n.name from provision_pin p join node n on n.id = p.provider
+		 where p.node = $1`, node.ID)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+
+	out := map[string]string{}
+	for rows.Next() {
+		var name, provider string
+		if err := rows.Scan(&name, &provider); err != nil {
+			return nil, err
+		}
+		out[name] = provider
+	}
+	return out, rows.Err()
+}
+
+// pinRows is how many pins a node holds, counted in the table rather than through the join that
+// reads them.
+//
+// For a test that would otherwise pass for the wrong reason: PinsFor joins on the provider, so a
+// pin left behind by a departed node is invisible through it whether it was cleaned up or not.
+func (i *Inventory) pinRows(ctx context.Context, nodeName string) (int, error) {
+	node, err := i.NodeByName(ctx, nodeName)
+	if err != nil {
+		return 0, err
+	}
+	var n int
+	err = i.store.Pool().QueryRow(ctx,
+		`select count(*) from provision_pin where node = $1`, node.ID).Scan(&n)
+	return n, err
+}