Raise genesis on the bus the mesh runs on, from one bundle (hq issue 146, ADR 0331)

The installer could not raise a first node: the old bundle's control plane had no
MESH_BUS_NATS and crash-looped, and the NATS bundle was refused because rootsecrets
only knew the previous broker's guest:guest@ template.

One bundle now, the NATS one. rootsecrets makes the controller's bus account and puts
it in both placeholders (the address and the bcrypt hash in the carried user list),
kept with its hash so a re-run writes the same bundle. Preflight refuses a template
naming no bus connection it knows, naming the placeholder, instead of waiting out the
verify step. The broker step adopts the bus as the nats module and refuses, by name,
a catalogue without it or a server that is not the foundation's container.

internal/bootstrap/rootsecrets.go

@@ -4,49 +4,58 @@ import (
 	"bytes"
 	"context"
 	"crypto/rand"
-	"crypto/sha256"
 	"encoding/base64"
-	"encoding/hex"
 	"fmt"
 	"os"
 	"path/filepath"
 	"strings"
 
+	"golang.org/x/crypto/bcrypt"
+
 	"github.com/novox/mesh-host/internal/declaration"
 )
 
 // The mesh's root credentials, made at genesis rather than copied from the template.
 //
-// **The template carries `bootstrap` and `guest`, and a mesh raised from it kept them** (novox/hq
-// issue 071). The store's superuser and the broker's administrator are the two credentials every
-// other one rests on, and they were the two that were not secret: constants in a file anybody can
-// read, carried into the mesh by `secret accept` and marked as something the mesh must never
-// replace — which is correct for a credential that already created the databases, and made the
-// well-known value permanent.
+// **The template carries `bootstrap`, and a mesh raised from it kept it** (novox/hq issue 071). The
+// store's superuser and the controller's account on the bus are the two credentials every other one
+// rests on, and they were the two that were not secret: constants in a file anybody can read,
+// carried into the mesh by `secret accept` and marked as something the mesh must never replace —
+// which is correct for a credential that already created the databases, and made the well-known
+// value permanent.
 //
-// So the installer makes them. Two random values, **made once and kept on this machine** at the
-// paths the postgres and lavinmq modules declare as their own secrets — so that when phase three
-// adopts the store and the broker, `secret accept` carries in exactly the value the servers were
-// raised with, and the host's later write of the sealed secret lands the same bytes in the same
-// file. A second run finds the files and changes nothing, which is what lets the installer say
-// "already done" about a store it must not restart.
+// So the installer makes them. Two random values, **made once and kept on this machine** — the
+// store's at the path the postgres module declares as its own secret, so that when phase three
+// adopts the store `secret accept` carries in exactly the value the server was raised with; the
+// bus's beside the other things only genesis holds. A second run finds the files and changes
+// nothing, which is what lets the installer say "already done" about a store it must not restart.
 //
 // **The store reads its password from a file, not its environment.** `POSTGRES_PASSWORD` in a
 // container's environment is in `docker inspect` for ever; the module that adopts the store
 // declares the same file mount, so the two specs are one and the applier reconciles rather than
-// recreates (phase3.go). The broker has no such file: its image's default administrator is changed
-// in place by an action once the broker answers, and the produced bundle carries that action.
+// recreates (phase3.go).
+//
+// **The bus has no administrator to change; it has a user list** (novox/hq ADR 0331 §3). The
+// bundle carries the first list — the controller's own account at `bootstrap` — and the
+// controller's address names the same account. Both are the bundle's placeholders: the address's
+// `nats://controller:bootstrap@` and the controller's entry in the carried list. The installer
+// puts the made password in the first and its bcrypt hash in the second, so the bus is raised with
+// the account the controller dials with. The controller records a hash of what it actually uses
+// (mesh-controller internal/broker, CredentialIn), so its own compositions carry the same account
+// onward, and the permanent controller is given it by `secret accept` (control.go deliverStores).
 const (
 	// StoreSuperuserFile is where the store's superuser password lives on the machine — the
 	// postgres module's own-secret path, so genesis and adoption write the same file.
 	StoreSuperuserFile = "/var/lib/postgres/superuser.secret"
-	// BrokerAdminFile is the same for the broker's administrator — the lavinmq module's.
-	BrokerAdminFile = "/var/lib/lavinmq-module/admin.secret"
-	// BrokerAdminUser is the broker's administrator. The image's default account, kept by name
-	// and given a password that is not the image's default; a renamed account would have to be
-	// created before anything can authenticate, and the thing that creates accounts is the thing
-	// that has to authenticate first.
-	BrokerAdminUser = "guest"
+	// BusControllerFile is where the controller's password on the bus lives on the machine. Not a
+	// module's path: no module reads it — the permanent controller is handed the address the
+	// temporary one was given — and genesis keeps it only so a second run puts the same account
+	// in the same two places.
+	BusControllerFile = "/var/lib/mesh-genesis/bus-controller.secret"
+	// BusControllerHashFile is the bcrypt hash of it, kept with it. bcrypt salts every hash, so a
+	// hash made afresh on every run would rewrite the bus's user list on every run; kept, the
+	// produced bundle is the same bundle twice.
+	BusControllerHashFile = BusControllerFile + ".bcrypt"
 
 	storeSuperuserMount = "/run/secrets/superuser"
 
@@ -55,15 +64,81 @@ const (
 	templateStorePassword = `"POSTGRES_PASSWORD": "bootstrap"`
 	templateStoreVolumes  = `"volumes": ["mesh-store-data:/var/lib/postgresql/data"]`
 	templateStoreURL      = "postgres:bootstrap@"
-	templateBrokerURL     = "guest:guest@"
-	templateBrokerReady   = "\"verify\": [\"lavinmqctl\", \"status\"]\n    },"
-	brokerAdminMarker     = "/var/lib/lavinmq/.mesh-admin"
+
+	// BusURLPlaceholder is the controller's account in its bus address, as the template names it.
+	BusURLPlaceholder = "nats://" + busControllerUser + ":bootstrap@"
+	// BusUserPlaceholder is the controller's entry in the user list the template carries, up to
+	// its password's hash — written as it stands inside the bundle's JSON string, quotes escaped.
+	BusUserPlaceholder = `{ user: \"` + busControllerUser + `\", password: \"`
+	busControllerUser  = "controller"
 )
 
-// RootCredentials are the two values, and whether this run made them.
+// BusVar is where the control plane finds the mesh's bus, with its own account (mesh-controller
+// internal/broker NATSVar), and CertificateVar the certificate it pins the bus to (ADR 0004).
+const (
+	BusVar         = "MESH_BUS_NATS"
+	CertificateVar = "MESH_BROKER_CERTIFICATE"
+)
+
+// BusPlaceholders refuses a template whose bus connection this installer does not recognise,
+// naming the placeholder it looked for (novox/hq ADR 0331 §3, §4).
+//
+// Asked at preflight, before anything is changed, and again where the bundle is rewritten. A
+// template with no `MESH_BUS_NATS` raises a control plane that cannot reach the mesh's bus and
+// says so every few seconds for ever (issue 146): the verify step would wait out its whole timeout
+// for an answer that cannot come. That is a fault in the bundle, and it is found here instead.
+func BusPlaceholders(template []byte, d *declaration.Declaration) error {
+	control, err := controlPlaneIn(d)
+	if err != nil {
+		return err
+	}
+	address := strings.TrimSpace(control.Env[BusVar])
+	if address == "" {
+		return fmt.Errorf("the template's control plane (%q) is given no %s, so it could not reach the mesh's "+
+			"bus and would never answer. A genesis template names the bus as %s<address>, which this "+
+			"installer fills with the account it makes; mesh-host's examples/foundation-first-node.lock "+
+			"is the shape", control.Name, BusVar, BusURLPlaceholder)
+	}
+	if !strings.HasPrefix(address, BusURLPlaceholder) {
+		return fmt.Errorf("the template's control plane reaches the bus at %s, and this installer looked for "+
+			"%s<address> there: the account it makes goes where that placeholder is, and it will not "+
+			"guess where else a bus connection could be", redactURL(address), BusURLPlaceholder)
+	}
+	if strings.TrimSpace(control.Env[CertificateVar]) == "" {
+		return fmt.Errorf("the template's control plane is given no %s, so it could not pin the bus's "+
+			"certificate (novox/hq ADR 0004) and would refuse to dial it", CertificateVar)
+	}
+	if n := bytes.Count(template, []byte(BusUserPlaceholder)); n != 1 {
+		return fmt.Errorf("the template's user list for the bus names the controller's account %d times, and "+
+			"this installer looked for it once, as %s<hash>: the bus has to be raised with the account the "+
+			"controller dials with, and that is where it goes", n, BusUserPlaceholder)
+	}
+	return nil
+}
+
+// redactURL is a bus address with any password in it replaced, for an error that quotes it.
+func redactURL(address string) string {
+	scheme, rest, has := strings.Cut(address, "://")
+	if !has {
+		rest, scheme = address, ""
+	}
+	at := strings.LastIndex(rest, "@")
+	user, _, hasPassword := strings.Cut(rest[:max(at, 0)], ":")
+	if at < 0 || !hasPassword {
+		return address
+	}
+	if scheme != "" {
+		scheme += "://"
+	}
+	return scheme + user + ":…" + rest[at:]
+}
+
+// RootCredentials are the two values, and whether this run made them. BusHash is the bcrypt hash of
+// Bus, as the bus's user list carries it.
 type RootCredentials struct {
-	Store, Broker         string
-	StoreMade, BrokerMade bool
+	Store, Bus         string
+	BusHash            string
+	StoreMade, BusMade bool
 }
 
 // RootSecrets reads the credentials this machine already holds, or makes them.
@@ -71,17 +146,44 @@ type RootCredentials struct {
 // A dry run makes them in memory and writes nothing — so the bundle it reports is the shape of the
 // real one, and a machine that was only asked is not left holding half a genesis.
 func RootSecrets(dryRun bool) (RootCredentials, error) {
+	return rootSecretsAt(StoreSuperuserFile, BusControllerFile, BusControllerHashFile, dryRun)
+}
+
+func rootSecretsAt(storeFile, busFile, hashFile string, dryRun bool) (RootCredentials, error) {
 	var out RootCredentials
 	var err error
-	if out.Store, out.StoreMade, err = keptOrMade(StoreSuperuserFile, dryRun); err != nil {
+	if out.Store, out.StoreMade, err = keptOrMade(storeFile, dryRun); err != nil {
 		return out, err
 	}
-	if out.Broker, out.BrokerMade, err = keptOrMade(BrokerAdminFile, dryRun); err != nil {
+	if out.Bus, out.BusMade, err = keptOrMade(busFile, dryRun); err != nil {
+		return out, err
+	}
+	if out.BusHash, err = keptHash(hashFile, out.Bus, dryRun); err != nil {
 		return out, err
 	}
 	return out, nil
 }
 
+// keptHash is the bcrypt hash of password kept at path, made again only when the kept one is
+// missing or no longer the hash of this password.
+func keptHash(path, password string, dryRun bool) (string, error) {
+	if kept, err := readCredentialFile(path); err == nil {
+		if bcrypt.CompareHashAndPassword([]byte(kept), []byte(password)) == nil {
+			return kept, nil
+		}
+	} else if !os.IsNotExist(err) {
+		return "", err
+	}
+	hash, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
+	if err != nil {
+		return "", err
+	}
+	if dryRun {
+		return string(hash), nil
+	}
+	return string(hash), writeCredentialFile(path, string(hash))
+}
+
 func keptOrMade(path string, dryRun bool) (value string, made bool, err error) {
 	value, err = readCredentialFile(path)
 	if err == nil {
@@ -97,8 +199,13 @@ func keptOrMade(path string, dryRun bool) (value string, made bool, err error) {
 	if dryRun {
 		return value, true, nil
 	}
+	return value, true, writeCredentialFile(path, value)
+}
+
+// writeCredentialFile writes a credential as genesis keeps it.
+func writeCredentialFile(path, value string) error {
 	if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
-		return "", false, err
+		return err
 	}
 	// Written whole and renamed into place, at 0600, owned by whoever runs the installer — root,
 	// which is also who the host runs as when it later writes the sealed copy here.
@@ -112,12 +219,9 @@ func keptOrMade(path string, dryRun bool) (value string, made bool, err error) {
 	// tolerates either ending, so a file an earlier genesis wrote still reads.
 	tmp := path + ".genesis"
 	if err := os.WriteFile(tmp, []byte(value), 0o600); err != nil {
-		return "", false, err
+		return err
 	}
-	if err := os.Rename(tmp, path); err != nil {
-		return "", false, err
-	}
-	return value, true, nil
+	return os.Rename(tmp, path)
 }
 
 // readCredentialFile is a credential as genesis keeps it: the value alone, its line ending gone.
@@ -134,13 +238,6 @@ func readCredentialFile(path string) (string, error) {
 	return value, nil
 }
 
-// credentialFingerprint names a credential without being one — what the broker-admin action
-// leaves on the broker's volume, so its verify holds for this value and not for any value.
-func credentialFingerprint(value string) string {
-	sum := sha256.Sum256([]byte(value))
-	return hex.EncodeToString(sum[:8])
-}
-
 // freshSecret is the same shape the controller mints: 30 random bytes as unpadded base64url, 40
 // characters, URL-safe — it lands inside connection strings.
 func freshSecret() (string, error) {
@@ -153,55 +250,48 @@ func freshSecret() (string, error) {
 
 // Masking makes a reporter that never says the credentials this run made.
 //
-// The applier reports each action with its command line, and two of them now carry a real
-// password — the context schemas' connection strings and the broker's change_password. Those
-// lines go to a terminal and to whatever keeps the transcript, which for the lab is a file. The
-// exact values are known here, so they are replaced wherever they appear, in every line said.
+// The applier reports each action with its command line, and the context schemas' connection
+// strings carry a real password. Those lines go to a terminal and to whatever keeps the
+// transcript, which for the lab is a file. The exact values are known here, so they are replaced
+// wherever they appear, in every line said.
 func Masking(say func(string), c RootCredentials) func(string) {
-	replacer := strings.NewReplacer(c.Store, "…", c.Broker, "…")
-	if c.Store == "" || c.Broker == "" {
+	if c.Store == "" || c.Bus == "" {
 		return say
 	}
+	replacer := strings.NewReplacer(c.Store, "…", c.Bus, "…")
 	return func(line string) { say(replacer.Replace(line)) }
 }
 
-// RefuseExistingServers stops a run that would put a made credential in front of a server raised
+// RefuseExistingServers stops a run that would put a made credential in front of a store raised
 // by an earlier installer with the template's.
 //
-// The store's password is set by initdb, once, on an empty volume; the broker's by the action
-// above, once. A machine that already holds `mesh-store-data` or `mesh-broker-data` and has no
-// credential file was raised with `bootstrap` and `guest`, and minting new values here would make a
-// bundle that dials with passwords the servers do not have — failing three steps later, in the
-// schemas, with nothing pointing back here. Refused by name instead, with the way forward.
+// The store's password is set by initdb, once, on an empty volume. A machine that already holds
+// `mesh-store-data` and has no credential file was raised with `bootstrap`, and minting a new
+// value here would make a bundle that dials with a password the server does not have — failing
+// three steps later, in the schemas, with nothing pointing back here. Refused by name instead,
+// with the way forward.
+//
+// **The bus is not asked.** Its accounts live in the user list beside its configuration, which
+// the bundle writes and the bus re-reads, not in its data volume — so a bus raised earlier takes
+// the made account as readily as a new one.
 func RefuseExistingServers(ctx context.Context, run Runner, c RootCredentials) error {
-	for _, check := range []struct {
-		made   bool
-		volume string
-		what   string
-		file   string
-	}{
-		{c.StoreMade, "mesh-store-data", "store", StoreSuperuserFile},
-		{c.BrokerMade, "mesh-broker-data", "broker", BrokerAdminFile},
-	} {
-		if !check.made {
-			continue
-		}
-		if _, err := run(ctx, "docker", "volume", "inspect", check.volume); err != nil {
-			continue // no such volume: a fresh machine, which is the case this installer makes
-		}
-		return fmt.Errorf(
-			"this machine already holds the %s's data (volume %s) and no credential at %s, so it was raised "+
-				"by an earlier installer with the template's password. A new one made here would not open it. "+
-				"Put the password the %s has into %s (0600, the value alone) and run again; then change it "+
-				"on the server and accept the new value — this installer does not rotate a running %s",
-			check.what, check.volume, check.file, check.what, check.file, check.what)
+	if !c.StoreMade {
+		return nil
 	}
-	return nil
+	if _, err := run(ctx, "docker", "volume", "inspect", "mesh-store-data"); err != nil {
+		return nil // no such volume: a fresh machine, which is the case this installer makes
+	}
+	return fmt.Errorf(
+		"this machine already holds the store's data (volume mesh-store-data) and no credential at %s, so it was raised "+
+			"by an earlier installer with the template's password. A new one made here would not open it. "+
+			"Put the password the store has into %s (0600, the value alone) and run again; then change it "+
+			"on the server and accept the new value — this installer does not rotate a running store",
+		StoreSuperuserFile, StoreSuperuserFile)
 }
 
 // RootRewrite says what RewriteRoot did to the bundle.
 type RootRewrite struct {
-	StoreURLs, BrokerURLs int
+	StoreURLs, BusURLs int
 }
 
 // RewriteRoot puts the made credentials into the produced bundle, in place of the template's.
@@ -225,37 +315,30 @@ func RewriteRoot(r *Rewritten, c RootCredentials) (RootRewrite, error) {
 		"the store's volumes"); err != nil {
 		return out, err
 	}
-	// Everything that dials the store or the broker with the template's credentials.
+	// Everything that dials the store with the template's credentials.
 	out.StoreURLs = bytes.Count(bundle, []byte(templateStoreURL))
 	if out.StoreURLs == 0 {
 		return out, fmt.Errorf("the template names no %q connection, so this installer cannot tell what it would be leaving well-known", templateStoreURL)
 	}
 	bundle = bytes.ReplaceAll(bundle, []byte(templateStoreURL), []byte("postgres:"+c.Store+"@"))
-	out.BrokerURLs = bytes.Count(bundle, []byte(templateBrokerURL))
-	if out.BrokerURLs == 0 {
-		return out, fmt.Errorf("the template names no %q connection, so this installer cannot tell what it would be leaving well-known", templateBrokerURL)
-	}
-	bundle = bytes.ReplaceAll(bundle, []byte(templateBrokerURL), []byte(BrokerAdminUser+":"+c.Broker+"@"))
 
-	// The broker's administrator, changed once the broker answers and before anything dials it.
-	// Verified by a marker on the broker's own data volume holding this password's fingerprint —
-	// the image carries nothing that can try a password from inside, and a marker that merely
-	// existed would let a regenerated password go unapplied for ever. What proves the password
-	// works is the control plane answering over it, a few resources later. The marker ends in a
-	// newline because the verify reads it with the shell's `read`, which fails at end of file
-	// without one — an action that ran and a verify that said no, once, in the lab.
-	fp := credentialFingerprint(c.Broker)
-	action := templateBrokerReady + "\n" +
-		"    {\n" +
-		"      \"id\": \"broker-admin\",\n" +
-		"      \"type\": \"action\",\n" +
-		"      \"in\": \"mesh-broker\",\n" +
-		"      \"command\": [\"sh\", \"-c\", \"lavinmqctl change_password " + BrokerAdminUser + " '" + c.Broker + "' && echo " + fp + " > " + brokerAdminMarker + "\"],\n" +
-		"      \"verify\": [\"sh\", \"-c\", \"read m < " + brokerAdminMarker + " && [ \\\"$m\\\" = " + fp + " ]\"]\n" +
-		"    },"
-	if bundle, err = replaceOnce(bundle, templateBrokerReady, action, "the broker's readiness check"); err != nil {
+	// The bus: the controller's account, in its address and in the user list the bus is raised
+	// with, the same account in both (novox/hq ADR 0331 §3).
+	if err := BusPlaceholders(bundle, r.Declaration); err != nil {
 		return out, err
 	}
+	out.BusURLs = bytes.Count(bundle, []byte(BusURLPlaceholder))
+	bundle = bytes.ReplaceAll(bundle, []byte(BusURLPlaceholder),
+		[]byte("nats://"+busControllerUser+":"+c.Bus+"@"))
+	at := bytes.Index(bundle, []byte(BusUserPlaceholder)) + len(BusUserPlaceholder)
+	end := bytes.Index(bundle[at:], []byte(`\"`))
+	if end < 0 {
+		return out, fmt.Errorf("the controller's entry in the template's user list for the bus has no end to its password")
+	}
+	if !strings.HasPrefix(c.BusHash, "$2") {
+		return out, fmt.Errorf("the controller's bus password was given no bcrypt hash, so the bus could not be raised with it; this is the installer's fault")
+	}
+	bundle = append(append(append([]byte{}, bundle[:at]...), c.BusHash...), bundle[at+end:]...)
 
 	parsed, err := declaration.ParseFileTrusted(bundle)
 	if err != nil {