The bootstrap reaches the broker, and survives a reboot

Five steps now instead of three. A sealed machine goes from bare to a container
runtime, a store, the inventory database, that database's schema applied by
mesh-control, and LavinMQ running and answering.

The database is called inventory rather than mesh. ADR 0008 grants a context
only what it exclusively owns and ADR 0006 says the mesh database names a thing
that will not exist -- so one database per context, and there is one context.

The broker is in the bundle because ADR 0006 now says it must be: the control
plane reaches a node only over the link, the link is the broker, so nothing can
provision the broker. Two images, which is the cost that record accepts.

Verified by reading the system rather than the report: inventory present and
mesh absent, the node table with its indexes, the migration row, lavinmqctl
answering, 5672 listening. Sealed confirmed both ways -- the internet times out,
the lab registry returns 200.

Then rebooted, which was the part worth doing rather than assuming. Everything
returned: docker from boot: enabled, both containers because this host creates
every container --restart unless-stopped, the schema intact in its volume. Three
reconciles before and one after all report no change.

One thing that reads as a success and was not: the first sealed check said the
machine could reach example.com. It was the test that was wrong -- a helper
script pasted arguments into a shell line, so a command with quotes was re-split
and ran on the workstation. The machine had been sealed the whole time. The
helper now requotes each argument.

examples/README.md

@@ -2,13 +2,20 @@
 
 ## `substrate-first-node.lock`
 
-What an Arch machine must be before a mesh exists — the first steps of the bootstrap in
+What a machine must be before a mesh exists — steps 0 to 4 of the bootstrap in
-[novox/hq `07-the-substrate.md`](https://git.novox.be/novox/hq): a container runtime, a store
+[novox/hq `07-the-substrate.md`](https://git.novox.be/novox/hq):
-running, and the control plane's database created inside it.
 
-**It stops there**, and the file says why: loading the control plane's schema and starting the
+```
-control plane are the next two steps, and there is no control plane yet. A bundle naming one
+0  a container runtime
-would be a bundle that cannot be applied.
+1  the store runs
+2  a database per context        one today, `inventory`
+3  that context's schema         mesh-control migrate
+4  the broker runs
+```
+
+**It stops there, and the file says why.** Step 5 is a virtual host, a credential and a
+certificate; step 6 is the control plane running. Nothing consumes any of them yet, and a bundle
+whose last step cannot be checked is worse than a shorter one.
 
 Build a host carrying it:
 
@@ -16,11 +23,26 @@ Build a host carrying it:
 make host SYSTEM=arch BUNDLE=examples/substrate-first-node.lock
 ```
 
-**The image reference has to be replaced before this is useful.** It is written as
+**The registry address and digests have to be replaced before this is useful.** They are written
-`REGISTRY/postgres@DIGEST` because the digest belongs to whatever registry serves it — in the
+as `192.0.2.250:5000/…@sha256:…` because a digest belongs to whatever registry serves it — here,
-lab, one the scenario raises, which reports its digests when it comes up. That is not a
+one a lab scenario raises, which reports its digests when it comes up. That is not a placeholder
-placeholder to be tidied away: a bundle is built *for a target*, and which registry that target
+to be tidied away: a bundle is built *for a target*, and which registry that target pulls from is
-pulls from is part of the target.
+part of the target.
 
-Verified end to end in a lab machine with no route out: five resources applied, idempotent on a
+### What was verified, and how
-second run, and `mesh` present in the store afterwards.
+
+On a lab machine confirmed sealed — `curl https://example.com` times out, the lab registry answers
+200 — the whole bundle applied from bare: eight resources, `inventory` created and `mesh` nowhere,
+the `node` table present with its indexes, the migration recorded, and LavinMQ answering
+`lavinmqctl status` with AMQP listening on 5672.
+
+Three consecutive reconciles after that: **already matches — 8 resource(s) checked**, each time.
+
+Then the machine was **rebooted**, and everything came back: docker from `boot: enabled`, both
+containers because the host creates every container `--restart unless-stopped`
+(`internal/apply/apply.go`), the schema intact in its named volume, and reconcile still finding
+nothing to do.
+
+The reboot is worth doing rather than assuming. Nothing in the declaration asks for a container to
+return, so that it does is a property of the host, and the only way to know it holds is to take
+the machine away and give it back.

examples/substrate-first-node.lock

@@ -1,18 +1,22 @@
-// substrate-arch.lock — what an Arch machine must be before a mesh exists.
+// substrate-first-node.lock — what a machine must be before a mesh exists.
 //
-// The first three steps of the bootstrap (novox/hq 03-DESIGN/01-to-be/07-the-substrate.md).
+// Steps 0 to 4 of the bootstrap (novox/hq 03-DESIGN/01-to-be/07-the-substrate.md): a container
-// Steps four and five — load the control plane's schema, start the control plane — are absent
+// runtime, a store, a database per context, that context's schema, and the broker.
-// because the control plane does not exist yet. A bundle that named it would be a bundle that
+//
-// cannot be applied.
+// It stops before step 5 (a virtual host, a credential, a certificate) and step 6 (the control
+// plane runs), because nothing consumes them yet. A bundle naming a control plane that serves
+// nothing would be a bundle whose last step cannot be checked.
 //
 // PINNED BY DIGEST, and the digest is not decoration: a tag can be made to point at a different
-// image, and this file is applied on a machine with no mesh to ask about anything.
+// image, and this file is applied on a machine with no mesh to ask about anything. These digests
+// belong to the registry the lab raises, which is what a real node pulls from anyway — what is
+// required is a reference that is exact and cannot move (novox/hq ADR 0006).
 //
-// The store's data is a NAMED VOLUME rather than a directory on the machine. A directory the
+// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
-// host creates is owned by root, and the database runs as somebody else inside the container —
+// creates is owned by root, and the database runs as somebody else inside the container — so it
-// so it could not write, and the container crash-looped. A named volume lets the image set up
+// could not write, and the container crash-looped. A named volume lets the image set up its own
-// its own ownership, which is what it is for. It also outlives the container, which is what you
+// ownership, and outlives the container, which is what you want for the thing holding the mesh's
-// want for the thing holding the mesh's state.
+// state.
 {
   "declaration": 1,
   "resources": [
@@ -32,7 +36,7 @@
       "id": "store",
       "type": "container",
       "name": "mesh-store",
-      "image": "REGISTRY/postgres@DIGEST",
+      "image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
       "env": {
         "POSTGRES_PASSWORD": "bootstrap",
         "PGDATA": "/var/lib/postgresql/data/pgdata"
@@ -47,11 +51,35 @@
       "verify": ["pg_isready", "-U", "postgres"]
     },
     {
-      "id": "control-plane-database",
+      "id": "inventory-database",
       "type": "action",
       "in": "mesh-store",
-      "command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE mesh'"],
+      "command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
-      "verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw mesh"]
+      "verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
+    },
+    {
+      "id": "inventory-schema",
+      "type": "action",
+      "command": ["docker", "run", "--rm", "--network", "container:mesh-store",
+                  "-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
+                  "192.0.2.250:5000/mesh-control@sha256:1c27a43c4431c2b580404e8e1768cd858009e265e80b6f1591eb6de1123fc411",
+                  "migrate"],
+      "verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node"]
+    },
+    {
+      "id": "broker",
+      "type": "container",
+      "name": "mesh-broker",
+      "image": "192.0.2.250:5000/cloudamqp/lavinmq@sha256:b117c254e6e269a29db479e6b410ca4e46e035b4981e49d24b159673ef09d336",
+      "ports": ["5672:5672"],
+      "volumes": ["mesh-broker-data:/var/lib/lavinmq"]
+    },
+    {
+      "id": "broker-ready",
+      "type": "action",
+      "in": "mesh-broker",
+      "command": ["sh", "-c", "for i in $(seq 1 60); do lavinmqctl status >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
+      "verify": ["lavinmqctl", "status"]
     }
   ]
 }