hq/03-DESIGN/01-to-be/07-the-substrate.md

layer, status, code, updated, decisions

layer	status	code	updated	decisions
to-be	designed	mesh-host examples/substrate-first-node.lock mesh-host internal/apply mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)	2026-08-31	02-DECISIONS/0004-a-node-and-how-it-joins.md 02-DECISIONS/0005-the-node-host.md 02-DECISIONS/0006-the-substrate-and-the-control-plane.md 02-DECISIONS/0007-connectivity.md 02-DECISIONS/0008-a-context-owns-its-store.md 02-DECISIONS/0019-how-this-repository-works.md

The substrate

Tier 1. Defined the same way the control plane is, because the same gap applied: the word was load-bearing and unpinned.

The definition

The substrate is what the control plane consumes and cannot grant itself.

Every module that needs a database asks the control plane's provisioning for one. The control plane needs a database too — and it cannot ask itself, because it is not running yet. That circularity is not an awkwardness to work around; it is the definition. Anything on the wrong side of it must be raised some other way, and the other way is the bundle the host carries (ADR 0004).

The test, applied:

	control plane needs it	can it grant itself one?
a relational store — PostgreSQL	its own state lives there	no — provisioning needs the store	substrate
a message bus — LavinMQ	it reaches nodes over it (ADR 0002)	no — it cannot grant itself a virtual host	substrate
an object store	artifacts and blobs it delivers	—	not substrate — ADR 0028
an image registry — the OCI registry	images it delivers to nodes	no — it needs a repository	substrate
an identity provider	only if it delegates authentication	—	conditional, below
ingress — Traefik	not to start; only to be reached by name	— it grants itself one afterwards	not substrate (ADR 0007)
anything else the mesh hosts	no	—	not substrate

The object-store row was wrong, and how it was wrong is worth keeping. It answered can it grant itself one — no, it cannot grant itself a bucket — while assuming the first column. The control plane does not need an object store: it has no S3 client and never has, and artifacts reach nodes as content-addressed blobs in the registry. The row was inherited from the system being replaced, where an object store distributed module tarballs, and was never re-tested against the definition above it. Both columns must be answered, and the second is true of almost any service.

An object store is an ordinary module, required through the module graph by whatever wants one. A mesh with no workload needing one runs none.

The role and the product are both written, here and everywhere (ADR 0006). The role is what the argument turns on — the test above works on roles, and would give the same answers for a different store. The product is what actually gets installed and pinned, and a design that names only the role does not record that the choice was ever made.

The dependency is on the protocol, not the product: AMQP for the bus, S3 for the object store, the OCI protocol for the registry. That is what keeps the naming safe rather than a commitment that cannot be revisited — replacing one is a substrate migration, not a redesign. The store is the exception, and the exception matters: the provisioning model uses databases, roles and schemas as PostgreSQL means them, so it is the one member that is not a swap.

What that resolves

Four or five? Research 006 asks whether the identity provider is a substrate service, and the test answers it conditionally — which is the honest answer rather than a number.

• If the control plane delegates authentication, it cannot serve anybody before the provider exists, and it cannot grant itself a client. Substrate.
• If it authenticates natively, the provider is an ordinary hosted service like any other. Not substrate.

So the count follows from a design decision that has not been taken, and the record should say that rather than assert four.

Why not "important infrastructure". An identity provider, a mail server and an analytics service are all infrastructure by any ordinary reading, and none of them are substrate — the control plane starts and runs without them. Important is not the test; the control plane cannot obtain it is.

What the substrate is not

• Not tier 0. The host raises the substrate; it is not part of it. The host carries the declaration that brings the substrate up, and depends on nothing.

• Not the control plane. These are services with no knowledge of the mesh. A store does not know what a node is.

• Not a place for logic. The skeleton is explicit: tier 1 is declarations only, no logic of its own. A substrate service is an upstream image, pinned, with configuration.

• Not privileged. The substrate is provisioned from by the control plane and grants nothing on its own initiative.

• Not the mesh's supply of anything (ADR 0028). A substrate service and a module of the same product are different instances. The mesh's own PostgreSQL and a PostgreSQL a workload was given are two servers, and a node hosting both runs two containers — expected, not duplication to be tidied away.

  The substrate is raised from the bundle before any mesh exists, so it is not in the module graph: a workload depending on it would depend on something the graph cannot see, cannot rotate a credential for, and cannot move. It would also put workload data in the store the control plane keeps its own state in, where a workload that fills a disk takes down the one thing needed to fix it.

The pinned bundle

substrate.lock holds what must exist before the control plane runs — which is a smaller set than the substrate, and the difference is easy to miss. It is the only place in the mesh where versions are pinned by hand rather than resolved.

Being substrate and being in the bundle are two different questions:

	is it substrate?	must it precede the control plane?
PostgreSQL	yes — the control plane's own state lives in it	yes — there is nowhere to put that state otherwise
LavinMQ	yes — it cannot grant itself a virtual host	yes — the control plane reaches a node only over the link, and the link is the broker (ADR 0006)
MinIO	yes — it cannot grant itself a bucket	no — nothing is delivered before the mesh exists
the OCI registry	yes — it cannot grant itself a repository	no — the first node fetches upstream (ADR 0006)

The two on the bottom rows are substrate by role and ordinary by delivery: by the time they are wanted there is a control plane, and it provisions them the way it provisions anything. That keeps the bundle to two images rather than four, which is what makes it small enough for the review ADR 0005 requires. It was one until ADR 0006 established that the broker has to precede the control plane.

Why pinned: the bundle is applied when no mesh exists, so nothing can resolve a version, ask a registry, or check a constraint. What the host carries must already be exact.

Why references and not payload: the bundle names images by digest and the host fetches them (ADR 0006). A first node is a real machine with a network; the sealed case is the lab, and the lab places images itself.

Reproducibility comes from pinning the identity of a thing rather than carrying its bytes, which is what keeps the bundle small enough for a person to read and check.

Raising it

The order, from research 011:

0  a container runtime exists           detected — docker or podman — or installed
1  PostgreSQL runs                      pulled by digest, from the bundle
2  a database per context is created    an action, run locally — one today, `inventory`
3  each context's schema is applied     an action, against its own database
4  LavinMQ runs                         pulled by digest, from the bundle
5  a virtual host, a credential, and    actions, run locally
   a self-signed certificate
6  the control plane starts             and only now is there a mesh
7  MinIO, the registry, and everything  the ordinary path
   else are provisioned

Steps 4 and 5 are why the bundle is not one image (ADR 0006). The control plane cannot provision the broker, because provisioning means telling a host, and telling a host happens over the broker. The first node does not escape this by being local: it enrols the ordinary way, by dialling the broker at the address in its token.

Step 2 is one database per context and not one called mesh. A context is granted only what it exclusively owns (ADR 0008), the mesh database names a thing that will not exist (ADR 0006), and a separate database is a boundary a cross-context join cannot casually cross where a separate schema is not.

Only PostgreSQL is raised from the bundle, for the reason in The pinned bundle above — the rest of the substrate is wanted only once there is a control plane to provision it.

Step 0 is easy to leave out and it is where several things meet. A substrate service is a container, so a container runtime must be working before anything else happens — and a runtime is a package, not a container.

Which runtime is detected, not chosen (ADR 0005): a machine that already has one keeps it. On a machine with none, the control plane names the package, because what it is called differs per system. It is:

• what the host's capability detection already reports, and the first use of that report by something other than a person;
• adopted rather than installed when the machine already has one with configuration somebody chose (research 012);
• a package, which needs the machine's own package manager and a network — both permitted by ADR 0006.

So the host's bootstrap vocabulary is six shapes: package, container, file, directory, service, and action. All six are built (05-the-node-host.md stage 2), so nothing in this bootstrap is blocked on the host any longer.

Steps 2 and 3 happen before there is a mesh to do them, which is why provisioning is part of the bootstrap rather than a service consumers use later. They are actions the bundle declares and the host runs (ADR 0005) — so the host's vocabulary grows by one shape rather than by one resource type per substrate service.

Open

• Whether identity is the fifth. Above; it follows from a decision not yet taken.
• Whether the bus must precede the control plane. Resolved by ADR 0006 — it must, and the question as posed here could not have answered it. This asked whether the control plane's contexts talk to each other over the bus; they do not, being one process, which under this framing would have kept LavinMQ out of the bundle. What decides it is how the control plane reaches a node, which is only ever over the link.
• What issues the broker's certificate at bootstrap. New, and created by the row above. A token pins the fingerprint a host must expect before it sends anything (09-the-node-lifecycle.md), so the broker needs a certificate at a moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and pinned is the shape that fits; how it is later replaced by the certificates in 08-connectivity.md is not decided.
• How a context added later gets its database. By then there is a control plane — but one holding a credential that can create databases holds more than what it exclusively owns (ADR 0008).
• Whether the host can do step 2. Resolved by ADR 0005. A service running on this machine is part of this machine, so the scope was never in question — the real question was whether the host must learn what a database is, and it must not. The bundle declares an action; the host runs it and verifies it, and what a database means stays with the module that provides one.
• Whether one host can raise all four. The claim under stage 2 of the node host, never proved. If it is false, the tier boundary moves.
• How the substrate is updated once a mesh exists. Pinned by hand at bootstrap; afterwards the control plane could deliver it like anything else, and nothing says whether it does.

Raised, and observed

Written 2026-08-30, the first time a bare machine became a running mesh and something joined it.

It works, and what that means precisely: a machine with a container runtime and nothing else applied the bundle its host carries and ended with a store, a database per context, those contexts' schemas, a broker holding a certificate it generated itself, and the control plane serving on top of them. Eleven resources, one command, no mesh to ask anything of.

Then it joined itself. The same machine took a token, checked the broker against the fingerprint pinned in it, generated three keypairs, and enrolled — which is ADR 0004's the first node is a node whose mesh is not up yet, observed rather than argued. Its specialness lasted one command.

And a credential crossed. With a second node recorded, the machine was declared the provider of a database and pushed to over the broker. What arrived and what did not is the whole of the secrets argument, measured on a real machine:

the password, in plain text	on the machine only, one file, mode 0600
in the declaration that crossed the broker	absent
in the control plane's database	absent
in what the node reported back	absent

One fault, and it was in the joining. The token did not say what the mesh calls the machine, so enrolment needed a flag its own help said it did not — and failed at the broker with an empty username. Recorded in ADR 0004 as the fifth thing a token carries.