diff --git a/02-DECISIONS/0024-running-the-control-plane.md b/02-DECISIONS/0024-running-the-control-plane.md new file mode 100644 index 0000000..69ec079 --- /dev/null +++ b/02-DECISIONS/0024-running-the-control-plane.md @@ -0,0 +1,140 @@ +--- +topic: the tiers +status: proposed +date: 2026-08-29 +deciders: jochen +reconstructed: false +extends: 0006-the-substrate-and-the-control-plane.md +--- + +# 24. Running the control plane + +[ADR 0006](0006-the-substrate-and-the-control-plane.md) settles what the control plane *is* — seven +contexts, one deployable, one node runs it. This settles what it takes to actually run one, which +is the half you discover by trying to build it. + +Two decisions. Neither is large; both were blocking, and one of them had been asked in the design +in a form that could not answer it. + +## It is written in Go + +The same language as the host, so tiers 0 and 2 are one language and not two. + +The reason that decides it is not familiarity. **The control plane's image is pinned by digest in +the bundle the host carries** ([ADR 0006](0006-the-substrate-and-the-control-plane.md)), which +means it is fetched and run on a machine where no mesh exists yet — nothing to check it against, +nothing watching, and a person expected to have read the bundle and believed it. A statically +linked binary makes that image the program and nothing else: no interpreter, no package tree, no +transitive dependency that arrived because something needed a date library. + +Everything under that line is a thing somebody would have to audit, on the one image the whole +mesh is raised from. + +A second reason, smaller and still real: the control plane runs a reconcile loop of its own +([ADR 0010](0010-delivery.md) — artifacts against source, the way the host reconciles machine state +against declarations). Two loops with the same shape are cheaper to hold in one head when they are +also the same language. + +### The option rejected, and why the obvious argument for it does not hold + +**TypeScript**, matching the lab and the surfaces that will speak to this. The argument is that +tier 3 is web and CLI, so tier 3 is TypeScript, so a TypeScript control plane shares its types with +its callers directly rather than through a generated contract. + +That argument is true and does not reach far enough. `mesh-sdk` is *contracts shared across tiers* +and **tier 0 is Go** — so the contracts have to survive a language boundary no matter what tier 2 +is written in. The choice is not between sharing types and generating them; it is between +generating them for one consumer or for two. That is a much smaller difference than it first looks, +and it does not outweigh what is in the image. + +**What this costs, stated plainly:** the control plane cannot import from the lab or from anything +that exists today, and a person moving between tier 2 and tier 3 changes language. Neither is +recovered later — a language is the most expensive thing in this record to reverse. + +## The message broker is in the bundle + +**LavinMQ is raised from the bundle, alongside PostgreSQL, before the control plane starts.** + +[ADR 0006](0006-the-substrate-and-the-control-plane.md) left this not established, and +[`07-the-substrate.md`](../03-DESIGN/01-to-be/07-the-substrate.md) recorded the open question as +turning on *whether the control plane's own contexts talk to each other over the bus*. + +**They do not** — they are one process and dispatch internally +([ADR 0006](0006-the-substrate-and-the-control-plane.md)). Under the question as asked, that answer +means the broker is provisioned like anything else, and the bundle stays at one image. + +**The question was asked on the wrong axis.** What decides it is not how the contexts reach each +other. It is how the control plane reaches a *node* — and the answer, already decided, is that it +never reaches one except over the link, and the link is AMQP +([ADR 0002](0002-nodes-communicate-over-a-broker.md), +[ADR 0004](0004-a-node-and-how-it-joins.md)). + +The first node then closes a circle that has no way out: + +``` +the bundle raises the control plane +the control plane provisions the broker ← by telling a host to run it +telling a host happens over the link +the link is the broker +``` + +And it is not avoided by the first node being local. `enrol` **dials the broker at the address in +the token** ([`09-the-node-lifecycle.md`](../03-DESIGN/01-to-be/09-the-node-lifecycle.md)), which is +step 3 of the first node's own path. A machine that raised the mesh still joins it the ordinary +way, and that was deliberate — *its specialness lasted two commands*. Making the first node join by +some other route would buy the smaller bundle by giving up the property the design was built to +have. + +So the broker precedes the control plane, and it precedes it for the same reason PostgreSQL does: +**the control plane cannot grant itself the thing it would need in order to grant it.** + +### What this costs + +**The bundle is two images rather than one**, against +[ADR 0006](0006-the-substrate-and-the-control-plane.md)'s wish to keep it at roughly one so that a +person can read it. Two is still readable. Four would not have been, which is why the object store +and the registry stay out — nothing needs them to reach a node. + +**And it grows three things that are not an image**, each an action the bundle declares and the +host runs, the way the database already is: + +- a virtual host, because the control plane cannot grant itself one + ([ADR 0006](0006-the-substrate-and-the-control-plane.md)); +- a credential on it for the control plane; +- **a certificate, and this is the awkward one.** A token pins the fingerprint the host must expect + *before it sends anything* + ([`09-the-node-lifecycle.md`](../03-DESIGN/01-to-be/09-the-node-lifecycle.md)), so the broker + needs a certificate at bootstrap, when there is no mesh to issue one and no public name to get + one for. Self-signed and pinned is the shape that fits, and how it is later replaced by the + connectivity design's certificates is **not decided here**. + +## What follows from ADR 0008, and was being built wrong + +Not a decision — a correction, recorded because the mistake was already in a file that runs. + +[ADR 0008](0008-a-context-owns-its-store.md) grants a context only what it **exclusively owns**, and +rejects a schema per consumer inside a shared database on the grounds that *a boundary that is +merely inconvenient to cross gets crossed*. +[ADR 0006](0006-the-substrate-and-the-control-plane.md) says there is no single mesh database and +that *the mesh database* names a thing that will not exist. + +The bootstrap created one database and called it `mesh`. + +**One database per context, named for the context.** Contexts that do not exist yet do not get one, +so the bundle today creates exactly one, called `inventory`. A separate database rather than a +separate schema is the point: in PostgreSQL a cross-schema join is a qualified name away, and a +cross-database join needs a foreign data wrapper somebody has to install and explain. + +**How a context added later gets its database is not answered here**, and it is a real question — +by then there is a control plane, but a control plane holding a credential that can create +databases is holding rather more than the thing it exclusively owns. + +## Consequences + +- `novox/mesh-control` exists, in Go, and tier 2 stops being the tier where nothing is built. +- The bundle carries two images and four actions, and the substrate bootstrap grows a step. +- Nothing in the first node's path is special-cased. Enrolment is walked on node one, as designed. +- The certificate at bootstrap is named as unfinished rather than assumed. It is the first thing + that will be wanted when the link is built, and it has no answer yet. +- The language cannot be revisited cheaply. It is the one line in this record that is close to + irreversible. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index b3a5767..7047fb3 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -92,6 +92,7 @@ python3 00-META/checks/index.py fail if stale - **0006** — [The substrate and the control plane](0006-the-substrate-and-the-control-plane.md) - **0007** — [Connectivity](0007-connectivity.md) - **0008** — [A context owns its store, exclusively](0008-a-context-owns-its-store.md) +- **0024** — [Running the control plane](0024-running-the-control-plane.md) *(proposed)* ### What runs on them, and how it gets there diff --git a/03-DESIGN/01-to-be/07-the-substrate.md b/03-DESIGN/01-to-be/07-the-substrate.md index 3002246..387b824 100644 --- a/03-DESIGN/01-to-be/07-the-substrate.md +++ b/03-DESIGN/01-to-be/07-the-substrate.md @@ -4,14 +4,12 @@ status: designed code: [] updated: 2026-08-27 decisions: - - 02-DECISIONS/0019-how-this-repository-works.md - - 02-DECISIONS/0005-the-node-host.md - 02-DECISIONS/0004-a-node-and-how-it-joins.md - - 02-DECISIONS/0006-the-substrate-and-the-control-plane.md - 02-DECISIONS/0005-the-node-host.md - 02-DECISIONS/0006-the-substrate-and-the-control-plane.md - 02-DECISIONS/0007-connectivity.md - - 02-DECISIONS/0005-the-node-host.md + - 02-DECISIONS/0008-a-context-owns-its-store.md + - 02-DECISIONS/0019-how-this-repository-works.md --- # The substrate @@ -94,15 +92,16 @@ Being substrate and being in the bundle are two different questions: | | is it substrate? | must it precede the control plane? | |---|---|---| | PostgreSQL | yes — the control plane's own state lives in it | **yes** — there is nowhere to put that state otherwise | -| LavinMQ | yes — it cannot grant itself a virtual host | **not established** — see below | +| LavinMQ | yes — it cannot grant itself a virtual host | **yes** — the control plane reaches a node only over the link, and the link is the broker ([ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md)) | | MinIO | yes — it cannot grant itself a bucket | no — nothing is delivered before the mesh exists | | the OCI registry | yes — it cannot grant itself a repository | no — the first node fetches upstream ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) | -The three on the bottom rows are **substrate by role and ordinary by delivery**: by the time -they are wanted there is a control plane, and it provisions them the way it provisions anything. -That keeps the bundle to roughly one image rather than four, which is what makes it small enough -for the review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) -requires. +The two on the bottom rows are **substrate by role and ordinary by delivery**: by the time they +are wanted there is a control plane, and it provisions them the way it provisions anything. +That keeps the bundle to two images rather than four, which is what makes it small enough for the +review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) requires. It was one until +[ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md) established that the broker has +to precede the control plane. **Why pinned:** the bundle is applied when no mesh exists, so nothing can resolve a version, ask a registry, or check a constraint. What the host carries must already be exact. @@ -121,13 +120,28 @@ The order, from [research 011](../../01-RESEARCH/011-the-module-graph/worked-pro ``` 0 a container runtime exists detected — docker or podman — or installed 1 PostgreSQL runs pulled by digest, from the bundle -2 a database is created in it an action, run locally -3 the control plane's schema applied an action, against that database -4 the control plane starts and only now is there a mesh -5 LavinMQ, MinIO, the registry, and the ordinary path - everything else are provisioned +2 a database per context is created an action, run locally — one today, `inventory` +3 each context's schema is applied an action, against its own database +4 LavinMQ runs pulled by digest, from the bundle +5 a virtual host, a credential, and actions, run locally + a self-signed certificate +6 the control plane starts and only now is there a mesh +7 MinIO, the registry, and everything the ordinary path + else are provisioned ``` +**Steps 4 and 5 are why the bundle is not one image** +([ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md)). The control plane cannot +provision the broker, because provisioning means telling a host, and telling a host happens over +the broker. The first node does not escape this by being local: it enrols the ordinary way, by +dialling the broker at the address in its token. + +**Step 2 is one database per context and not one called `mesh`.** A context is granted only what it +exclusively owns ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), *the mesh +database* names a thing that will not exist +([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)), and a separate +database is a boundary a cross-context join cannot casually cross where a separate schema is not. + Only PostgreSQL is raised from the bundle, for the reason in *The pinned bundle* above — the rest of the substrate is wanted only once there is a control plane to provision it. @@ -161,14 +175,21 @@ host's vocabulary grows by one shape rather than by one resource type per substr ## Open - **Whether identity is the fifth.** Above; it follows from a decision not yet taken. -- **Whether the bus must precede the control plane.** The bundle table marks this *not - established*, and it is the one row that could still move. The control plane reaches nodes over - AMQP, but at step 4 there is exactly one node and it is the local machine — so whether LavinMQ - is needed to *start* or only to *reach a second node* depends on whether the control plane's own - contexts talk to each other over the bus. If they do, LavinMQ joins PostgreSQL in the bundle and - the bootstrap grows a step; if they do not, it is provisioned like anything else. **This is a - question about the control plane's internal shape, not about the substrate**, which is why it is - not answered here. +- ~~**Whether the bus must precede the control plane.**~~ **Resolved** by + [ADR 0024](../../02-DECISIONS/0024-running-the-control-plane.md) — it must, and the question as + posed here could not have answered it. This asked whether the control plane's contexts talk to + each other over the bus; they do not, being one process, which under this framing would have + kept LavinMQ out of the bundle. What decides it is how the control plane reaches a *node*, which + is only ever over the link. +- **What issues the broker's certificate at bootstrap.** New, and created by the row above. A + token pins the fingerprint a host must expect before it sends anything + ([`09-the-node-lifecycle.md`](09-the-node-lifecycle.md)), so the broker needs a certificate at a + moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and + pinned is the shape that fits; how it is later replaced by the certificates in + [`08-connectivity.md`](08-connectivity.md) is not decided. +- **How a context added later gets its database.** By then there is a control plane — but one + holding a credential that can create databases holds more than what it exclusively owns + ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)). - ~~**Whether the host can do step 2.**~~ **Resolved** by [ADR 0005](../../02-DECISIONS/0005-the-node-host.md). A service running on this machine is part of this machine, so the scope was never in question — the real