Glossary, the mesh-controller/foundation vocabulary, ADR 0076, Phase 3 closed #43

Merged
jschoubben merged 32 commits from issue/047-the-other-half into main 2026-09-16 21:25:51 +00:00
39 changed files with 2162 additions and 205 deletions
+1 -1
View File
@@ -30,7 +30,7 @@ named, and nothing should be designed around a particular one existing.
- **A hosted model provider** supplies the thinking for non-human agents, drawn from a - **A hosted model provider** supplies the thinking for non-human agents, drawn from a
shared pool of subscriptions — which is why budget pacing is a first-class concern. shared pool of subscriptions — which is why budget pacing is a first-class concern.
- **Long-lived user services** rather than an orchestrator. No cluster scheduler, no cloud - **Long-lived user services** rather than an orchestrator. No cluster scheduler, no cloud
control plane. controller.
Defaults, not mandates. A second model provider is anticipated by design; nothing in the Defaults, not mandates. A second model provider is anticipated by design; nothing in the
domain may assume one vendor's credential lifecycle. domain may assume one vendor's credential lifecycle.
+60
View File
@@ -0,0 +1,60 @@
# Glossary — the words this repository uses, and the ones it stopped using
One name per thing. This page is the authority; where an older record says something else, that
record is being superseded, not this page. It exists because the terms kept drifting in
conversation — control plane / controller / master / hub for one thing, substrate / foundation for
another — and a mesh you cannot name precisely is a mesh two people describe differently.
## The mesh and its machines
- **node** — a machine in the mesh. There are 0..n of them, and each runs the host agent. A node is
just a machine that has joined; being one implies nothing about what it runs.
- **control-node** — the one node that also holds the `the-controller` seat. There is exactly one
per mesh. "control-node" is not a separate kind of machine — it is a node that additionally runs
the controller (and, today, the foundation). Lose it and the other nodes keep running what they
were last told; they simply cannot be told anything new.
- ~~master / slave~~, ~~hub / peer~~ — not used. The relationship is *controller and nodes*, and no
node is subordinate: a node applies declarations on its own and survives the control-node dying.
## What runs the mesh
- **controller** — the component that decides what each node should be, holds the mesh's records,
and tells nodes over the broker. Replaces **"control plane"** (borrowed from networking's
control-plane/data-plane, and opaque here).
- **mesh-controller** — the module that runs the controller. It **claims** the `the-controller`
seat at mesh scope, which is what makes it singular. Replaces the module name **`mesh-control`**.
(The git repository is still named `mesh-control` until it is renamed on the forge; the module,
container and image it produces are `mesh-controller`.)
- **foundation** — the store and the broker, raised at genesis before any module system exists.
Replaces **"substrate"** (a biology metaphor that landed for no one). The foundation is not a
third thing beside the store and broker — it *is* those two, named together.
- **store** — the one postgres server. It holds the controller's own context databases
(`inventory`, `identity`, `licences` — a context owns its store, [ADR 0008](../02-DECISIONS/0008-a-context-owns-its-store.md))
and every module's own database. One server, many databases — never one shared "mesh database".
- **broker** — the one lavinmq message bus. It carries the mesh bus on the `/` vhost and a vhost per
consumer that requires `amqp`.
## What the mesh stores and serves
- **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by
**version**. Served by the **package-registry** (gitea). Only a builder talks to it.
- **artifact** — what the mesh delivers to a machine to **install and run**: an OCI image, by
**digest**. Served by the **artifact-store** (distribution). Every node pulls from it.
- These are two protocols, not one store being weak — see [ADR 0075](../02-DECISIONS/0075-two-stores-and-which-provides-what.md).
## How modules relate to the mesh
- **seat** — a named position at a scope (node / site / mesh) with a **capacity**. A capacity-1 seat
is exclusive (one holder); a higher-capacity seat is a **bench** (several holders coexist).
- **claim** — a module taking a spot on a seat. `claims: [{name, scope}]` in a manifest. A
mesh-scoped exclusive claim is how the mesh says "there is one of me" — e.g. `mesh-controller`
claims `the-controller`.
- **provision** — a service one module `provides` and others `require`; the mesh resolves a provider
and wires the two with an endpoint and a credential. This is separate from seats: a provision is
a service you offer, a seat is a slot you occupy.
## How this page is kept
A new name for an existing thing lands here first, in the same change that introduces it in code. A
record under `02-DECISIONS/` keeps whatever word it was written with — those are immutable — so a
term retired here may still appear there, and the mapping above is how to read it.
+2 -2
View File
@@ -28,8 +28,8 @@ target, not the present.
| Repository | Tier | Holds | | Repository | Tier | Holds |
|---|---|---| |---|---|---|
| `mesh-host` | 0 | **exists.** The node host — one statically linked binary, requiring nothing present ([ADR 0005](../02-DECISIONS/0005-the-node-host.md)) | | `mesh-host` | 0 | **exists.** The node host — one statically linked binary, requiring nothing present ([ADR 0005](../02-DECISIONS/0005-the-node-host.md)) |
| `mesh-substrate` | 1 | the four pinned services, as declarations | | `mesh-foundation` | 1 | the four pinned services, as declarations |
| `mesh-control` | 2 | **exists.** The control plane and its contexts — one of seven built ([ADR 0006](../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) | | `mesh-control` | 2 | **exists.** The controller and its contexts — one of seven built ([ADR 0006](../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
| `mesh-surfaces` | 3 | tools, web, cli | | `mesh-surfaces` | 3 | tools, web, cli |
| `mesh-sdk` | — | the stable spine modules build against — the tool-serving harness, the messaging/event framework, the contracts and core primitives. Holds nothing per-module and nothing volatile ([ADR 0039](../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)). | | `mesh-sdk` | — | the stable spine modules build against — the tool-serving harness, the messaging/event framework, the contracts and core primitives. Holds nothing per-module and nothing volatile ([ADR 0039](../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)). |
| `mesh-lab` | — | **exists.** The lab — scenario lifecycle, networking, placement. Ships to nobody; runs on a workstation. | | `mesh-lab` | — | **exists.** The lab — scenario lifecycle, networking, placement. Ships to nobody; runs on a workstation. |
@@ -0,0 +1,172 @@
---
topic: the tiers
status: accepted
date: 2026-09-15
deciders: jochen
reconstructed: false
extends: 0039-what-the-sdk-holds-and-refuses.md
---
# 74. The mesh defines a module protocol; an SDK is an implementation of it
## Context
[ADR 0039](0039-what-the-sdk-holds-and-refuses.md) says what the SDK holds: the tool-serving
harness, the messaging and event framework, the contracts, and core primitives. It settles what
belongs in *an* SDK. It does not say what happens when there is more than one.
There is already more than one. **The contracts are expressed twice** — as Go types in the control
plane and the host, and as TypeScript types in the SDK — and nobody has felt it because both live
in one repository and one head.
**A correction, made after inspecting the wire rather than the types** (2026-09-16). This record
first claimed the two implementations already disagreed — `resource` vs `Provision`, `consumer`
meaning the module in one and the node in the other, headers declared on one side and emitted by
neither. **On inspection the live wire agrees**, and the claim was wrong:
- The grant types that disagreed (`Grant`, `Interface`, `Credential` in the SDK's `contracts`)
were **dead** — exported and imported by nothing. The live provisioning wire is the contributions
file, whose shape (`as`, `secret`, `node`, `at`, `values`) is the same on both sides. Those dead
types have been removed.
- The envelope agrees too: Go emits all five required headers, and `x-causation-id`/`x-schema` are
**optional** — the SDK sets them when a handler has a causation or a schema, and a bare event
carrying neither is correct, not a drift.
So the danger was never live disagreement. It was **dead types that contradicted the live wire**,
which read as the contract and were not — and are exactly what led this record to assert a drift
that inspection did not find. That is a sharper reason for the decision below, not a weaker one: a
type is only as good as its being the wire, and the way to guarantee that is to specify the wire and
check implementations against it, rather than to trust a hand-kept type to still describe it.
A failure of this kind does not announce itself. Two implementations that disagree about an
envelope do not fail to compile — they ignore each other's messages, and a mesh where a module
stops reacting looks exactly like a mesh where nothing happened.
## The question this settles
A module may be written in any language the mesh can build
([`18-building-a-module`](../03-DESIGN/01-to-be/18-building-a-module.md)). Every language needs an
SDK. What is an SDK *of*?
Two answers were available, and the obvious one is wrong.
**Shared types, generated.** Write the shapes once — a schema, an IDL — and generate Go, TypeScript,
Rust. It is the familiar answer and it solves the smaller half of the problem. The shapes are not
where the difficulty is.
**A specified wire, with a conformance suite.** The shapes are a consequence; what an SDK must get
right is *behaviour*.
## Decision
**The mesh defines a module protocol. An SDK is an implementation of that protocol in one
language, and nothing more.**
That is the whole of what an SDK is. Not a library a language happens to have, not a convenience
layer, not a place for helpers to accumulate — an implementation of a specified protocol, finished
when it implements it and correct when it agrees with every other implementation.
### The protocol is split per capability
**A module does not use all of it, so an SDK need not implement all of it.** A module that only
consumes events uses the event capability. One that serves tools uses the tool capability. A
provider uses provisioning. Nothing about consuming an event requires knowing how a grant is
answered.
So the protocol is a floor plus capabilities:
| part | what it covers | who needs it |
|---|---|---|
| **connection** — the floor | reading the sealed credential, pinning the certificate fingerprint, taking identity from the credential rather than the environment | everything |
| **events** | the envelope and its headers, the durable per-consumer queue, binding, at-least-once with dedup on `x-event-id` | a module that emits or consumes |
| **tools** | registration, the shared durable `serve.<key>` queue, request and reply | a module with a surface |
| **provisioning** | a grant in, a credential out, and what each carries | a module that provides something |
**This is the same shape the host already has.** A host declares which resource kinds it can apply,
and a partial host — one that can write files and run things but not manage users or containers —
is a real thing rather than a broken one ([ADR 0005](0005-the-node-host.md)). An SDK that implements
the floor and events is exactly as legitimate, and a module written against it is a module that
does events.
**So a language arrives in pieces rather than all at once.** A Rust SDK implementing connection and
events is useful the day it exists; tools and provisioning follow when something needs them. The
alternative — a language is unsupported until it is entirely supported — is what makes adding one a
project rather than a contribution.
**And what a language can be used for is then a fact the mesh can state**, rather than something an
author discovers by writing a module that cannot be built: the toolchain list says which languages
exist, and the conformance results say what each can do.
### What the specification covers
Per capability, what two implementations can disagree about:
- **the exchanges and queues** — which exchanges exist, that a consumer's queue is durable and
named `<node>.<module>.events`, that a tool is served from a shared durable `serve.<key>`
- **the envelope** — every header, which are required, what an unknown `x-` header means, and that
ignoring one is correct rather than lax
- **identity** — that a module's node and module name come from its sealed credential and not from
its environment, so what it emits matches what the mesh authorised
- **delivery** — at-least-once, and that dedup is on `x-event-id`, which only the emitter can make
- **the credential** — the sealed document's fields, and that a connection pins a certificate
fingerprint rather than trusting an authority
- **provisioning** — a grant in, a credential out, and what each carries
- **the vocabulary** — that `consumer` is one thing, named once
### Conformance is per capability
**A suite per part, and an SDK claims the parts it passes.** A monolithic pass/fail would make a
partial implementation indistinguishable from a broken one, which is the distinction this is built
on.
**And the suite is executable, not prose.** A specification nobody can run is a document two
implementations drift from while both believe they conform. Conformance is a set of fixtures — an
emitted event, a served tool call, a grant and its answer — that every SDK must produce and consume
byte-for-byte.
**The existing two implementations are the first two to be made to pass it.** Not a future language:
the drift above is present, and a suite that only new SDKs must satisfy would leave the disagreement
that already exists in place while certifying everything added afterwards against it.
## Why not generated types
Generation makes the shapes agree and leaves everything that matters unspecified. Two SDKs
generated from one schema can still name their queues differently, take identity from the
environment, dedup on the wrong field, or omit a header the other requires — and every one of those
is a mesh that runs and quietly does not work.
It also makes the contract into whatever the generator supports, which is a decision nobody made
about a boundary everything else depends on.
**The shapes are worth generating once the wire is specified.** That is a convenience, and it comes
second.
## Consequences
**A language is a commitment, and now a divisible one.** Adding one means implementing the protocol
and passing the suites for the parts it claims. That is more work than transliterating types, and
it is the work that was always there — the difference is that it can be finished, and finished in
pieces, rather than believed.
**Versioning becomes possible.** `x-schema` exists for it and is never written. A specified envelope
with a version on the body is what lets a mesh hold a module built against an older SDK, which is
the ordinary state of any mesh that has been running for a while.
**The two current implementations agree on the live wire** — inspection showed it. What was wrong
was a set of dead types beside the wire, now removed. The suite's job here is therefore prevention:
to keep that agreement true as the wire changes, and to hold a new language's SDK to it, rather than
to repair a break that exists today.
**This does not make the mesh polyglot by itself**, and should not be reported as though it does. It
makes polyglot possible to do correctly. A Rust SDK is still a Rust SDK.
## How this is checked
| Rule | Checked by |
|---|---|
| One vocabulary | A word means one thing across implementations, checked by the fixtures using it. |
| The wire is what is specified | Both existing SDKs run the conformance suite in their own test suites, and a change to one that breaks a fixture fails there rather than in a mesh. |
| A new SDK is a passing SDK | A language is not listed as buildable for a capability until its SDK passes that capability's suite; the toolchain list and the conformance results name the same set. |
| A partial SDK is a real thing | An SDK implementing the floor and one capability passes, is listed for that capability, and a module using another is refused with the reason — rather than failing at runtime in a language nobody said was finished. |
| An unknown header is ignored | A fixture carries one, and every implementation accepts it. |
| Identity comes from the credential | A fixture sets an environment that disagrees with the credential, and the emitted event carries the credential's. |
@@ -0,0 +1,129 @@
---
topic: the tiers
status: proposed
date: 2026-09-15
deciders: jochen
reconstructed: false
extends: 0014-no-npm-workspace.md
---
# 75. An artifact store is a provision; a package registry is a different one
## Context
Two questions have been circling, and they turn out to be one question asked twice.
**"Should gitea be the mesh's registry?"** It serves OCI images and a dozen package ecosystems, it
is already needed — genesis clones from one — and the mesh's own registry has neither
authentication ([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md))
nor a transport a runtime will accept over a network
([issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)). Gitea
has both.
**"Where does the SDK come from?"** [ADR 0014](0014-no-npm-workspace.md) already answers it — each
module consumes its dependencies, the mesh's own shared library included, *from the private
registry* — and nothing installs one, so today it comes from a git URL, which is
[issue 053](../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md).
**The framing that dissolves both:** `artifact-store` is already a provision, and `registry`
already provides it. So "should gitea be the registry" is not a question about replacing a
component. It is a question about **a second provider of an existing provision** — which this mesh
has a mechanism for, and uses for certificate authorities and VPNs already.
## Decision
**Two provisions, because they are two jobs.**
| provision | is | for |
|---|---|---|
| `artifact-store` | content-addressed blobs, pinned by digest, no versions, no ranges | what the **mesh** delivers to **machines** |
| `package-registry` | an ecosystem's own registry — npm, cargo, PyPI, Go | what **code** resolves when it is compiled |
They are not the same store with different clients. One is addressed by digest and immutable by
construction; the other is addressed by name and version, and resolves ranges. Conflating them is
how a mesh that pins everything ends up rebuilding one commit into two different things.
**`registry` remains the provider genesis installs.** Not because it is better, but because of what
it is: a directory and one container, no database, no control plane, installable at step 8 of an
install where neither exists yet. Gitea needs a store and provisioning, which means a control plane,
which means the pivot has already happened — and the pivot needs somewhere to publish to.
**Gitea also provides `artifact-store`, and a mesh may choose it.** Two providers of one provision
is a thing the mesh understands: it refuses, names both, and choosing is assigning the one you want.
**And it does not claim `the-artifact-store`.** That claim is node-scoped, so a module holding it
cannot share a machine with another that does — and a machine running gitea for git and packages
*alongside* a registry serving artifacts is an ordinary arrangement, not a conflict. They are
different ports doing different jobs.
The exclusivity that matters is mesh-wide and is already expressed: `provides` at mesh scope means
two providers are two answers, and the resolver refuses until one is assigned. Forbidding
co-residence adds nothing to that and forbids something reasonable. **Whether `registry` should
still hold that claim is left open here** — it may be protecting something about the port or the
data directory that is not written down, and removing a claim is not a thing to do from the outside
of a manifest.
A mesh that assigns gitea gets authentication and TLS for its artifacts — which is to say, **issues
042 and 048 are answered by choosing a provider that already solved them**, rather than by
reimplementing accounts and certificates in a registry that has none.
**Gitea provides `package-registry`.** That is ADR 0014's private registry, and it is one service
rather than one per ecosystem. `verdaccio` may provide it too, for npm alone, and is then a choice
somebody makes rather than the answer.
**The registry is not removed at the end of installing.** A mesh that never runs gitea still has an
artifact store. Retiring it is a migration a mesh performs, not a step an installation ends with.
## Why not simply gitea, from the start
Because genesis would need a control plane before the thing that stores the control plane's image,
and that is circular rather than merely awkward. It would also make one of the three things the
build loop cannot produce for itself into a stateful application with a database — the pivot is the
hardest part of this design already.
And it puts every artifact in the service that is also the trust anchor for everything the mesh will
ever run ([ADR 0071](0071-where-genesis-gets-its-source.md)), which records that forge serving a
cryptominer with tampered git operations. Two blast radii are better than one.
## Moving from one provider to the other is a designed act
**Not a removal.** Every image a machine runs is pinned to a digest at a named store, the control
plane's own included. Changing the provider means:
1. gitea installed, reachable, and holding an account the builder may publish with
2. every artifact mirrored
3. every declaration re-pinned, the control plane's **last**, because it is what performs the others
4. **every machine verified to have converged and to be able to pull from the new store**
5. only then the old provider unassigned, and its volume kept ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md))
Step 4 is the one that is easy to skip and the only thing between this and a mesh that cannot
restart its own control plane. A machine that reboots mid-migration pulls from a store that no
longer exists, and a local image cache hides that until exactly the moment it matters.
## Consequences
**The bootstrap is unchanged**, which is the point of keeping the small provider.
**042 and 048 gain a second answer.** They can be fixed in the registry, or dissolved by choosing a
provider that already has accounts and TLS. The second is less work and more service.
**ADR 0014 becomes satisfiable.** There is a provision for the private registry, something that
provides it, and a module may depend on it — so the SDK can be published and consumed rather than
cloned, and issue 053 has somewhere to go.
**A mesh can be minimal or complete, and both are legitimate.** One with the small registry and no
gitea builds and runs modules and cannot serve packages. That is a real configuration, not a broken
one — the same way a partial host is real.
**And the bootstrap still has no package registry.** The first build of the shared base happens
before anything has installed one. That is the same pivot as everything else and it is **not solved
here**: it is named, so the next person does not discover it.
## How this is checked
| Rule | Checked by |
|---|---|
| Genesis needs no database | The installer raises a mesh of one on a machine with nothing, and the artifact store it installs has no store of its own. |
| Two providers are a choice, not a conflict | A mesh holding both is asked to resolve `artifact-store` and refuses, naming both, until one is assigned. |
| Providers may share a machine | A node is assigned both gitea and a registry, and both run — only one of them answers `artifact-store`. |
| The two stores are not interchangeable | A module depending on `package-registry` is not satisfied by `artifact-store`, and the refusal says why. |
| A migration is verified before it is finished | The old provider cannot be unassigned while any machine's declaration still names it. |
@@ -0,0 +1,87 @@
---
topic: building it
status: proposed
date: 2026-09-16
deciders: jochen
reconstructed: false
extends: 0075-two-stores-and-which-provides-what.md
---
# 76. The SDK is a published package, and the toolchain resolves it by version
## Context
[ADR 0014](0014-no-npm-workspace.md) decided a module consumes its dependencies — the mesh's own
shared library included — from the private registry. [ADR 0075](0075-two-stores-and-which-provides-what.md)
decided the private registry is a `package-registry` provision, and that gitea provides it. What
neither settled, and what [issue 053](../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md)
left open, is the one build where the rule cannot simply be obeyed: **the first one.**
The TypeScript toolchain image is built *from* the SDK — it carries the SDK so that every module
compiled inside it resolves the shared library without each build fetching it. So the thing that
compiles TypeScript and the thing that contains the SDK were the same object, and that object
cannot be what builds the SDK. Stated as a question — "how does the SDK reach the registry before
the toolchain exists, when the toolchain is what builds it?" — it reads as a paradox.
It is not one. The paradox exists only because the toolchain *bakes a git-cloned copy* of the SDK.
The SDK itself is plain TypeScript: it needs `node` and `tsc` and nothing the mesh makes. A public
base image can build it. The circularity is a property of the workaround, not of the SDK.
## Decision
**The SDK is an ordinary published package in the mesh's `package-registry`, consumed by version.**
The git URL in the toolchain's manifest and the sibling-path lock beside it — the two halves of
issue 053 — are both removed. A build resolves the SDK the way it resolves any dependency, with a
lock that agrees with its manifest, so `npm ci` is the command and reproducibility is by
construction rather than by the machine the build ran on.
**The SDK is built with a public base image, not with the mesh's toolchain.** It is *not* one of
the components the loop cannot build — the control plane, the registry, the builder, the catalogue
([`12-a-module-repository`](../03-DESIGN/01-to-be/12-a-module-repository.md)), which arrive by
carrying an init builder because they are the loop's own machinery. The SDK is machinery for
nothing; it is an ordinary dependency the loop builds and publishes like any other. The only
constraint is narrow: it cannot be compiled *in the mesh toolchain*, because that toolchain is built
from it. So it is compiled on a public base image instead — which needs nothing the mesh makes — and
published before the toolchain that consumes it. It is not carried, because building it does not
wait on a mesh existing first.
**The toolchain base stays, thinned.** mesh-tools remains the image bundles are compiled in and the
one place the SDK is resolved — but it `npm ci`s the SDK by version from the registry instead of
baking a copy cloned from a git URL. Bundles keep borrowing its resolved dependencies; what changes
is that the version they borrow is named and honest. This was the shape chosen over dropping the
shared base entirely and having every bundle resolve the SDK itself: one resolution point, one
place to be right about the version.
**Genesis orders the publish before the first compile.** The package-registry provider is a public
image (gitea), so it comes up needing no toolchain; the SDK is published into it; only then is the
toolchain built, so the first `npm ci` has a registry to read from. Nothing in that chain is
circular, because the only thing that needed the toolchain — baking the SDK — is gone.
## Consequences
Each language's toolchain repeats the shape: its own SDK, built from that language's public base
image, published to the same registry, resolved by version with that ecosystem's lockfile-honest
install (`npm ci`, `cargo` against a vendored or registry source, `pip` against a pinned set). The
warning in issue 053 — that whatever the TypeScript repository does the others will copy — is
answered by making the copied thing the correct one.
A change to the SDK is publish-then-consume, exactly as [ADR 0014](0014-no-npm-workspace.md) already
priced it: publish the new SDK version, then bump the toolchain (and any module pinning it directly)
to consume it. There is no shortcut that resolves an unpublished SDK, which is the property that was
missing.
mesh-tools is no longer an SDK carrier in the sense that mattered — it does not contain a copy
whose provenance is a branch head somebody force-pushes. It contains a version.
A mesh with no package-registry cannot build TypeScript. This is accepted and is not new: it is the
same shape as a mesh that cannot reach a forge being unable to be raised
([ADR 0071](0071-where-genesis-gets-its-source.md)). Installing brings the registry up first.
## How this is checked
| Rule | Checked by |
|---|---|
| The SDK a build compiles against is named, not cloned from a branch | The toolchain manifest pins `@novox/mesh-sdk` to a version, and the build runs `npm ci`, which refuses a lock that disagrees with the manifest. Issue 053's two checks become this one. |
| The SDK builds without the mesh's own toolchain | The SDK's build recipe names a public base image. A recipe that named the mesh toolchain would reintroduce the cycle and is refused in review. |
| The registry is up before the first compile | The genesis bed asserts the package-registry answers, and the SDK is published, before the base build runs. A base build that ran first would fail its `npm ci` with no registry, which is the positive control. |
| A second language repeats the shape, not a new one | When a second SDK is added, its recipe is compared to this one: public base, publish by version, lockfile-honest install. |
+3
View File
@@ -106,6 +106,9 @@ python3 00-META/checks/index.py fail if stale
- **0071** — [Genesis clones from a mesh, and checks what it got](0071-where-genesis-gets-its-source.md) - **0071** — [Genesis clones from a mesh, and checks what it got](0071-where-genesis-gets-its-source.md)
- **0072** — [Two graphs, and a build chain that orders itself](0072-two-graphs-and-the-build-chain.md) - **0072** — [Two graphs, and a build chain that orders itself](0072-two-graphs-and-the-build-chain.md)
- **0073** — [The installer carries a builder, and the registry stays where it is](0073-the-installer-carries-a-builder.md) - **0073** — [The installer carries a builder, and the registry stays where it is](0073-the-installer-carries-a-builder.md)
- **0074** — [The mesh defines a module protocol; an SDK is an implementation of it](0074-the-wire-is-specified-not-the-types.md)
- **0075** — [An artifact store is a provision; a package registry is a different one](0075-two-stores-and-which-provides-what.md) *(proposed)*
- **0076** — [The SDK is a published package, and the toolchain resolves it by version](0076-the-sdk-is-a-published-package.md) *(proposed)*
### What runs on them, and how it gets there ### What runs on them, and how it gets there
+4 -4
View File
@@ -92,7 +92,7 @@ What needs something *usable* retries, which is what both provisioners do and is
anyway, because a dependency can restart long after everything was applied. anyway, because a dependency can restart long after everything was applied.
**The network was a real gap, and the first thing in Phase 1 that needed a decision.** Adding a **The network was a real gap, and the first thing in Phase 1 that needed a decision.** Adding a
shape widens what a compromised control plane can express, so shape widens what a compromised controller can express, so
[ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md) [ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)
records why this one is worth it: an `action` could create a network and **nothing could ever records why this one is worth it: an `action` could create a network and **nothing could ever
remove it**, because an action leaves no footprint the host can undo. The vocabulary is nine. remove it**, because an action leaves no footprint the host can undo. The vocabulary is nine.
@@ -106,7 +106,7 @@ not after.
*Done 2026-08-31. Worth recording because the task was not the one written down.* *Done 2026-08-31. Worth recording because the task was not the one written down.*
**The control plane special-cases nothing.** `provides`, `requires`, `contributes` and `grants` **The controller special-cases nothing.** `provides`, `requires`, `contributes` and `grants`
are name-agnostic — asking for a bucket needed no change to the mesh at all. What was missing was are name-agnostic — asking for a bucket needed no change to the mesh at all. What was missing was
a provider, and the last step where something on the machine turns a delivered secret into a key a provider, and the last step where something on the machine turns a delivered secret into a key
that works. So "add an object-store provision" was never mesh work. that works. So "add an object-store provision" was never mesh work.
@@ -200,7 +200,7 @@ losing something.
*2026-08-31.* **The old system's brain is switched off; its services keep running.** *2026-08-31.* **The old system's brain is switched off; its services keep running.**
Not a migration and not a period of dual control. The old control plane — provisioning, the Not a migration and not a period of dual control. The old controller — provisioning, the
coordinator, the pipeline, the things that *decide* and *write* — is stopped. Every workload it coordinator, the pipeline, the things that *decide* and *write* — is stopped. Every workload it
was managing goes on running exactly as it is, because nothing is managing it. Then the new mesh was managing goes on running exactly as it is, because nothing is managing it. Then the new mesh
takes ownership of them one at a time. takes ownership of them one at a time.
@@ -217,7 +217,7 @@ stop having opinions.
**Disabled, not merely stopped**, and this is the part that is easy to get wrong: those units are **Disabled, not merely stopped**, and this is the part that is easy to get wrong: those units are
enabled, so stopping them lasts until the machine reboots. A reboot mid-conversion would bring the enabled, so stopping them lasts until the machine reboots. A reboot mid-conversion would bring the
old control plane back and it would resume regenerating managed files underneath the new one — old controller back and it would resume regenerating managed files underneath the new one —
which is the one situation where two systems really would be fighting over the same machine. which is the one situation where two systems really would be fighting over the same machine.
**A service left running with nothing managing it is the safe state.** It has its data, its **A service left running with nothing managing it is the safe state.** It has its data, its
+4 -4
View File
@@ -40,13 +40,13 @@ not the first one built** ([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
| | **Bootstrap scenario** | **Full scenario** | | | **Bootstrap scenario** | **Full scenario** |
|---|---|---| |---|---|---|
| Contains | virtual machines, the host binary, a pinned substrate bundle | a complete mesh: forge, coordinator, delivery, modules | | Contains | virtual machines, the host binary, a pinned foundation bundle | a complete mesh: forge, coordinator, delivery, modules |
| Verdict from | what the host reports about the state it reconciled | a pipeline result ending in verify | | Verdict from | what the host reports about the state it reconciled | a pipeline result ending in verify |
| Exercises | the node host and the substrate | the control plane and everything above it | | Exercises | the node host and the foundation | the controller and everything above it |
| Exists to | **develop the mesh** | **test what runs on it** | | Exists to | **develop the mesh** | **test what runs on it** |
The bootstrap scenario is a **strict subset**: same virtualisation, same networking, same The bootstrap scenario is a **strict subset**: same virtualisation, same networking, same
lifecycle — it simply stops before a control plane exists. Everything from *"Where this sits in lifecycle — it simply stops before a controller exists. Everything from *"Where this sits in
the way work happens"* onward describes the full scenario, and applies once there is a the way work happens"* onward describes the full scenario, and applies once there is a
coordinator to describe. coordinator to describe.
@@ -343,7 +343,7 @@ from the existing system has run against any of it yet.
--- ---
## The substrate ## The foundation
### A node is a system container ### A node is a system container
@@ -205,7 +205,7 @@ machines:
place: place:
all: [host] all: [host]
anchor: [substrate] anchor: [foundation]
snapshot: raised snapshot: raised
``` ```
@@ -334,12 +334,12 @@ than a fork.
# bootstrap — tiers 0 and 1 # bootstrap — tiers 0 and 1
place: place:
all: [host] all: [host]
anchor: [substrate] anchor: [foundation]
# full — adds a control plane, a forge, and a module under test # full — adds a controller, a forge, and a module under test
place: place:
all: [host] all: [host]
anchor: [substrate, control, forge] anchor: [foundation, control, forge]
module: a-web-service module: a-web-service
assert: assert:
- the service answers on its published name - the service answers on its published name
@@ -524,7 +524,7 @@ machines:
place: place:
all: [host] all: [host]
anchor: [substrate] anchor: [foundation]
snapshot: raised snapshot: raised
``` ```
+18 -18
View File
@@ -47,10 +47,10 @@ mesh database, and it has no listening surface.
|---|---| |---|---|
| `apply` | reconciling declared state on this machine | | `apply` | reconciling declared state on this machine |
| `store` | local state, authoritative while disconnected | | `store` | local state, authoritative while disconnected |
| `link` | the single outbound connection to the control plane | | `link` | the single outbound connection to the controller |
| `profile` | what this machine can be asked to do | | `profile` | what this machine can be asked to do |
| `inventory` | what this machine is and has | | `inventory` | what this machine is and has |
| `substrate.lock` | the pinned tier-1 descriptor, appliable with no mesh present | | `foundation.lock` | the pinned tier-1 descriptor, appliable with no mesh present |
### apply ### apply
@@ -74,7 +74,7 @@ the machine in whatever state it reached, and nothing must claim otherwise.
### store ### store
Local, and **authoritative while disconnected**. Not a cache of the control plane — the record Local, and **authoritative while disconnected**. Not a cache of the controller — the record
of what this node has applied and what it currently holds. of what this node has applied and what it currently holds.
This is structural rather than convenient: if disconnection is an ordinary situation rather This is structural rather than convenient: if disconnection is an ordinary situation rather
@@ -84,7 +84,7 @@ not come back and ask what it is.
### link ### link
The node's one connection to the control plane, and its security boundary The node's one connection to the controller, and its security boundary
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
It is the broker connection that already exists It is the broker connection that already exists
@@ -137,11 +137,11 @@ One behaviour, two sources
| Situation | Source | | Situation | Source |
|---|---| |---|---|
| no mesh reachable | `substrate.lock` — the pinned bundle the host carries | | no mesh reachable | `foundation.lock` — the pinned bundle the host carries |
| mesh reachable | the control plane, over the link | | mesh reachable | the controller, over the link |
**The first node is not a different kind of node.** It is a node whose mesh is not up yet. It **The first node is not a different kind of node.** It is a node whose mesh is not up yet. It
applies the bundle it carries, the control plane comes up on top of it, and from that moment it applies the bundle it carries, the controller comes up on top of it, and from that moment it
takes declarations like every other node. Its specialness is temporary and self-erasing. takes declarations like every other node. Its specialness is temporary and self-erasing.
**A joining node does the minimum to be reachable and nothing else** — an identity, an address, **A joining node does the minimum to be reachable and nothing else** — an identity, an address,
@@ -155,7 +155,7 @@ Settled by [ADR 0005](../../02-DECISIONS/0005-the-node-host.md).
**JSON**, because the host has no dependencies to spend and the standard library carries no **JSON**, because the host has no dependencies to spend and the standard library carries no
YAML. **An ordered list of typed resources**, each with a stable identity — the order is stated YAML. **An ordered list of typed resources**, each with a stable identity — the order is stated
rather than derived, because deriving it would be the host deciding the thing most likely to rather than derived, because deriving it would be the host deciding the thing most likely to
differ from what the control plane intended. differ from what the controller intended.
**Unknown is refused, never skipped.** An unknown version, type or field refuses the whole **Unknown is refused, never skipped.** An unknown version, type or field refuses the whole
declaration. A host that skipped what it did not understand would apply most of it and report declaration. A host that skipped what it did not understand would apply most of it and report
@@ -173,16 +173,16 @@ without one, applying the bundle it carries, has nothing to check against.
Staged so each stage is verifiable in the lab before the next exists. Staged so each stage is verifiable in the lab before the next exists.
**1 — profile and inventory.** The host runs on a machine, detects what it can do, and reports **1 — profile and inventory.** The host runs on a machine, detects what it can do, and reports
what it is. No control plane, no declarations, no network. Verifiable immediately: the lab's what it is. No controller, no declarations, no network. Verifiable immediately: the lab's
`place:` gains its first implementation, and a raised scenario finally contains something. `place:` gains its first implementation, and a raised scenario finally contains something.
**2 — apply, from the bundle.** The host applies `substrate.lock` with no mesh present. This is **2 — apply, from the bundle.** The host applies `foundation.lock` with no mesh present. This is
the first node's path, and it is the claim the skeleton's Move 1 rests on and has never proved: the first node's path, and it is the claim the skeleton's Move 1 rests on and has never proved:
that one host can raise the substrate alone. that one host can raise the foundation alone.
Raising the substrate uses **four** shapes — `package`, `container`, `service`, `action` — Raising the foundation uses **four** shapes — `package`, `container`, `service`, `action` —
counted from the bundle that exists rather than reasoned about. `file` and `directory` are listed counted from the bundle that exists rather than reasoned about. `file` and `directory` are listed
below because they are the cheapest to be sure of and a substrate that needed them would find them below because they are the cheapest to be sure of and a foundation that needed them would find them
ready; the current bundle simply does not. **All of them are built:** ready; the current bundle simply does not. **All of them are built:**
| | | | | | | |
@@ -225,7 +225,7 @@ container runtime. All three were instead verified against a real machine — a
labelled, replaced when its declaration changed, exec'd into and removed; an action that exits labelled, replaced when its declaration changed, exec'd into and removed; an action that exits
zero and satisfies nothing failing the apply. That is lab-installation work zero and satisfies nothing failing the apply. That is lab-installation work
([`04-lab-installation.md`](04-lab-installation.md)) rather than a constraint on the design, but ([`04-lab-installation.md`](04-lab-installation.md)) rather than a constraint on the design, but
until it is done the substrate bootstrap has no end-to-end test. until it is done the foundation bootstrap has no end-to-end test.
**3 — link and store.** The node connects, receives declarations, and holds what it applied. **3 — link and store.** The node connects, receives declarations, and holds what it applied.
@@ -313,7 +313,7 @@ reported to be distinguishable from one that reported an empty list.*
## Open ## Open
- **Whether one host can raise the substrate alone.** Move 1 assumes it. Stage 2 tests it, and - **Whether one host can raise the foundation alone.** Move 1 assumes it. Stage 2 tests it, and
if it is false the tier boundary moves. if it is false the tier boundary moves.
- **What the host carries versus what it finds.** It manages `wg`, `nft`, `pacman`, `docker`; - **What the host carries versus what it finds.** It manages `wg`, `nft`, `pacman`, `docker`;
it does not contain them, and how it obtains one it lacks is undecided — it does not contain them, and how it obtains one it lacks is undecided —
@@ -329,15 +329,15 @@ reported to be distinguishable from one that reported an empty list.*
## What was added to the vocabulary, and why each cost was worth paying ## What was added to the vocabulary, and why each cost was worth paying
*Written 2026-08-30. Every addition widens what a compromised control plane can express, so the *Written 2026-08-30. Every addition widens what a compromised controller can express, so the
count is asserted by a test and a change to it is a decision rather than a convenience.* count is asserted by a test and a change to it is a decision rather than a convenience.*
Four shapes raise the substrate. Five more exist because most of what a person installs is not a Four shapes raise the foundation. Five more exist because most of what a person installs is not a
service: service:
| | why | | | why |
|---|---| |---|---|
| **file**, **directory** | the substrate needs neither, and almost everything else does | | **file**, **directory** | the foundation needs neither, and almost everything else does |
| **user** | a shell, a terminal, a chat client, a desktop are a package plus configuration **in somebody's home**. A mesh with no user owns `/etc` and nothing anybody looks at | | **user** | a shell, a terminal, a chat client, a desktop are a package plus configuration **in somebody's home**. A mesh with no user owns `/etc` and nothing anybody looks at |
| **archive** | a theme is hundreds of files. Inlining them makes every declaration enormous and rewrites all of them when one changes | | **archive** | a theme is hundreds of files. Inlining them makes every declaration enormous and rewrites all of them when one changes |
| **network** | a module of several containers has to let them reach each other by name, and doing it with an action would create something nothing could ever remove ([ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)) | | **network** | a module of several containers has to let them reach each other by name, and doing it with an action would create something nothing could ever remove ([ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)) |
@@ -12,7 +12,7 @@ decisions:
- 02-DECISIONS/0019-how-this-repository-works.md - 02-DECISIONS/0019-how-this-repository-works.md
--- ---
# The control plane # The controller
Tier 2. The term appears seventy-nine times across this repository and was defined nowhere, Tier 2. The term appears seventy-nine times across this repository and was defined nowhere,
which is `how-we-build` §5 failing on this repository's own vocabulary. which is `how-we-build` §5 failing on this repository's own vocabulary.
@@ -22,7 +22,7 @@ This document defines it. It does **not** design the contexts inside it; those a
## The definition ## The definition
> **The control plane is everything that needs to know about more than one node.** > **The controller is everything that needs to know about more than one node.**
That is the whole test, and it is not arbitrary — it follows from That is the whole test, and it is not arbitrary — it follows from
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). The host applies and [ADR 0005](../../02-DECISIONS/0005-the-node-host.md). The host applies and
@@ -32,11 +32,11 @@ exactly there:
| Question | Whose | | Question | Whose |
|---|---| |---|---|
| write this file, with this content, with this mode | the **host** — one machine | | write this file, with this content, with this mode | the **host** — one machine |
| which nodes should run the store | the **control plane** — needs every node | | which nodes should run the store | the **controller** — needs every node |
| is this unit running | the **host** — one machine | | is this unit running | the **host** — one machine |
| which peers belong in this node's overlay | the **control plane** — needs every node | | which peers belong in this node's overlay | the **controller** — needs every node |
| what does this machine have installed | the **host** reports; the control plane **records** | | what does this machine have installed | the **host** reports; the controller **records** |
| has this node been unreachable for a week | the **control plane** — nobody else is watching | | has this node been unreachable for a week | the **controller** — nobody else is watching |
A useful consequence: **anything a single machine could answer alone is not the control A useful consequence: **anything a single machine could answer alone is not the control
plane's.** If it needs no second node, putting it here is a mistake, and the tier rule will not plane's.** If it needs no second node, putting it here is a mistake, and the tier rule will not
@@ -67,7 +67,7 @@ something infrastructure*. `ai` is folded into `config`: a provider licence is a
**`record` is an open question rather than an eighth entry.** Contexts integrate through it **`record` is an open question rather than an eighth entry.** Contexts integrate through it
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), which makes it load-bearing, ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), which makes it load-bearing,
and [research 006](../../01-RESEARCH/006-mesh-from-scratch/skeleton.md) leaves *where it lives* and [research 006](../../01-RESEARCH/006-mesh-from-scratch/skeleton.md) leaves *where it lives*
unresolved — putting it in the substrate risks recreating the circularity the tier design just unresolved — putting it in the foundation risks recreating the circularity the tier design just
removed. Listing it here would settle by naming what has not been settled by arguing. removed. Listing it here would settle by naming what has not been settled by arguing.
**One of the seven is built.** `inventory` owns a database of that name and holds the node records; **One of the seven is built.** `inventory` owns a database of that name and holds the node records;
@@ -89,7 +89,7 @@ in front of them.
The question this answers: **can a node write to the registry database?** No — and not "only The question this answers: **can a node write to the registry database?** No — and not "only
through one node", which is the weaker arrangement it might be mistaken for. through one node", which is the weaker arrangement it might be mistaken for.
> **No node holds a credential to any control-plane store, for writing or for reading.** > **No node holds a credential to any controller store, for writing or for reading.**
That is not a new rule here; it is four already taken, and it is worth seeing them together That is not a new rule here; it is four already taken, and it is worth seeing them together
because each one alone reads like a detail: because each one alone reads like a detail:
@@ -118,11 +118,11 @@ Reads work the same way in reverse — a node is *told*, in declarations. It nev
### Who actually consumes, and who writes ### Who actually consumes, and who writes
**The control plane is the consumer. There is one of it, and the context that owns the data does **The controller is the consumer. There is one of it, and the context that owns the data does
the write.** the write.**
``` ```
node ──► broker ──► the control plane, consuming node ──► broker ──► the controller, consuming
├─ a node reported what it applied ─► inventory writes the registry ├─ a node reported what it applied ─► inventory writes the registry
├─ a node reported health ─► observability writes its own store ├─ a node reported health ─► observability writes its own store
└─ a grant was requested ─► provisioning writes its own store └─ a grant was requested ─► provisioning writes its own store
@@ -139,9 +139,9 @@ the store it exclusively owns
receiving half of what it expects* — which has happened, between a module's daemon and its receiving half of what it expects* — which has happened, between a module's daemon and its
capability server. With one consumer that class of fault cannot arise. capability server. With one consumer that class of fault cannot arise.
**And the broker is the buffer while the control plane is down.** Nodes go on publishing; **And the broker is the buffer while the controller is down.** Nodes go on publishing;
messages queue; the control plane drains them when it returns. That is what makes messages queue; the controller drains them when it returns. That is what makes
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)'s single control plane [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)'s single controller
tolerable — an outage delays the mesh's *knowledge* rather than losing it. tolerable — an outage delays the mesh's *knowledge* rather than losing it.
**With one consequence that must be bounded before it is discovered:** a queue with no limit **With one consequence that must be bounded before it is discovered:** a queue with no limit
@@ -177,7 +177,7 @@ volume genuinely argues against a relational store.
node except through the host. node except through the host.
- **Not a surface.** Tier 3 is how people and agents reach it. It has one interface; the - **Not a surface.** Tier 3 is how people and agents reach it. It has one interface; the
surfaces are what speak to that interface. surfaces are what speak to that interface.
- **Not the substrate.** It *runs on* tier 1 — PostgreSQL, LavinMQ, an OCI registry - **Not the foundation.** It *runs on* tier 1 — PostgreSQL, LavinMQ, an OCI registry
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) — and cannot start without ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) — and cannot start without
them, which is what makes them a lower tier. them, which is what makes them a lower tier.
- **Not privileged on a node.** It has no more access to a machine than the declaration - **Not privileged on a node.** It has no more access to a machine than the declaration
@@ -185,19 +185,19 @@ volume genuinely argues against a relational store.
## It is also a consumer ## It is also a consumer
The property that makes tier 2 unlike the others: **the control plane has requirements of its The property that makes tier 2 unlike the others: **the controller has requirements of its
own.** It needs a PostgreSQL database, an AMQP virtual host, and a bucket — the same things any own.** It needs a PostgreSQL database, an AMQP virtual host, and a bucket — the same things any
module needs, granted the same way. module needs, granted the same way.
That is the circularity the tiers exist to resolve rather than hide: the control plane cannot That is the circularity the tiers exist to resolve rather than hide: the controller cannot
provision its own database, because it is not running yet. So its **store** is raised from the provision its own database, because it is not running yet. So its **store** is raised from the
bundle the host carries, before there is a control plane to ask bundle the host carries, before there is a controller to ask
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md),
[research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md)). [research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md)).
Its virtual host and its bucket are **not** in the bundle — by the time they are wanted there is Its virtual host and its bucket are **not** in the bundle — by the time they are wanted there is
a control plane to grant them. Whether the bus must come first is a controller to grant them. Whether the bus must come first is
[open](07-the-substrate.md#open), and it turns on whether these contexts talk to each other over [open](07-the-foundation.md#open), and it turns on whether these contexts talk to each other over
it. it.
## Where it runs ## Where it runs
@@ -209,10 +209,10 @@ hosts, assigned to nodes by the same mechanism as everything else.
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The node is assigned, ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The node is assigned,
never elected — no promotion, no quorum, no split brain. never elected — no promotion, no quorum, no split brain.
That is sound rather than merely cheap, because the design already tolerates the control plane That is sound rather than merely cheap, because the design already tolerates the controller
being absent by construction: a node reconciles from **its own** store being absent by construction: a node reconciles from **its own** store
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) and ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) and
never needed to ask anybody to hold the state it was last given. So the control plane being down never needed to ask anybody to hold the state it was last given. So the controller being down
is not a new failure mode — it is is not a new failure mode — it is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s ordinary disconnected [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s ordinary disconnected
situation, happening to every node at once. **What is lost is change, not operation.** situation, happening to every node at once. **What is lost is change, not operation.**
@@ -226,13 +226,13 @@ every public name.
- ~~**The contexts themselves.**~~ **Decided** — seven, by - ~~**The contexts themselves.**~~ **Decided** — seven, by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
What remains open is narrower and named there: **where the record lives**, which research 006 What remains open is narrower and named there: **where the record lives**, which research 006
leaves unresolved because the substrate is the one place it must not go. leaves unresolved because the foundation is the one place it must not go.
- **How far it may be split.** One deployable today. Splitting a context out costs the single - **How far it may be split.** One deployable today. Splitting a context out costs the single
interface a surface depends on interface a surface depends on
([research 011](../../01-RESEARCH/011-the-module-graph/00-overview.md)). ([research 011](../../01-RESEARCH/011-the-module-graph/00-overview.md)).
- ~~**How many run, and what a node does without one.**~~ **Resolved** by - ~~**How many run, and what a node does without one.**~~ **Resolved** by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). What remains is [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). What remains is
measurement: nothing reports how long the control plane has been unreachable, or how close a measurement: nothing reports how long the controller has been unreachable, or how close a
certificate is to expiry — both needed for restore-not-failover to be a plan rather than a certificate is to expiry — both needed for restore-not-failover to be a plan rather than a
hope. hope.
- **What the interface is.** One interface is stated; its shape, and whether it is request, - **What the interface is.** One interface is stated; its shape, and whether it is request,
@@ -2,7 +2,7 @@
layer: to-be layer: to-be
status: in-progress status: in-progress
code: code:
- mesh-host examples/substrate-first-node.lock - mesh-host examples/foundation-first-node.lock
- mesh-host internal/apply - mesh-host internal/apply
- mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh) - mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)
updated: 2026-08-31 updated: 2026-08-31
@@ -15,16 +15,16 @@ decisions:
- 02-DECISIONS/0019-how-this-repository-works.md - 02-DECISIONS/0019-how-this-repository-works.md
--- ---
# The substrate # The foundation
Tier 1. Defined the same way [the control plane](06-the-control-plane.md) is, because the same Tier 1. Defined the same way [the controller](06-the-controller.md) is, because the same
gap applied: the word was load-bearing and unpinned. gap applied: the word was load-bearing and unpinned.
## The definition ## The definition
> **The substrate is what the control plane consumes and cannot grant itself.** > **The foundation is what the controller consumes and cannot grant itself.**
Every module that needs a database asks the control plane's provisioning for one. The control Every module that needs a database asks the controller's provisioning for one. The control
plane needs a database too — and it cannot ask itself, because it is not running yet. That plane needs a database too — and it cannot ask itself, because it is not running yet. That
circularity is not an awkwardness to work around; it *is* the definition. Anything on the wrong circularity is not an awkwardness to work around; it *is* the definition. Anything on the wrong
side of it must be raised some other way, and the other way is the bundle the host carries side of it must be raised some other way, and the other way is the bundle the host carries
@@ -32,19 +32,19 @@ side of it must be raised some other way, and the other way is the bundle the ho
The test, applied: The test, applied:
| | control plane needs it | can it grant itself one? | | | | controller needs it | can it grant itself one? | |
|---|---|---|---| |---|---|---|---|
| a relational store — **PostgreSQL** | its own state lives there | no — provisioning needs the store | **substrate** | | a relational store — **PostgreSQL** | its own state lives there | no — provisioning needs the store | **foundation** |
| a message bus — **LavinMQ** | it reaches nodes over it ([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)) | no — it cannot grant itself a virtual host | **substrate** | | a message bus — **LavinMQ** | it reaches nodes over it ([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)) | no — it cannot grant itself a virtual host | **foundation** |
| ~~an object store~~ | ~~artifacts and blobs it delivers~~ | — | **not substrate** — [ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) | | ~~an object store~~ | ~~artifacts and blobs it delivers~~ | — | **not foundation** — [ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) |
| ~~an image registry~~ | ~~images it delivers to nodes~~ | — | **not substrate** — needed to operate, not to start ([ADR 0033](../../02-DECISIONS/0033-the-substrate-is-a-store-and-a-broker.md)) | | ~~an image registry~~ | ~~images it delivers to nodes~~ | — | **not foundation** — needed to operate, not to start ([ADR 0033](../../02-DECISIONS/0033-the-substrate-is-a-store-and-a-broker.md)) |
| ~~an identity provider~~ | ~~only if it delegates authentication~~ | — | **not substrate** — it delegates to nothing ([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)) | | ~~an identity provider~~ | ~~only if it delegates authentication~~ | — | **not foundation** — it delegates to nothing ([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)) |
| ingress — **Traefik** | not to start; only to be reached by name | — it grants itself one afterwards | **not substrate** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | | ingress — **Traefik** | not to start; only to be reached by name | — it grants itself one afterwards | **not foundation** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) |
| anything else the mesh hosts | no | — | not substrate | | anything else the mesh hosts | no | — | not foundation |
*The object-store row was wrong, and how it was wrong is worth keeping.* It answered *can it *The object-store row was wrong, and how it was wrong is worth keeping.* It answered *can it
grant itself one* — no, it cannot grant itself a bucket — while assuming the first column. **The grant itself one* — no, it cannot grant itself a bucket — while assuming the first column. **The
control plane does not need an object store**: it has no S3 client and never has, and artifacts controller does not need an object store**: it has no S3 client and never has, and artifacts
reach nodes as content-addressed blobs in the registry. The row was inherited from the system being reach nodes as content-addressed blobs in the registry. The row was inherited from the system being
replaced, where an object store distributed module tarballs, and was never re-tested against the replaced, where an object store distributed module tarballs, and was never re-tested against the
definition above it. *Both columns must be answered, and the second is true of almost any service.* definition above it. *Both columns must be answered, and the second is true of almost any service.*
@@ -60,76 +60,76 @@ does not record that the choice was ever made.
The dependency is on the **protocol**, not the product: AMQP for the bus, S3 for the object The dependency is on the **protocol**, not the product: AMQP for the bus, S3 for the object
store, the OCI protocol for the registry. That is what keeps the naming safe rather than a store, the OCI protocol for the registry. That is what keeps the naming safe rather than a
commitment that cannot be revisited — replacing one is a substrate migration, not a redesign. commitment that cannot be revisited — replacing one is a foundation migration, not a redesign.
The store is the exception, and the exception matters: the provisioning model uses databases, The store is the exception, and the exception matters: the provisioning model uses databases,
roles and schemas as PostgreSQL means them, so it is the one member that is not a swap. roles and schemas as PostgreSQL means them, so it is the one member that is not a swap.
## What that resolves ## What that resolves
**Four or five?** [Research 006](../../01-RESEARCH/006-mesh-from-scratch/00-overview.md) asks **Four or five?** [Research 006](../../01-RESEARCH/006-mesh-from-scratch/00-overview.md) asks
whether the identity provider is a substrate service, and the test answers it *conditionally* — whether the identity provider is a foundation service, and the test answers it *conditionally* —
which is the honest answer rather than a number. which is the honest answer rather than a number.
- If the control plane **delegates** authentication, it cannot serve anybody before the provider - If the controller **delegates** authentication, it cannot serve anybody before the provider
exists, and it cannot grant itself a client. **Substrate.** exists, and it cannot grant itself a client. **Foundation.**
- If it **authenticates natively**, the provider is an ordinary hosted service like any other. - If it **authenticates natively**, the provider is an ordinary hosted service like any other.
**Not substrate.** **Not foundation.**
So the count follows from a design decision that has not been taken, and the record should say So the count follows from a design decision that has not been taken, and the record should say
that rather than assert four. that rather than assert four.
**Why not "important infrastructure".** An identity provider, a mail server and an analytics **Why not "important infrastructure".** An identity provider, a mail server and an analytics
service are all infrastructure by any ordinary reading, and none of them are substrate — the service are all infrastructure by any ordinary reading, and none of them are foundation — the
control plane starts and runs without them. *Important* is not the test; *the control plane controller starts and runs without them. *Important* is not the test; *the controller
cannot obtain it* is. cannot obtain it* is.
## What the substrate is not ## What the foundation is not
- **Not tier 0.** The host raises the substrate; it is not part of it. The host carries the - **Not tier 0.** The host raises the foundation; it is not part of it. The host carries the
declaration that brings the substrate up, and depends on nothing. declaration that brings the foundation up, and depends on nothing.
- **Not the control plane.** These are services with no knowledge of the mesh. A store does not - **Not the controller.** These are services with no knowledge of the mesh. A store does not
know what a node is. know what a node is.
- **Not a place for logic.** The skeleton is explicit: tier 1 is *declarations only, no logic of - **Not a place for logic.** The skeleton is explicit: tier 1 is *declarations only, no logic of
its own.* A substrate service is an upstream image, pinned, with configuration. its own.* A foundation service is an upstream image, pinned, with configuration.
- **Not privileged.** The substrate is provisioned *from* by the control plane and grants - **Not privileged.** The foundation is provisioned *from* by the controller and grants
nothing on its own initiative. nothing on its own initiative.
- **Not the mesh's supply of anything** - **Not the mesh's supply of anything**
([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)). ([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)).
A substrate service and a module of the same product are **different instances**. The mesh's own A foundation service and a module of the same product are **different instances**. The mesh's own
PostgreSQL and a PostgreSQL a workload was given are two servers, and a node hosting both runs PostgreSQL and a PostgreSQL a workload was given are two servers, and a node hosting both runs
two containers — expected, not duplication to be tidied away. two containers — expected, not duplication to be tidied away.
The substrate is raised from the bundle before any mesh exists, so **it is not in the module The foundation is raised from the bundle before any mesh exists, so **it is not in the module
graph**: a workload depending on it would depend on something the graph cannot see, cannot rotate graph**: a workload depending on it would depend on something the graph cannot see, cannot rotate
a credential for, and cannot move. It would also put workload data in the store the control plane a credential for, and cannot move. It would also put workload data in the store the controller
keeps its own state in, where a workload that fills a disk takes down the one thing needed to fix keeps its own state in, where a workload that fills a disk takes down the one thing needed to fix
it. it.
## The pinned bundle ## The pinned bundle
`substrate.lock` holds **what must exist before the control plane runs** — which is a smaller `foundation.lock` holds **what must exist before the controller runs** — which is a smaller
set than the substrate, and the difference is easy to miss. It is the only place in the mesh set than the foundation, and the difference is easy to miss. It is the only place in the mesh
where versions are pinned by hand rather than resolved. where versions are pinned by hand rather than resolved.
Being substrate and being in the bundle are two different questions: Being foundation and being in the bundle are two different questions:
| | is it substrate? | must it precede the control plane? | | | is it foundation? | must it precede the controller? |
|---|---|---| |---|---|---|
| PostgreSQL | yes — the control plane's own state lives in it | **yes** — there is nowhere to put that state otherwise | | PostgreSQL | yes — the controller's own state lives in it | **yes** — there is nowhere to put that state otherwise |
| LavinMQ | yes — it cannot grant itself a virtual host | **yes** — the control plane reaches a node only over the link, and the link is the broker ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) | | LavinMQ | yes — it cannot grant itself a virtual host | **yes** — the controller reaches a node only over the link, and the link is the broker ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
| the OCI registry | yes — it cannot grant itself a repository | no — the first node fetches upstream ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) | | the OCI registry | yes — it cannot grant itself a repository | no — the first node fetches upstream ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
The registry is **substrate by role and ordinary by delivery**: by the time it is wanted there is The registry is **foundation by role and ordinary by delivery**: by the time it is wanted there is
a control plane, and it provisions it the way it provisions anything. That keeps the bundle small a controller, and it provisions it the way it provisions anything. That keeps the bundle small
enough for the review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) requires — one enough for the review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) requires — one
substrate image until foundation image until
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) established that the [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) established that the
broker has to precede the control plane, and two since. broker has to precede the controller, and two since.
*Corrected 2026-08-31, from counting what the bundle holds rather than reasoning about it.* **It *Corrected 2026-08-31, from counting what the bundle holds rather than reasoning about it.* **It
carries three images, not two** — PostgreSQL, LavinMQ, and the control plane itself, which the carries three images, not two** — PostgreSQL, LavinMQ, and the controller itself, which the
sentence above had overlooked by counting only substrate services. The control plane is what the sentence above had overlooked by counting only foundation services. The controller is what the
substrate exists to start, and it is in the bundle for the same reason they are: there is nothing foundation exists to start, and it is in the bundle for the same reason they are: there is nothing
to fetch it with yet. It also carries seven actions, a package and a service. to fetch it with yet. It also carries seven actions, a package and a service.
**Why pinned:** the bundle is applied when no mesh exists, so nothing can resolve a version, ask **Why pinned:** the bundle is applied when no mesh exists, so nothing can resolve a version, ask
@@ -154,13 +154,13 @@ The order, from [research 011](../../01-RESEARCH/011-the-module-graph/worked-pro
4 LavinMQ runs pulled by digest, from the bundle 4 LavinMQ runs pulled by digest, from the bundle
5 a virtual host, a credential, and actions, run locally 5 a virtual host, a credential, and actions, run locally
a self-signed certificate a self-signed certificate
6 the control plane starts and only now is there a mesh 6 the controller starts and only now is there a mesh
7 the registry, and everything else the ordinary path 7 the registry, and everything else the ordinary path
are provisioned are provisioned
``` ```
**Steps 4 and 5 are why the bundle is not one image** **Steps 4 and 5 are why the bundle is not one image**
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The control plane cannot ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The controller cannot
provision the broker, because provisioning means telling a host, and telling a host happens over provision the broker, because provisioning means telling a host, and telling a host happens over
the broker. The first node does not escape this by being local: it enrols the ordinary way, by the broker. The first node does not escape this by being local: it enrols the ordinary way, by
dialling the broker at the address in its token. dialling the broker at the address in its token.
@@ -172,15 +172,15 @@ database* names a thing that will not exist
database is a boundary a cross-context join cannot casually cross where a separate schema is not. database is a boundary a cross-context join cannot casually cross where a separate schema is not.
Only PostgreSQL is raised from the bundle, for the reason in *The pinned bundle* above — the Only PostgreSQL is raised from the bundle, for the reason in *The pinned bundle* above — the
rest of the substrate is wanted only once there is a control plane to provision it. rest of the foundation is wanted only once there is a controller to provision it.
**Step 0 is easy to leave out and it is where several things meet.** A substrate service is a **Step 0 is easy to leave out and it is where several things meet.** A foundation service is a
container, so a container runtime must be working before anything else happens — and a runtime container, so a container runtime must be working before anything else happens — and a runtime
is a *package*, not a container. is a *package*, not a container.
**Which runtime is detected, not chosen** **Which runtime is detected, not chosen**
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)): a machine that ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)): a machine that
already has one keeps it. On a machine with none, the control plane names the package, because already has one keeps it. On a machine with none, the controller names the package, because
what it is called differs per system. It is: what it is called differs per system. It is:
- what the host's capability detection already reports, and the first use of that report by - what the host's capability detection already reports, and the first use of that report by
@@ -191,7 +191,7 @@ what it is called differs per system. It is:
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
So the bootstrap uses four shapes: **package**, **container**, **service** and **action** — So the bootstrap uses four shapes: **package**, **container**, **service** and **action** —
*counted from `substrate-first-node.lock`, which is the only bundle there is*. It had said six, *counted from `foundation-first-node.lock`, which is the only bundle there is*. It had said six,
adding `file` and `directory`, which this bootstrap never asks for. adding `file` and `directory`, which this bootstrap never asks for.
All four are built, as are the host's other five All four are built, as are the host's other five
@@ -202,7 +202,7 @@ on the host any longer — which is the claim that mattered, and it was true eit
the bootstrap rather than a service consumers use later. They are **actions** the bundle the bootstrap rather than a service consumers use later. They are **actions** the bundle
declares and the host runs declares and the host runs
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) — so the ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) — so the
host's vocabulary grows by one shape rather than by one resource type per substrate service. host's vocabulary grows by one shape rather than by one resource type per foundation service.
## Open ## Open
@@ -210,12 +210,12 @@ host's vocabulary grows by one shape rather than by one resource type per substr
[ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md): the control [ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md): the control
plane delegates authentication to nothing, so identity is an ordinary module. With the object plane delegates authentication to nothing, so identity is an ordinary module. With the object
store gone ([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)) store gone ([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md))
the substrate is three, and no member is conditional. the foundation is three, and no member is conditional.
- ~~**Whether the bus must precede the control plane.**~~ **Resolved** by - ~~**Whether the bus must precede the controller.**~~ **Resolved** by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) — it must, and the question as [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) — it must, and the question as
posed here could not have answered it. This asked whether the control plane's contexts talk to posed here could not have answered it. This asked whether the controller's contexts talk to
each other over the bus; they do not, being one process, which under this framing would have each other over the bus; they do not, being one process, which under this framing would have
kept LavinMQ out of the bundle. What decides it is how the control plane reaches a *node*, which kept LavinMQ out of the bundle. What decides it is how the controller reaches a *node*, which
is only ever over the link. is only ever over the link.
- **What issues the broker's certificate at bootstrap.** New, and created by the row above. A - **What issues the broker's certificate at bootstrap.** New, and created by the row above. A
token pins the fingerprint a host must expect before it sends anything token pins the fingerprint a host must expect before it sends anything
@@ -223,7 +223,7 @@ host's vocabulary grows by one shape rather than by one resource type per substr
moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and
pinned is the shape that fits; how it is later replaced by the certificates in pinned is the shape that fits; how it is later replaced by the certificates in
[`08-connectivity.md`](08-connectivity.md) is not decided. [`08-connectivity.md`](08-connectivity.md) is not decided.
- **How a context added later gets its database.** By then there is a control plane — but one - **How a context added later gets its database.** By then there is a controller — but one
holding a credential that can create databases holds more than what it exclusively owns holding a credential that can create databases holds more than what it exclusively owns
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)). ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)).
- ~~**Whether the host can do step 2.**~~ **Resolved** by - ~~**Whether the host can do step 2.**~~ **Resolved** by
@@ -234,8 +234,8 @@ host's vocabulary grows by one shape rather than by one resource type per substr
the module that provides one. the module that provides one.
- **Whether one host can raise all three.** The claim under stage 2 of - **Whether one host can raise all three.** The claim under stage 2 of
[the node host](05-the-node-host.md), never proved. If it is false, the tier boundary moves. [the node host](05-the-node-host.md), never proved. If it is false, the tier boundary moves.
- **How the substrate is updated once a mesh exists.** Pinned by hand at bootstrap; afterwards - **How the foundation is updated once a mesh exists.** Pinned by hand at bootstrap; afterwards
the control plane could deliver it like anything else, and nothing says whether it does. the controller could deliver it like anything else, and nothing says whether it does.
## Raised, and observed ## Raised, and observed
@@ -243,7 +243,7 @@ host's vocabulary grows by one shape rather than by one resource type per substr
**It works, and what that means precisely:** a machine with a container runtime and nothing else **It works, and what that means precisely:** a machine with a container runtime and nothing else
applied the bundle its host carries and ended with a store, a database per context, those applied the bundle its host carries and ended with a store, a database per context, those
contexts' schemas, a broker holding a certificate it generated itself, and the control plane contexts' schemas, a broker holding a certificate it generated itself, and the controller
serving on top of them. Eleven resources, one command, no mesh to ask anything of. serving on top of them. Eleven resources, one command, no mesh to ask anything of.
**Then it joined itself.** The same machine took a token, checked the broker against the **Then it joined itself.** The same machine took a token, checked the broker against the
@@ -259,7 +259,7 @@ of a database and pushed to over the broker. What arrived and what did not is th
|---|---| |---|---|
| the password, in plain text | **on the machine only**, one file, mode 0600 | | the password, in plain text | **on the machine only**, one file, mode 0600 |
| in the declaration that crossed the broker | absent | | in the declaration that crossed the broker | absent |
| in the control plane's database | absent | | in the controller's database | absent |
| in what the node reported back | absent | | in what the node reported back | absent |
**One fault, and it was in the joining.** The token did not say what the mesh calls the machine, **One fault, and it was in the joining.** The token did not say what the mesh calls the machine,
+17 -17
View File
@@ -21,25 +21,25 @@ decisions:
# Connectivity # Connectivity
One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most One of [the controller's](06-the-controller.md) ten contexts, and the one with the most
moving parts: **overlay, resolution, exposure, filtering, certificates.** moving parts: **overlay, resolution, exposure, filtering, certificates.**
It is written as a whole because the five are one design. They share inputs, they must agree, and It is written as a whole because the five are one design. They share inputs, they must agree, and
every one of them today is computed in a different place by a different module from a different every one of them today is computed in a different place by a different module from a different
copy of the same facts. copy of the same facts.
## Why it is control-plane work ## Why it is controller work
Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one Apply [the test](06-the-controller.md) — *everything that needs to know about more than one
node* — to each responsibility: node* — to each responsibility:
| | needs to know | whose | | | needs to know | whose |
|---|---|---| |---|---|---|
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane | | **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | controller |
| **resolution** — which name is which node | **every node** | control plane | | **resolution** — which name is which node | **every node** | controller |
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | control plane | | **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | controller |
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies | | **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | controller decides, host applies |
| **certificates** — who may present which name | which name belongs to which node | control plane | | **certificates** — who may present which name | which name belongs to which node | controller |
**Not one of the five can be answered by a machine on its own.** That is the whole reason this is **Not one of the five can be answered by a machine on its own.** That is the whole reason this is
a context rather than a set of node-local modules — and it is exactly what the current a context rather than a set of node-local modules — and it is exactly what the current
@@ -57,7 +57,7 @@ exist; WireGuard, the resolver and the proxy are all *a container or a package,
It is also what removes the last two upward dependencies. It is also what removes the last two upward dependencies.
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules [Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and opening a direct connection to the controller's database — `wireguard` and `traefik` — and
they are the reason every node permanently holds a credential to it they are the reason every node permanently holds a credential to it
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity
modules. **Closing this context closes that set.** modules. **Closing this context closes that set.**
@@ -74,7 +74,7 @@ wanted the exception.
**What made it look unavoidable:** a peer list cannot be written in a manifest. It is derived from **What made it look unavoidable:** a peer list cannot be written in a manifest. It is derived from
every other machine, so it differs on each one and changes when any of them changes. So the every other machine, so it differs on each one and changes when any of them changes. So the
manifest says its resources are **computed** — it names something in the control plane that works manifest says its resources are **computed** — it names something in the controller that works
them out per node — and it is a module in every other respect: assigned, resolved, configured by them out per node — and it is a module in every other respect: assigned, resolved, configured by
settings, and absent from a machine nobody gave it to. settings, and absent from a machine nobody gave it to.
@@ -108,7 +108,7 @@ claim, and the collision is refused by name.
**And the proxy's half, which was the other module reaching into the database.** A web application **And the proxy's half, which was the other module reaching into the database.** A web application
requiring a reverse proxy has to say *which name, which port*, and there was nowhere to put it — requiring a reverse proxy has to say *which name, which port*, and there was nowhere to put it —
`requires` says a thing must exist and never said what to do with it. A module now contributes to `requires` says a thing must exist and never said what to do with it. A module now contributes to
a requirement, the control plane collects every contribution on a node, and the provider is given a requirement, the controller collects every contribution on a node, and the provider is given
them as a file at a path it named. It reloads when that file changes, by the same `restart-on` the them as a file at a path it named. It reloads when that file changes, by the same `restart-on` the
private network needed when a peer list changed under a running interface. private network needed when a peer list changed under a running interface.
@@ -177,7 +177,7 @@ which of those it may dial, and which must dial it.
**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the **Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
public key is published to the mesh. This is already true and it is already right — it is public key is published to the mesh. This is already true and it is already right — it is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own
identity* applied to the overlay, and it means the control plane computes a graph it cannot identity* applied to the overlay, and it means the controller computes a graph it cannot
itself impersonate. itself impersonate.
**Shape: a hub, with direct peering between co-located nodes.** **Shape: a hub, with direct peering between co-located nodes.**
@@ -217,7 +217,7 @@ files were right, the services were up, and every node reported success.
document's own warning, arriving in its implementation: *a more specific route to a dead document's own warning, arriving in its implementation: *a more specific route to a dead
endpoint blackholes; it does not fall back to the general one.* endpoint blackholes; it does not fall back to the general one.*
- **The container runtime closes the door the overlay needs.** Docker sets the FORWARD policy to - **The container runtime closes the door the overlay needs.** Docker sets the FORWARD policy to
DROP, so a hub with `ip_forward` enabled still carries nothing between its spokes. The substrate DROP, so a hub with `ip_forward` enabled still carries nothing between its spokes. The foundation
at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The
hub inserts its own rule above those chains and removes it on the way down. hub inserts its own rule above those chains and removes it on the way down.
@@ -287,7 +287,7 @@ node. What routes it once it arrives is a proxy's, and stays separate.
**The mesh writes the data and runs no daemon.** One wildcard per machine, from the same set that **The mesh writes the data and runs no daemon.** One wildcard per machine, from the same set that
writes the hosts file. A resolver is third-party software and runs *on* the mesh rather than being writes the hosts file. A resolver is third-party software and runs *on* the mesh rather than being
*of* it: the mesh has no business shipping one, choosing which one, or knowing its configuration *of* it: the mesh has no business shipping one, choosing which one, or knowing its configuration
language. Swapping dnsmasq for unbound changes that module and nothing in the control plane. language. Swapping dnsmasq for unbound changes that module and nothing in the controller.
**Two roles, two claims, because they are different things.** systemd-resolved cannot answer a **Two roles, two claims, because they are different things.** systemd-resolved cannot answer a
wildcard at all — it routes the mesh's suffix to something that can. Treating serving and asking wildcard at all — it routes the mesh's suffix to something that can. Treating serving and asking
@@ -377,7 +377,7 @@ vocabulary — the mirror of a database grant, where the consumer supplies a tar
name rather than supplying nothing and receiving credentials. name rather than supplying nothing and receiving credentials.
**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is **A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work. the case is a mesh-level fact, which is the fourth reason exposure is controller work.
### What was built ### What was built
@@ -482,7 +482,7 @@ something:
| | why not | | | why not |
|---|---| |---|---|
| decide what the machine **forwards** | the container runtime writes its own forwarding rules and a second policy is consulted alongside them, so a drop here stops every container on the node — the control plane included. Nothing in a manifest says what a machine routes, so there is nothing to derive it from either | | decide what the machine **forwards** | the container runtime writes its own forwarding rules and a second policy is consulted alongside them, so a drop here stops every container on the node — the controller included. Nothing in a manifest says what a machine routes, so there is nothing to derive it from either |
| **flush the ruleset** when loading | that empties every table on the machine, the runtime's among them. Only the mesh's own table is replaced, and it is declared empty first so the replacement works on a machine loading one for the first time | | **flush the ruleset** when loading | that empties every table on the machine, the runtime's among them. Only the mesh's own table is replaced, and it is declared empty first so the replacement works on a machine loading one for the first time |
| carry a **command to load itself** | the link may not carry an action ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A service is declared to reflect the file instead, so replacing it restarts what loads it — the shape that rule leaves, used here for the first time for its real purpose | | carry a **command to load itself** | the link may not carry an action ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A service is declared to reflect the file instead, so replacing it restarts what loads it — the shape that rule leaves, used here for the first time for its real purpose |
@@ -549,7 +549,7 @@ defaults to the public authority's *production* endpoint. Two consequences, and
worse than the lab problem that found it — every certificate experiment on a real node consumes worse than the lab problem that found it — every certificate experiment on a real node consumes
production issuance quota, and a retry loop can exhaust it for a week. production issuance quota, and a retry loop can exhaust it for a week.
**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the **The mesh CA is not a bootstrap concern.** A joining node verifies the controller against the
fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)), fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)),
so nothing needs the CA before membership. It certifies internal names afterwards, and that is so nothing needs the CA before membership. It certifies internal names afterwards, and that is
all it does. all it does.
+15 -15
View File
@@ -142,12 +142,12 @@ nox-mesh-host enrol --token <one-time token>
The token carries **four** things and is carried by a person The token carries **four** things and is carried by a person
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)): the broker's ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)): the broker's
address, the fingerprint to expect, **the control plane's signing identity**, and the right to address, the fingerprint to expect, **the controller's signing identity**, and the right to
join once. join once.
**The fourth is the one this document listed three of.** A node connects to the broker and takes **The fourth is the one this document listed three of.** A node connects to the broker and takes
instruction from the control plane behind it, and those are two different identities. Pinning only instruction from the controller behind it, and those are two different identities. Pinning only
the broker would make the control plane's authority *transitive* — a compromised broker could then the broker would make the controller's authority *transitive* — a compromised broker could then
forge declarations, which, since the host applies whatever the link delivers, is the whole machine. forge declarations, which, since the host applies whatever the link delivers, is the whole machine.
So the transport is verified once at connect, and **each declaration is verified by its signature, So the transport is verified once at connect, and **each declaration is verified by its signature,
every time**. every time**.
@@ -158,10 +158,10 @@ What happens, in order:
2. it checks the broker's certificate against the pinned fingerprint — *before* sending anything; 2. it checks the broker's certificate against the pinned fingerprint — *before* sending anything;
3. it presents the one-time secret **and its own public key**, which the mesh records; 3. it presents the one-time secret **and its own public key**, which the mesh records;
4. it reports its `profile` and `inventory` upward; 4. it reports its `profile` and `inventory` upward;
5. the control plane decides what this machine should be, and sends a declaration; 5. the controller decides what this machine should be, and sends a declaration;
6. the host applies it, reads back, and reports. 6. the host applies it, reads back, and reports.
**Step 4 is the one that is easy to miss and is what makes step 5 possible.** The control plane **Step 4 is the one that is easy to miss and is what makes step 5 possible.** The controller
cannot decide what a machine should run without knowing what it *can* run — a graphical session, cannot decide what a machine should run without knowing what it *can* run — a graphical session,
a container runtime, an architecture. The profile is not a diagnostic; it is the input. a container runtime, an architecture. The profile is not a diagnostic; it is the input.
@@ -209,7 +209,7 @@ channel, not about network reachability.** What it forbids is a listening thing
instructions and changes the machine. A node being reachable on the overlay — the whole purpose instructions and changes the machine. A node being reachable on the overlay — the whole purpose
of the overlay — is untouched by it, and so is a person opening a shell on it. of the overlay — is untouched by it, and so is a person opening a shell on it.
The distinction is *who can tell this machine what to be*: only the control plane, only over the The distinction is *who can tell this machine what to be*: only the controller, only over the
link the node opened, only in declarations of known shape. link the node opened, only in declarations of known shape.
--- ---
@@ -219,18 +219,18 @@ link the node opened, only in declarations of known shape.
The same path, with the mesh built in the middle of it. The same path, with the mesh built in the middle of it.
``` ```
# 1 — raise the substrate and the control plane from the carried bundle # 1 — raise the foundation and the controller from the carried bundle
nox-mesh-host reconcile nox-mesh-host reconcile
# 2 — the control plane now exists, and issues the first token # 2 — the controller now exists, and issues the first token
mesh-control token issue mesh-control token issue
# 3 — the machine joins the mesh it just raised # 3 — the machine joins the mesh it just raised
nox-mesh-host enrol --token <token> nox-mesh-host enrol --token <token>
``` ```
Step 1 is the bootstrap from [`07-the-substrate.md`](07-the-substrate.md): a container runtime, Step 1 is the bootstrap from [`07-the-foundation.md`](07-the-foundation.md): a container runtime,
then PostgreSQL, then the database, then the schema, then the control plane. It needs no identity then PostgreSQL, then the database, then the schema, then the controller. It needs no identity
because nothing is being asked of anyone — the host is applying a declaration it already because nothing is being asked of anyone — the host is applying a declaration it already
carries, to the machine it is already on. carries, to the machine it is already on.
@@ -238,7 +238,7 @@ carries, to the machine it is already on.
the bootstrap script never had. Its specialness lasted two commands. the bootstrap script never had. Its specialness lasted two commands.
**And enrolment is exercised on node one.** The path every other node depends on is walked **And enrolment is exercised on node one.** The path every other node depends on is walked
immediately, against a control plane on the same machine, rather than being written and first immediately, against a controller on the same machine, rather than being written and first
used months later on node two. used months later on node two.
--- ---
@@ -265,7 +265,7 @@ closes — an authoritative local store, reconcile on start, *last heard from* r
alarm — is what an episodic host needs, at a shorter period. alarm — is what an episodic host needs, at a shorter period.
**It cannot be the first node**, and that is not a limitation to work around. Every step of **It cannot be the first node**, and that is not a limitation to work around. Every step of
raising a substrate is a `package`, a `container` or an `action` against one, and a partial host raising a foundation is a `package`, a `container` or an `action` against one, and a partial host
refuses the first two. So `mesh-host-android bundle` returns a file that says so rather than an refuses the first two. So `mesh-host-android bundle` returns a file that says so rather than an
empty placeholder waiting to be filled in. empty placeholder waiting to be filled in.
@@ -350,7 +350,7 @@ rest. The rule that exists to stop the host lying about what it did also makes i
## Updating what the node holds ## Updating what the node holds
An ordinary declaration. Someone assigns a module; the control plane recomputes what that node An ordinary declaration. Someone assigns a module; the controller recomputes what that node
should be and sends it; the host applies the difference and removes what is no longer declared. should be and sends it; the host applies the difference and removes what is no longer declared.
**Removal is not symmetric, and the asymmetry is the design:** **Removal is not symmetric, and the asymmetry is the design:**
@@ -410,7 +410,7 @@ one binary that has always been the same binary.
Two cases, and they are genuinely different. Two cases, and they are genuinely different.
**Graceful.** The control plane sends a final declaration that names nothing. The host removes **Graceful.** The controller sends a final declaration that names nothing. The host removes
what it owns by the table above, reports, and drops its identity. The machine keeps the host what it owns by the table above, reports, and drops its identity. The machine keeps the host
installed and is back to `hosted`. Nothing is left behind that anybody has to remember. installed and is back to `hosted`. Nothing is left behind that anybody has to remember.
@@ -638,7 +638,7 @@ lets a node verify a mesh it has never spoken to
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). A token emailed, ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). A token emailed,
committed, or dropped in shared storage has lost the only property that makes it worth carrying. committed, or dropped in shared storage has lost the only property that makes it worth carrying.
**On the first node it comes from the control plane that was raised two commands ago**, which is **On the first node it comes from the controller that was raised two commands ago**, which is
the same command against a mesh that is one machine old. the same command against a mesh that is one machine old.
--- ---
+4 -4
View File
@@ -78,7 +78,7 @@ whenever anybody writes something reusable, which is constantly.
## Delivery is a comparison, not a pipeline ## Delivery is a comparison, not a pipeline
The control plane holds two facts and builds the difference: The controller holds two facts and builds the difference:
``` ```
what source exists ─┐ what source exists ─┐
@@ -94,7 +94,7 @@ That is the same shape the host uses on a machine, one layer up:
| | reconciles | against | | | reconciles | against |
|---|---|---| |---|---|---|
| the control plane | artifacts | source | | the controller | artifacts | source |
| the host | machine state | declarations | | the host | machine state | declarations |
**There is no pipeline as a state machine.** No stage list something can be omitted from, and no **There is no pipeline as a state machine.** No stage list something can be omitted from, and no
@@ -178,9 +178,9 @@ Not aspirations — things without which the above does not work:
same digest, a cascade would stop at the first module whose output did not move. Without them, same digest, a cascade would stop at the first module whose output did not move. Without them,
one core-library commit redeploys the fleet with no behavioural change. one core-library commit redeploys the fleet with no behavioural change.
- **How a module publishes its own types**, which differs per language. - **How a module publishes its own types**, which differs per language.
- **How the control plane upgrades itself.** It declares its own new version and the host applies - **How the controller upgrades itself.** It declares its own new version and the host applies
it — but if the new one is broken, the thing that would fix it is the thing that is broken. The it — but if the new one is broken, the thing that would fix it is the thing that is broken. The
host has a launcher for exactly this; the control plane has nothing. host has a launcher for exactly this; the controller has nothing.
## What "behind" means, and what it used to mean ## What "behind" means, and what it used to mean
+3 -3
View File
@@ -41,9 +41,9 @@ it is enough to freeze it.** A board that reads the provisioning tables directly
breaks when provisioning changes its tables, and the change then gets weighed against the board. breaks when provisioning changes its tables, and the change then gets weighed against the board.
**So a board reads through interfaces and holds nothing.** Everything on the mesh page above is **So a board reads through interfaces and holds nothing.** Everything on the mesh page above is
already answerable by asking the control plane — what nodes exist, what each resolves to, what it already answerable by asking the controller — what nodes exist, what each resolves to, what it
takes from elsewhere, which module came from which commit. A board that asks those questions is a takes from elsewhere, which module came from which commit. A board that asks those questions is a
client. A board that queries `inventory` is a second control plane with a worse contract. client. A board that queries `inventory` is a second controller with a worse contract.
**It stores nothing of its own.** No cache that can disagree, no table of "what the mesh looked **It stores nothing of its own.** No cache that can disagree, no table of "what the mesh looked
like last time". If a question is slow to answer, the answer belongs in the context that owns it, like last time". If a question is slow to answer, the answer belongs in the context that owns it,
@@ -63,7 +63,7 @@ calls the security boundary, and a login there would guard a room whose door is
building. This one faces everybody. building. This one faces everybody.
**Which makes the identity provider the mesh's outermost gate.** The board is a presentation layer **Which makes the identity provider the mesh's outermost gate.** The board is a presentation layer
over the control plane and the control plane's networked surfaces can change the mesh over the controller and the controller's networked surfaces can change the mesh
([ADR 0035](../../02-DECISIONS/0035-one-implementation-several-surfaces.md)), so **whoever that ([ADR 0035](../../02-DECISIONS/0035-one-implementation-several-surfaces.md)), so **whoever that
provider admits can assign modules, from anywhere.** Said flatly because it is easy to arrive at provider admits can assign modules, from anywhere.** Said flatly because it is easy to arrive at
one reasonable step at a time and then be surprised by. one reasonable step at a time and then be surprised by.
+25 -9
View File
@@ -85,10 +85,10 @@ the worst possible moment.
## The builder runs on a node ## The builder runs on a node
**Not in the control plane, and this is the same boundary as everywhere else.** Building needs a **Not in the controller, and this is the same boundary as everywhere else.** Building needs a
container runtime and a working tree; what the control plane may send a machine is bounded by the container runtime and a working tree; what the controller may send a machine is bounded by the
declaration language ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and *run this build* declaration language ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and *run this build*
is not in it. The alternative — the control plane holding a container socket — would make it the is not in it. The alternative — the controller holding a container socket — would make it the
one component that can do anything on any machine, which is the property the whole design is one component that can do anything on any machine, which is the property the whole design is
arranged to avoid. arranged to avoid.
@@ -96,7 +96,7 @@ So the builder is a program a machine runs, given work over the broker like anyt
its own credential and nothing more. its own credential and nothing more.
**A build is work, not state**, and that is why it does not travel as a declaration. Everything **A build is work, not state**, and that is why it does not travel as a declaration. Everything
else the control plane sends a node is *what you should be*, reconciled forever. A build happens else the controller sends a node is *what you should be*, reconciled forever. A build happens
once and is finished; as a declaration it would either rebuild on every reconcile or carry "and I once and is finished; as a declaration it would either rebuild on every reconcile or carry "and I
already did this" — state about an event rather than about a machine. already did this" — state about an event rather than about a machine.
@@ -203,7 +203,7 @@ at something. A build that failed before it knew what it was building keeps the
is what a person goes and looks at. is what a person goes and looks at.
Recording is idempotent on the correlation, because a result arrives twice — once as the answer to Recording is idempotent on the correlation, because a result arrives twice — once as the answer to
whoever asked and once on the exchange, where the control plane is also listening. Two rows would whoever asked and once on the exchange, where the controller is also listening. Two rows would
show one build as two, and which is real is not answerable afterwards. show one build as two, and which is real is not answerable afterwards.
That is what a builds view reads, and until it existed there was nothing to read: a result was That is what a builds view reads, and until it existed there was nothing to read: a result was
@@ -225,9 +225,16 @@ answered to the asker and kept nowhere.
| kind | is | | kind | is |
|---|---| |---|---|
| **image** | built from a Dockerfile in this repository | | **image** | built from a Dockerfile in this repository |
| **bundle** | this module's own code, compiled by the toolchain its language implies, then packed |
| **archive** | a directory in this repository, packed | | **archive** | a directory in this repository, packed |
| **upstream** | an image somebody else built, mirrored into the mesh's own registry | | **upstream** | an image somebody else built, mirrored into the mesh's own registry |
**The second was added later and is why most modules now need no Dockerfile.** An archive packs a
directory as it stands, so shipping compiled output meant compiling somewhere first — and that
meant every module repeating a recipe that is easy to get wrong in ways that fail elsewhere. The
whole surface a module has, and a module that exercises all of it, are in
[`18-building-a-module`](18-building-a-module.md).
**The third exists because a module usually runs software it did not write.** A database module **The third exists because a module usually runs software it did not write.** A database module
ships configuration and a provisioner and does not build a database. Naming the upstream reference ships configuration and a provisioner and does not build a database. Naming the upstream reference
directly would need every machine to reach a public registry, and would pin to a tag its owner can directly would need every machine to reach a public registry, and would pin to a tag its owner can
@@ -374,13 +381,13 @@ have a route and one does not:
| What | Why it cannot come through the loop | How it arrives | | What | Why it cannot come through the loop | How it arrives |
|---|---|---| |---|---|---|
| The control plane | It is what installs modules. Nothing can install it before it runs. | **Carried inside the installer** and published once there is a registry ([ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md)) | | The controller | It is what installs modules. Nothing can install it before it runs. | **Carried inside the installer** and published once there is a registry ([ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md)) |
| The registry | It *is* where artifacts are delivered from. A store cannot be delivered through itself. | **Pulled from the public internet** — its image is an ordinary public one, never built ([`04-ISSUES/029`](../../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md)) | | The registry | It *is* where artifacts are delivered from. A store cannot be delivered through itself. | **Pulled from the public internet** — its image is an ordinary public one, never built ([`04-ISSUES/029`](../../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md)) |
| The builder | It is what builds. Nothing builds it before it runs. | Built at genesis by the init builder ([ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md)) | | The builder | It is what builds. Nothing builds it before it runs. | Built at genesis by the init builder ([ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md)) |
| The catalogue | It owns the module graph, and nothing can be resolved or installed without it. | Built at genesis by the init builder ([ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md)) | | The catalogue | It owns the module graph, and nothing can be resolved or installed without it. | Built at genesis by the init builder ([ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md)) |
**How they arrive is settled and not yet built.** The installer carries an init builder, which **How they arrive is settled and not yet built.** The installer carries an init builder, which
clones the source and builds the control plane, the catalogue and the builder before a mesh exists clones the source and builds the controller, the catalogue and the builder before a mesh exists
to install anything. Two things about that are open and named in to install anything. Two things about that are open and named in
[ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md): where the init builder [ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md): where the init builder
clones from, given the forge normally runs on the mesh it would be rebuilding, and what it clones from, given the forge normally runs on the mesh it would be rebuilding, and what it
@@ -393,7 +400,7 @@ three. This section previously said the list was closed at three, which was writ
catalogue had an owner and is corrected here rather than left to be reasoned from. catalogue had an owner and is corrected here rather than left to be reasoned from.
**And the answer for all four is now one mechanism, not four special cases.** Genesis carries an **And the answer for all four is now one mechanism, not four special cases.** Genesis carries an
*init builder* and builds the core modules on the machine — control plane, catalogue and builder — *init builder* and builds the core modules on the machine — controller, catalogue and builder —
rather than carrying a finished image of any of them. So the question is no longer "how does this rather than carrying a finished image of any of them. So the question is no longer "how does this
one get here first?" asked once per component; it is answered once, by the thing that is carried one get here first?" asked once per component; it is answered once, by the thing that is carried
being a builder rather than a result. being a builder rather than a result.
@@ -401,12 +408,21 @@ being a builder rather than a result.
Everything outside those four is either upstream — a third-party image pulled by digest — or built Everything outside those four is either upstream — a third-party image pulled by digest — or built
by the builder from a repository and a path, and published to the registry. by the builder from a repository and a path, and published to the registry.
**One of those built things has an ordering constraint worth naming, because it looks like a fifth
member of the list and is not.** The SDK the toolchain compiles against is built by the ordinary
builder and published like anything else — but it cannot be compiled *in the mesh toolchain*, since
that toolchain is built from it, and it is published to the *package* registry rather than the
artifact store. So it is compiled on a public base image and published before the toolchain that
consumes it ([ADR 0076](../../02-DECISIONS/0076-the-sdk-is-a-published-package.md)). It is not
carried and it is not machinery; it is a dependency with a sequence, which is why it belongs here as
a footnote to the rule rather than a row in the table.
**A carried artifact is not a differently-pinned artifact.** Once published it is named by a digest **A carried artifact is not a differently-pinned artifact.** Once published it is named by a digest
the mesh's registry assigned, exactly like everything the builder produces. A reader cannot tell the mesh's registry assigned, exactly like everything the builder produces. A reader cannot tell
from a running mesh which of its images were carried, and that is the point: carrying is how the from a running mesh which of its images were carried, and that is the point: carrying is how the
first copy arrives, not what it permanently is. first copy arrives, not what it permanently is.
*Checked by the thing already checked at genesis: after installing, the running control plane is *Checked by the thing already checked at genesis: after installing, the running controller is
pinned to a digest the mesh's own registry assigned, and not to the id of the image the installer pinned to a digest the mesh's own registry assigned, and not to the id of the image the installer
carried. The same check applies to the builder and to the registry, and it is the same check — carried. The same check applies to the builder and to the registry, and it is the same check —
an image id where a registry digest belongs means the pivot did not finish.* an image id where a registry digest belongs means the pivot did not finish.*
@@ -70,7 +70,7 @@ The mesh generated the password, sealed it to the machine that must accept it, a
plaintext** — so it cannot tell a database to start accepting it. Something on that machine reads plaintext** — so it cannot tell a database to start accepting it. Something on that machine reads
what the host wrote and makes it true. what the host wrote and makes it true.
That something is part of the module, not part of the control plane. **The control plane decides That something is part of the module, not part of the controller. **The controller decides
and never touches a machine; a provisioner runs on the machine and touches it.** Two files, because and never touches a machine; a provisioner runs on the machine and touches it.** Two files, because
the mesh could not compose a document containing a value it does not have: the mesh could not compose a document containing a value it does not have:
+4 -4
View File
@@ -59,7 +59,7 @@ names neither the licence nor the mesh.
**A key is read from a file or standard input, never an argument.** A key on a command line is a **A key is read from a file or standard input, never an argument.** A key on a command line is a
key in shell history and in every process listing taken while it ran. It is never echoed back: key in shell history and in every process listing taken while it ran. It is never echoed back:
what is stored is unreadable by whoever holds it, the control plane included, and printing it what is stored is unreadable by whoever holds it, the controller included, and printing it
would put the one copy that matters on a terminal. would put the one copy that matters on a terminal.
## Refusing is felt, and that is the design working ## Refusing is felt, and that is the design working
@@ -83,7 +83,7 @@ per machine, which is a step toward it and is not it.
*2026-08-31: this gap now has named consumers rather than hypothetical ones.* *2026-08-31: this gap now has named consumers rather than hypothetical ones.*
[ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) puts two sessions on the [ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) puts two sessions on the
control-plane node — the node's own and the mesh's — each bound in its own right. See controller node — the node's own and the mesh's — each bound in its own right. See
[`15-the-agent-session.md`](15-the-agent-session.md). [`15-the-agent-session.md`](15-the-agent-session.md).
**And for sessions the gap is already closed, which was not obvious.** A binding is per module per **And for sessions the gap is already closed, which was not obvious.** A binding is per module per
@@ -172,12 +172,12 @@ it; the metric is vendor-defined, so no false common unit is forced. Anthropic b
In the lab, on real machines, in the order a person would meet it: a consumer is refused with both In the lab, on real machines, in the order a person would meet it: a consumer is refused with both
candidates named; put on one and still refused because no key exists; the key is given on standard candidates named; put on one and still refused because no key exists; the key is given on standard
input and not echoed; the public half arrives saying it came from a record rather than a machine; input and not echoed; the public half arrives saying it came from a record rather than a machine;
the key arrives readable only by that machine — and it is **nowhere in the control plane's own the key arrives readable only by that machine — and it is **nowhere in the controller's own
database**, nor in anything that crossed the broker. database**, nor in anything that crossed the broker.
For the generalisation ([ADR 0050](../../02-DECISIONS/0050-model-access-is-vendor-agnostic.md)): a For the generalisation ([ADR 0050](../../02-DECISIONS/0050-model-access-is-vendor-agnostic.md)): a
second vendor — `anthropic-api-key`, static-key — is bound to a consumer and exercises the whole path second vendor — `anthropic-api-key`, static-key — is bound to a consumer and exercises the whole path
with the carve-out switched off, its key sealed per node and absent from the control plane's database. with the carve-out switched off, its key sealed per node and absent from the controller's database.
For a `refreshable-grant` licence, the refresh token is asserted to exist (encrypted) **only on the For a `refreshable-grant` licence, the refresh token is asserted to exist (encrypted) **only on the
manager node**, to be **absent from every holder's delivery**, and the delivered credential to be manager node**, to be **absent from every holder's delivery**, and the delivered credential to be
access-token-only; a `static-key` licence stores no refresh token anywhere. A scenario with two access-token-only; a `static-key` licence stores no refresh token anywhere. A scenario with two
+7 -7
View File
@@ -22,7 +22,7 @@ so, and the differences are few enough to list here:
| **context root** | the node's | the mesh's | | **context root** | the node's | the mesh's |
| **engram** | that node's | the mesh's | | **engram** | that node's | the mesh's |
| **licence** | bound in its own right | bound in its own right | | **licence** | bound in its own right | bound in its own right |
| **runs on** | that node | the node holding the control plane | | **runs on** | that node | the node holding the controller |
| **how many** | one per node | one | | **how many** | one per node | one |
Everything below applies to both unless it says otherwise. Everything below applies to both unless it says otherwise.
@@ -64,9 +64,9 @@ as it reports anything else.
**A node's session runs on that node**, and cannot be moved. Moved, one machine is answering as **A node's session runs on that node**, and cannot be moved. Moved, one machine is answering as
another (ADR 0004). another (ADR 0004).
**The mesh's session runs on the node holding the control plane.** The reasoning is in ADR 0026 **The mesh's session runs on the node holding the controller.** The reasoning is in ADR 0026
and is worth carrying here because it is easy to get backwards: this is not *the important agent and is worth carrying here because it is easy to get backwards: this is not *the important agent
goes on the important machine*. It is that the control-plane node is already the one place goes on the important machine*. It is that the controller node is already the one place
excepted from *compromise of a node is compromise of that node*, and an agent able to reach excepted from *compromise of a node is compromise of that node*, and an agent able to reach
everything, placed anywhere else, would create a second such place. everything, placed anywhere else, would create a second such place.
@@ -102,7 +102,7 @@ machine*.
That is not sufficient here, and the shortfall is concrete rather than theoretical: That is not sufficient here, and the shortfall is concrete rather than theoretical:
- the control-plane node hosts **two** sessions, which must be able to hold **different** - the controller node hosts **two** sessions, which must be able to hold **different**
licences — a per-machine binding cannot express it at all; licences — a per-machine binding cannot express it at all;
- *this node's session uses the personal licence, the mesh's uses the company one* is the ordinary - *this node's session uses the personal licence, the mesh's uses the company one* is the ordinary
case, not an exotic one. case, not an exotic one.
@@ -125,7 +125,7 @@ session a different agent from another, and memory is part of what makes it *tha
and it is not a view over theirs. What the mesh has been asked, and what it worked out, is held and it is not a view over theirs. What the mesh has been asked, and what it worked out, is held
in the mesh's root — not in the root of the node that happens to host it. in the mesh's root — not in the root of the node that happens to host it.
**That distinction is the point of putting it there.** The control-plane node runs two sessions **That distinction is the point of putting it there.** The controller node runs two sessions
on one machine. If memory belonged to the machine rather than to the root, they would share it, on one machine. If memory belonged to the machine rather than to the root, they would share it,
and the mesh's recollection of a fortnight of questions would be indistinguishable from that and the mesh's recollection of a fortnight of questions would be indistinguishable from that
node's own — which is the collision ADR 0026 exists to avoid, arriving through the back door. node's own — which is the collision ADR 0026 exists to avoid, arriving through the back door.
@@ -188,7 +188,7 @@ here so the shape is not rediscovered.
## Consequences ## Consequences
**Two sessions on one node, and no ambiguity.** The control-plane node hosts its own node session **Two sessions on one node, and no ambiguity.** The controller node hosts its own node session
and the mesh session. ADR 0004's *one per node* forbids ambiguity about who answers when a **node** and the mesh session. ADR 0004's *one per node* forbids ambiguity about who answers when a **node**
is addressed; these answer to different addresses. is addressed; these answer to different addresses.
@@ -207,7 +207,7 @@ and these run in the lab on real machines:
| Check | Defends | | Check | Defends |
|---|---| |---|---|
| a node is asked something and its session answers | ADR 0004 | | a node is asked something and its session answers | ADR 0004 |
| the mesh is asked something and the mesh session answers, on the control-plane node | ADR 0026 | | the mesh is asked something and the mesh session answers, on the controller node | ADR 0026 |
| both sessions on that node answer, to their own addresses, without ambiguity | ADR 0026 | | both sessions on that node answer, to their own addresses, without ambiguity | ADR 0026 |
| a session switched off replies saying so, rather than timing out | ADR 0004 | | a session switched off replies saying so, rather than timing out | ADR 0004 |
| a session whose engram was changed reports having applied it, like any declared file | ADR 0005 | | a session whose engram was changed reports having applied it, like any declared file | ADR 0005 |
+5 -5
View File
@@ -24,7 +24,7 @@ checklist: what is already sayable, what is deliberately not, and what is missin
| **depends on another module** | 65 | `requires` naming a module. A requirement naming a module means *that module*, not anything providing the name | | **depends on another module** | 65 | `requires` naming a module. A requirement naming a module means *that module*, not anything providing the name |
| **system packages** | 28 | the `package` shape | | **system packages** | 28 | the `package` shape |
| **a container** | 48 | the `container` shape, pinned by digest | | **a container** | 48 | the `container` shape, pinned by digest |
| **systemd units** | 17 | a `file` for the unit, a `service` for the state it should be in | | **systemd units** | 17 | a `process`, which the mesh writes the unit for. *Was: a `file` for the unit and a `service` for its state — which made every module author write unit syntax, and is why `process` exists ([`18`](18-building-a-module.md))* |
| **how to reach it** | 17 | `serves`, with the mesh adding which machine and where | | **how to reach it** | 17 | `serves`, with the mesh adding which machine and where |
| **a public name** | 11 | requiring `route` and contributing the name | | **a public name** | 11 | requiring `route` and contributing the name |
| **ports it opens** | 11 | `listens`, from which filtering is computed | | **ports it opens** | 11 | `listens`, from which filtering is computed |
@@ -33,7 +33,7 @@ checklist: what is already sayable, what is deliberately not, and what is missin
| **restart when something changes** | 11 | `restart-on` | | **restart when something changes** | 11 | `restart-on` |
| **a generated credential** | 20 | `own-secrets`, sealed to the machine | | **a generated credential** | 20 | `own-secrets`, sealed to the machine |
| **a value that differs per node** | 43 | settings, and `computed` for what only the mesh knows | | **a value that differs per node** | 43 | settings, and `computed` for what only the mesh knows |
| **images built from source** | 2 | `build.artifacts` | | **images built from source** | 2 | `build.artifacts` — an `image` from a Dockerfile, or a `bundle`, which names a language and lets the mesh choose the toolchain ([`18`](18-building-a-module.md)) |
## Deliberately not sayable ## Deliberately not sayable
@@ -163,14 +163,14 @@ same module. There is one derivation here, and there should stay one.
sealing is worth its inconvenience. sealing is worth its inconvenience.
**An image store is a module, and was written up here as something the mesh does.** It was **An image store is a module, and was written up here as something the mesh does.** It was
considered for the substrate and removed, because the test is not *can it grant itself one* — considered for the foundation and removed, because the test is not *can it grant itself one* —
nearly anything passes that — but whether the control plane needs it before it can give its first nearly anything passes that — but whether the controller needs it before it can give its first
instruction. It does not. So a registry somebody runs for their own images is the same module as instruction. It does not. So a registry somebody runs for their own images is the same module as
the one the mesh runs for its own: it offers a place to push, and claims that role once per the one the mesh runs for its own: it offers a place to push, and claims that role once per
machine. machine.
**A rule was enforced only at the far end.** A module may not declare an action, and the host **A rule was enforced only at the far end.** A module may not declare an action, and the host
refused one correctly — but the control plane accepted it into the catalogue, resolved it and refused one correctly — but the controller accepted it into the catalogue, resolved it and
pushed it, so the refusal arrived on a machine with nothing tying it back to the manifest. The pushed it, so the refusal arrived on a machine with nothing tying it back to the manifest. The
rule held; it was just unusable, which is the same shape as the network shape that cost five rule held; it was just unusable, which is the same shape as the network shape that cost five
failing tests before anyone read the host's log. It is now refused where it is written. failing tests before anyone read the host's log. It is now refused where it is written.
+31 -21
View File
@@ -30,7 +30,7 @@ rules cannot all hold at once, and it is resolved by a pivot
**Joining** happens on every machine after the first, and it is ordinary. A mesh exists, so it can **Joining** happens on every machine after the first, and it is ordinary. A mesh exists, so it can
be asked for a token and told what the machine should be. Joining installs the host and nothing be asked for a token and told what the machine should be. Joining installs the host and nothing
else: no temporary anything, no substrate raised by hand, no registry. else: no temporary anything, no foundation raised by hand, no registry.
Confusing the two is what produced a procedure that only ever worked in a fixture. A bed that Confusing the two is what produced a procedure that only ever worked in a fixture. A bed that
raises four machines the same way has not tested genesis at all — it has tested joining, four raises four machines the same way has not tested genesis at all — it has tested joining, four
@@ -38,13 +38,13 @@ times, with the first one hand-fed.
## What changed, and what did not ## What changed, and what did not
*2026-09-13.* The installer carries a builder now, and builds the control plane it raises. Three *2026-09-13.* The installer carries a builder now, and builds the controller it raises. Three
records settle it: [ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md) that records settle it: [ADR 0070](../../02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md) that
genesis builds rather than carries, [ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md) genesis builds rather than carries, [ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md)
where it clones from, and [ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md) how where it clones from, and [ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md) how
the builder arrives — which also records an argument that failed. It was put that a produced image the builder arrives — which also records an argument that failed. It was put that a produced image
must be published before anything can fetch it, so the registry would have to come up before the must be published before anything can fetch it, so the registry would have to come up before the
control plane. It does not: the machine that builds the image is the machine that runs it, and a controller. It does not: the machine that builds the image is the machine that runs it, and a
local image is named by the digest of its own configuration exactly as a carried one is. **Building local image is named by the digest of its own configuration exactly as a carried one is. **Building
changes where the bytes came from, not where they are.** changes where the bytes came from, not where they are.**
@@ -53,7 +53,7 @@ written. What follows describes the program that exists.
## Genesis ## Genesis
The installer is a single program carrying **the builder** inside it — not the control plane The installer is a single program carrying **the builder** inside it — not the controller
([ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md)). What cannot be fetched is ([ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md)). What cannot be fetched is
the thing that does the fetching, so that is what is carried; everything else is made here. the thing that does the fetching, so that is what is carried; everything else is made here.
@@ -62,12 +62,12 @@ It proceeds in one direction, and every step is safe to run again.
**First it refuses to start if the machine is not ready.** A container runtime, the ability to **First it refuses to start if the machine is not ready.** A container runtime, the ability to
write where it must write, the host binary where it expects it — and a repository and a commit to write where it must write, the host binary where it expects it — and a repository and a commit to
build from, because an installer told nothing would raise a store and a broker and then have build from, because an installer told nothing would raise a store and a broker and then have
nothing to raise a control plane from. A machine that is not ready is told what is missing rather nothing to raise a controller from. A machine that is not ready is told what is missing rather
than half-changed. than half-changed.
**Then it loads the carried builder and builds the control plane with it**, from a repository on a **Then it loads the carried builder and builds the controller with it**, from a repository on a
mesh that already exists and a named commit ([ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md)). mesh that already exists and a named commit ([ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md)).
This is the same repository and path every later rebuild of the control plane will use, so what This is the same repository and path every later rebuild of the controller will use, so what
raises the mesh is the same thing that will maintain it. raises the mesh is the same thing that will maintain it.
**Then it describes what the machine will become.** The image it just made is named by the digest **Then it describes what the machine will become.** The image it just made is named by the digest
@@ -75,7 +75,7 @@ of its own configuration — content-addressed and unforgeable, and requiring no
it. That is legal precisely where nothing could have served one, and it is why building here needs it. That is legal precisely where nothing could have served one, and it is why building here needs
no registry: the machine that made the image is the machine that will run it. no registry: the machine that made the image is the machine that will run it.
**Then it raises the substrate and a temporary control plane, and waits for that control plane to **Then it raises the foundation and a temporary controller, and waits for that controller to
answer.** At this point the machine is a mesh of one node with nothing joined to it. answer.** At this point the machine is a mesh of one node with nothing joined to it.
**Then the machine joins the mesh it is itself running.** It enrols, and an agent runs on it. Being **Then the machine joins the mesh it is itself running.** It enrols, and an agent runs on it. Being
@@ -84,13 +84,13 @@ enrolling is itself the thing that makes a mesh hear from a machine.
**Then it installs a registry**, so the mesh has somewhere to keep its own images. **Then it installs a registry**, so the mesh has somewhere to keep its own images.
**Then it publishes the control plane's image to that registry**, which is the moment the image **Then it publishes the controller's image to that registry**, which is the moment the image
first receives a digest assigned by something other than itself. This is the carrying step, and it first receives a digest assigned by something other than itself. This is the carrying step, and it
is the same step for all three things the build loop cannot produce for itself — the control plane, is the same step for all three things the build loop cannot produce for itself — the controller,
the registry, and the builder. The rule and its closed list are in the registry, and the builder. The rule and its closed list are in
[`12-a-module-repository`](12-a-module-repository.md#the-three-the-loop-cannot-build-and-there-are-only-three). [`12-a-module-repository`](12-a-module-repository.md#the-three-the-loop-cannot-build-and-there-are-only-three).
**Then it installs the control plane again, as an ordinary module pinned to that digest, and drops **Then it installs the controller again, as an ordinary module pinned to that digest, and drops
the temporary one.** The pivot is complete: what raised the mesh is gone, and what runs is a module the temporary one.** The pivot is complete: what raised the mesh is gone, and what runs is a module
like any other. From here the mesh can build and roll out its own upgrades, including to the thing like any other. From here the mesh can build and roll out its own upgrades, including to the thing
that runs it. that runs it.
@@ -98,7 +98,7 @@ that runs it.
## After the pivot, and still part of installing ## After the pivot, and still part of installing
Genesis ends with a mesh of one that runs, and that is not the same as a mesh that works. What it Genesis ends with a mesh of one that runs, and that is not the same as a mesh that works. What it
has is a control plane, a store, a queue and a registry. What it cannot yet do is **produce has is a controller, a store, a queue and a registry. What it cannot yet do is **produce
anything** — and almost every module in the catalogue is waiting to be produced, because a manifest anything** — and almost every module in the catalogue is waiting to be produced, because a manifest
names what its artifacts are and nothing has made them. names what its artifacts are and nothing has made them.
@@ -107,9 +107,9 @@ So installing continues:
**The builder arrives, and installing is what brings it.** It is a module like any other and is **The builder arrives, and installing is what brings it.** It is a module like any other and is
assigned to a machine like any other, but it cannot be built by the thing it is — see assigned to a machine like any other, but it cannot be built by the thing it is — see
[`12-a-module-repository`](12-a-module-repository.md#the-three-the-loop-cannot-build-and-there-are-only-three). [`12-a-module-repository`](12-a-module-repository.md#the-three-the-loop-cannot-build-and-there-are-only-three).
So it is carried, and it is already here: it is what built the control plane. The last step of So it is carried, and it is already here: it is what built the controller. The last step of
installing publishes it into the mesh's own registry, installs it as an ordinary module pinned to installing publishes it into the mesh's own registry, installs it as an ordinary module pinned to
that digest, and issues it a broker account — the same two acts the control plane went through, that digest, and issues it a broker account — the same two acts the controller went through,
plus the one thing only a builder needs. The account is issued before the machine is sent anything, plus the one thing only a builder needs. The account is issued before the machine is sent anything,
because a builder that arrives without its credential starts, finds nothing it may read, and waits, because a builder that arrives without its credential starts, finds nothing it may read, and waits,
which looks exactly like a builder with no work. which looks exactly like a builder with no work.
@@ -121,12 +121,12 @@ publishes each artifact into the mesh's own registry, and hands back the module
pinned and the commit recorded. The mesh records that, and from then on the module is described by pinned and the commit recorded. The mesh records that, and from then on the module is described by
something it made rather than by a placeholder. something it made rather than by a placeholder.
**The control plane is built like the rest.** It was carried in and published once, which got the **The controller is built like the rest.** It was carried in and published once, which got the
mesh running; building it from its own repository and path is what makes it upgradeable. The first mesh running; building it from its own repository and path is what makes it upgradeable. The first
time that happens is the moment the mesh stops depending on the installer for anything. time that happens is the moment the mesh stops depending on the installer for anything.
**And then the catalogue.** Every module with source of its own is built the same way. Until this **And then the catalogue.** Every module with source of its own is built the same way. Until this
has happened a mesh can install only what is public or carried, which is the substrate and little has happened a mesh can install only what is public or carried, which is the foundation and little
else. else.
Only after all of that is the ordinary loop available: change a module's source, the mesh notices Only after all of that is the ordinary loop available: change a module's source, the mesh notices
@@ -138,7 +138,7 @@ What remains after *that* belongs to somebody else: adding machines, and decidin
## Joining ## Joining
A machine joins with the host binary and a token. It does not raise a substrate, does not install a A machine joins with the host binary and a token. It does not raise a foundation, does not install a
registry, and is never enrolled twice. The mesh already knows how to tell a machine what to be; registry, and is never enrolled twice. The mesh already knows how to tell a machine what to be;
joining is the point at which a machine starts listening. joining is the point at which a machine starts listening.
@@ -169,8 +169,8 @@ paragraphs above describing the catalogue being built are a thing somebody now t
thing that cannot happen. thing that cannot happen.
**A module's declaration still has to be copied onto the machine by hand.** The installer reads the **A module's declaration still has to be copied onto the machine by hand.** The installer reads the
registry's and the control plane's manifests from a checkout somebody put there. The control plane's registry's and the controller's manifests from a checkout somebody put there. The controller's
now lives in the control plane's own repository, which the installer clones anyway, so this is a now lives in the controller's own repository, which the installer clones anyway, so this is a
thing that can be removed rather than a thing that must be designed. thing that can be removed rather than a thing that must be designed.
**A machine has no account for a registry that asks for one.** The mesh grants a consumer a **A machine has no account for a registry that asks for one.** The mesh grants a consumer a
@@ -178,6 +178,16 @@ credential for a database; it does not yet do so for the store its own images li
avoids the question by carrying the image it needs, which makes this a joining problem and a avoids the question by carrying the image it needs, which makes this a joining problem and a
pulling problem, not a genesis one. pulling problem, not a genesis one.
**The SDK still comes from a git URL, and the ordering that fixes it is decided but not built.**
[ADR 0076](../../02-DECISIONS/0076-the-sdk-is-a-published-package.md) settles that the package
registry (gitea) comes up and the SDK is published into it *before* the base toolchain is built, so
the toolchain resolves the SDK by version rather than cloning it — closing
[issue 053](../../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md). The
builder already knows how to be handed a package-registry credential and inject it into a build; what
is not yet wired is the genesis step that raises gitea and publishes the SDK ahead of the base, and
the toolchain's own manifest still names the SDK by a git URL. Until both land, the base build clones
the SDK inside `docker build`, which is slow and names a branch head rather than a version.
## How these rules are checked ## How these rules are checked
| Rule | Checked by | | Rule | Checked by |
@@ -188,6 +198,6 @@ pulling problem, not a genesis one.
| An image is named exactly | A machine refuses a bundle naming an image by tag. The refusal is exercised, not assumed. | | An image is named exactly | A machine refuses a bundle naming an image by tag. The refusal is exercised, not assumed. |
| The installer is what installed this | **Nothing.** See above. | | The installer is what installed this | **Nothing.** See above. |
| The builder can arrive on a fresh mesh | The installer carries it and installs it as its last step, and the genesis bed raises a machine by running the installer. A bed that raises one any other way fails its own acceptance check. | | The builder can arrive on a fresh mesh | The installer carries it and installs it as its last step, and the genesis bed raises a machine by running the installer. A bed that raises one any other way fails its own acceptance check. |
| The control plane a mesh runs is one it built | The genesis bed asserts the running control plane is pinned to a digest this mesh's own registry serves, for an image built from a named repository and commit — not one the installer carried. | | The controller a mesh runs is one it built | The genesis bed asserts the running controller is pinned to a digest this mesh's own registry serves, for an image built from a named repository and commit — not one the installer carried. |
| A core module is built rather than only carried | The control plane is rebuilt from its own repository and path, and the running mesh is upgraded to the result — the same path any module takes. | | A core module is built rather than only carried | The controller is rebuilt from its own repository and path, and the running mesh is upgraded to the result — the same path any module takes. |
| Installing produced a mesh that can produce | A module with source of its own is asked for and comes back pinned to a digest this mesh's registry assigned, not to a placeholder. **Done by hand on a raised machine, not yet by a bed** — it built the shared base and then a module naming that base. Nothing automated asserts it, which makes this the weakest check on this page. | | Installing produced a mesh that can produce | A module with source of its own is asked for and comes back pinned to a digest this mesh's registry assigned, not to a placeholder. **Done by hand on a raised machine, not yet by a bed** — it built the shared base and then a module naming that base. Nothing automated asserts it, which makes this the weakest check on this page. |
+222
View File
@@ -0,0 +1,222 @@
---
layer: to-be
status: proposed
code:
- mesh-control cmd/mesh-builder
- mesh-control internal/builder
- mesh-catalog modules/builder
updated: 2026-09-15
decisions:
- 02-DECISIONS/0040-what-a-module-is.md
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
- 02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md
- 02-DECISIONS/0009-modules-and-the-graph.md
---
# Building a module
[`12-a-module-repository`](12-a-module-repository.md) says what a module may build — an image, an
archive, an upstream mirror — and where the result goes. This says **how a build is modelled**, and
why the current model does not fit what a module is.
## The domain, in one sentence
Turning a module's source into artifacts the mesh can pin, publish and deliver — and saying
truthfully what was produced and what it was produced against.
Everything else is somebody else's: *what* to build is the controller's, *what a build means* is
the catalogue's ([ADR 0072](../../02-DECISIONS/0072-two-graphs-and-the-build-chain.md)), *where an artifact runs* is the
controller's again. The builder's whole responsibility is the middle.
## The language
| term | is |
|---|---|
| **source** | a repository, a path within it, and a ref — resolved to one commit ([ADR 0069](../../02-DECISIONS/0069-a-module-is-a-repository-and-a-path.md)) |
| **recipe** | how *one* artifact is produced from that source |
| **toolchain** | what a recipe runs inside — a compiler, a runtime, the SDK |
| **artifact** | what a recipe produced, named by the digest of its content |
| **publication** | putting an artifact where machines can fetch it, by that digest |
| **announcement** | telling the mesh what was built, and every artifact it stood on |
| **build** | one request and its outcome, correlated, recorded whether it worked or not |
## What the model is today, and where it does not fit
**A toolchain is not modelled at all** — it arrives as two build arguments the module's own
Dockerfile declares and the mesh fills in. **A language is not a concept.** And a recipe is
effectively singular: producing anything *compiled* means writing a Dockerfile.
**Archives already work, and that is the corrected half of this.** An earlier draft of this
document said the builder refused them. It does not: an archive is packed deterministically,
hashed, published by digest, fetched by the machine and unpacked. Only the *local* builder used at
genesis refuses one, deliberately — an archive is bytes that mean nothing until something serves
them, and there is no registry yet.
**What an archive cannot do is compile.** `from` names a directory and the directory is packed as
it stands, so shipping compiled output means compiling somewhere first — which means a Dockerfile,
which is the burden this is about. The gap is not the artifact kind. It is that **no recipe both
builds and packs**.
The cost is not theoretical. To add a module that carries its own code today, an author writes a
Dockerfile that: declares two `ARG` bases with no defaults; compiles under a specific working
directory so the SDK resolves upward; invokes the compiler *by absolute path*, because the usual
`node_modules/.bin` entry is a symlink that the base image's own assembly resolves away; copies the
output into a second stage; and sets an environment variable naming the compiled entrypoints. Every
module repeats it. Miss any step and the failure arrives somewhere else — as a crash loop, a
placeholder digest, a module that builds and does nothing.
The evidence that this is too hard is in the catalogue: **most modules are not converted**, and the
two converted during one session were each wrong twice before they were right, against a working
example sitting open in the next window.
**A module is not a container** ([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)) — it is
one piece of software and everything that makes it real: its provisioner, its tools, its hooks, its
scheduled steps, its event consumers. A build model whose only output is a container image is
modelling one of ten resource kinds and calling it the module.
## The model
**Recipe becomes explicit, and there is more than one kind.**
| recipe | produces | from |
|---|---|---|
| `image` | an image | a Dockerfile, when the software genuinely needs one |
| `archive` | a bundle, fetched by digest and unpacked | a directory, packed as it stands — **today** |
| `bundle` | the same, fetched and unpacked | this module's source, *compiled* by a toolchain and then packed — **the missing one** |
| `upstream` | a mirror | somebody else's pinned reference |
**Toolchain becomes explicit, and is derived rather than written.** A module says what it is written
in; the builder knows what that implies. The two base images stop being something an author names
and become something a toolchain *is*.
```
module says: language: typescript
builder knows: compile in the typescript toolchain, bundle, produce an archive
```
A Dockerfile remains available and stops being compulsory. It is the right answer for software that
needs a particular base, and the wrong answer for "compile my module's code", which is the same
operation every time.
**Why an archive and not always an image.** An archive is a content-addressed blob fetched over
plain HTTP and verified by its own digest, so it needs no registry account and no trusted transport
— it verifies itself. A container image is refused by a runtime over plain HTTP as *policy*, which
is why [issue 048](../../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)
exists. Modules that are the mesh's own code do not need a container's isolation from the mesh; they
need to run. Third-party software still arrives as an image, because that is how its author ships it.
## The invariants
1. **An artifact is named by the digest of its content.** Not by a tag, not by a path, never by a
placeholder. A recipe that cannot produce a digest has produced nothing.
2. **A build announces exactly what it made and everything it stood on.** The second half is what
makes build edges derived rather than declared ([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)).
3. **A failed build produces nothing and announces nothing.** A partial artifact under a digest the
mesh will later trust is worse than no artifact.
4. **A recipe with nowhere to publish refuses before it builds, not after.** An archive is bytes that
mean nothing until something serves them; producing one with no destination has produced nothing
usable, and saying so beats returning a path no other machine can read.
5. **A build is recorded whether or not it worked**, with everything it was told — the resolved
manifest, the path, what it stood on. A record that keeps only the outcome cannot be replayed to
anything that missed it ([issue 050](../../04-ISSUES/050-the-catalogue-knows-nothing-built-before-it/00-report.md)).
## What this costs, stated before it is chosen
**Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the
event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the
spine changes rarely but cascades when it does
([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)): with one language a contract
change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree
about the envelope fails by ignoring messages rather than by failing to compile.
**So the contracts have to stop being expressed twice before they are expressed four times.** They
are already: the manifest, declaration and link shapes exist as Go structs in the controller and
as TypeScript types in the SDK, kept in step by hand. Nobody has felt it because both live in one
repository. A second *language* makes that drift; a specified envelope and schema that every SDK
implements makes a second language an implementation rather than a translation.
**Order matters, then.** Language-neutral contracts, then a second language. The other way round
makes the drift worse while hiding it.
## How these rules are checked
| rule | checked by |
|---|---|
| A module with its own code needs no Dockerfile | A module declaring only a language builds, and its artifact is pinned to a digest the mesh's registry assigned. |
| An archive is produced and delivered | A module declaring an archive is built, and the machine assigned it has the unpacked files — verified on the machine, not in the build's own output. |
| A Dockerfile still works | A module that declares one builds exactly as before, because software that needs a particular base has not stopped existing. |
| Nothing is announced that was not made | A build made to fail announces nothing, and the catalogue's graph is unchanged afterwards. |
| A build keeps what it was told | A catalogue started after a build asks for what it missed and receives the manifest and the artifacts stood on, not a summary. |
| The toolchain is the mesh's, not the author's | Changing the toolchain makes every module built against it stale, and each names what moved. |
## The whole surface, in one place
**A reference, and it is kept true by a test rather than by care.** A table like this goes stale the
day somebody adds a field, so `mesh-catalog/modules/showcase` is a module that uses all of it, and
`TestTheShowcaseModuleIsAValidManifest` fails when it stops doing so. Read the module when this
disagrees with it.
### What a module declares
| field | is |
|---|---|
| `module`, `version`, `slug` | its identity; the slug is the short name generated names are built from |
| `capabilities` | what a machine must have for this to run there |
| `provides` | **the shared seat** — several modules may fill one capability and coexist |
| `claims` | **the exclusive seat** — two modules claiming one thing in a scope cannot both be assigned there |
| `serves` | what a consumer must know to connect. The mesh fills in the assigned port ([ADR 0038](../../02-DECISIONS/0038-the-mesh-assigns-the-port.md)) |
| `requires` | what must be provided by something on the same node |
| `binds` | where the mesh writes what a requirement resolved to |
| `secrets` | where the mesh seals the credential for a requirement |
| `own-secrets` | secrets that are the module's own — a superuser, a broker account |
| `emits` / `consumes` | the event graph: 1:many, broadcast, no credential |
| `listens` | the port **its own software** uses, and from where. Filtering is computed from these |
| `contributes` / `receives` | values one module adds to another's configuration, and the other half |
| `accesses` | operator-owned paths it may use and must not own ([ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md)) |
| `certificate` | a certificate for a name it serves |
| `grants` | credentials it must create for its consumers |
| `filtering` | rules beyond its own ports |
| `computed` | marks a module the controller generates rather than an author writing |
| `build.artifacts` | what it produces |
### What it builds
| kind | is |
|---|---|
| `bundle` | its own code, compiled by the toolchain its language implies, then packed |
| `archive` | a directory, packed as it stands |
| `image` | built from a Dockerfile — for software that needs a particular base |
| `upstream` | somebody else's image, mirrored and pinned by a digest this mesh assigned |
### What it puts on a machine
| resource | is | a module may |
|---|---|---|
| `directory` | a directory with a mode and an owner | ✅ |
| `file` | literal content, with `${bound:…}` and `${secret:…}` filled in | ✅ |
| `user` | a login | ✅ |
| `access` | a pre-existing path it may use and must not own | ✅ |
| `archive` | files fetched by digest and unpacked | ✅ |
| `package` | a package that must be present | ✅ |
| `network` | a named container network | ✅ |
| `container` | an image, in three modes | ✅ |
| `process` | **its own code**, in three modes | ✅ |
| `service` | an **existing** unit put into a state | for software shipping its own unit |
| `action` | a command to run | ❌ **refused** — [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
**The two at the bottom are the interesting rows.** `action` is refused outright: the link may not
carry a command, so a module needing something done ships a program that reconciles — which is what
a run-once `process` is. `service` installs no unit by design, which is right for software that
ships one and wrong for code the mesh built, which has no unit until the mesh writes it.
### How its code runs, and what that code can be
| mode | is | | shape | loaded by |
|---|---|---|---|---|
| *(default)* | a unit restarted when it exits | | tools | a tool host, over the broker |
| `run-once` | run to completion; what follows is gated on it | | event consumer | the same host, reacting |
| `schedule` | a timer; a missed fire happens when the machine returns | | provisioner | invoked when a consumer is granted |
| | | | a process | the machine's supervisor |
**Tools, hooks and consumers are not further modes**, which is the test of whether three is the
right number: they are loaded by a tool host, and a tool host is a process that stays up.
@@ -0,0 +1,186 @@
---
layer: to-be
status: proposed
code:
- mesh-sdk src
- mesh-tools src/broker-amqp.ts
- mesh-control internal/link
updated: 2026-09-15
decisions:
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
- 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
- 02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md
---
# The module protocol
**What a module's code and the mesh say to each other.** An SDK is an implementation of this in one
language and nothing more ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md)).
This is a specification, so it says what is required rather than how anything is arranged. Where it
describes current behaviour that is *not yet* specified-and-conformed, it says so.
## The shape of it
A **floor** every implementation needs, and three **capabilities** that are independent of each
other. An SDK implements the floor plus whatever capabilities it claims; a module is refused at
build time if it uses a capability its language's SDK does not implement.
| part | a module uses it to |
|---|---|
| **connection** | reach the broker as itself |
| **events** | emit, and react to what others emit |
| **tools** | answer questions asked of it |
| **provisioning** | give a consumer an instance of what it provides |
---
## The floor: connection
### The credential
A module is given a **sealed credential** as a file, and told where by its declaration. The
document:
| field | is | required |
|---|---|---|
| `url` | an `amqps://` URL carrying the account's user and password | yes |
| `fingerprint` | sha256 of the certificate the broker must present | yes for a scoped account |
| `node` | the machine this account was issued for | yes for a scoped account |
| `module` | the module this account was issued for | yes for a scoped account |
A plain string rather than a document is a **bootstrap URL** — unscoped, for the moment before a
mesh can issue anything. An implementation accepts both and must not treat the second as ordinary.
### Connecting
- The connection **pins the fingerprint**. It does not trust a certificate authority, and it does
not skip verification. A broker presenting a different certificate is refused, whatever else is
true of it.
- A scoped account **does not declare exchanges**. The foundation owns them; an account that may
declare one is an account that may create a parallel mesh by typo.
- An implementation **declares its own queue** and nothing else.
### Identity
**A module's node and module name come from its credential, never from its environment.**
This is not a convenience. It is what makes what a module emits match what the mesh authorised: an
environment variable can be set by anything on the machine, and a module that took its identity
from one could emit events attributing them to another module. Where an environment variable and
the credential disagree, the credential wins and the variable is overwritten.
---
## Capability: events
### The exchanges
| exchange | carries |
|---|---|
| `mesh.events` | every event |
| `mesh.events.dead` | what could not be handled |
### The queue
One **durable** queue per consumer, named `<node>.<module>.events`, with as many bindings as the
module has patterns. Durable because an event emitted while a module is restarting is exactly the
one that must not be lost.
**A message matching two bindings is delivered once**, so an implementation must match the routing
key against its own patterns locally to decide which handlers run. An implementation that ran every
handler whose exchange binding matched would run the wrong one.
### The envelope
Headers ride as AMQP headers. The body is JSON.
| header | is | required |
|---|---|---|
| `x-event-id` | a unique id, made by the emitter | yes |
| `x-source` | the module, context or node that emitted it | yes |
| `x-node` | the machine it was emitted from | yes |
| `x-time` | emit time, RFC-3339 | yes |
| `content-type` | always `application/json` | yes |
| `x-causation-id` | the event or command that caused this one | no |
| `x-schema` | a version of the body's shape | no |
**An unknown `x-` header is ignored, never refused.** An event is observed by parties that need not
all understand every header, and an implementation that refused one would make adding a header a
breaking change for everybody.
### Delivery
At-least-once. **Deduplication is on `x-event-id`**, which only the emitter can produce — a
consumer cannot tell a redelivery from a second event any other way.
### What is true, checked (2026-09-16)
Go emits all five required headers; the SDK requires exactly those. `x-causation-id` and `x-schema`
are **optional** — the SDK sets them when a handler has a causation or a schema, and reads them
back; a bare event carrying neither is correct. So the envelope agrees across the two
implementations. `x-schema` is available for versioning a body's shape and is set by whoever has a
version to declare.
---
## Capability: tools
A module's tools are its operator-facing surface.
- A tool is served from a **shared durable queue**, `serve.<key>`. Shared, so several runtimes
serving one tool compete for a call rather than each answering it.
- A call is request and reply. The reply returns through the RPC exchange `mesh.rpc`, keyed by the
caller's own reply queue — **not** through the default exchange, which would let a caller publish
into any queue on the broker.
- A caller needs a **reply queue**, and that is what a module's scoped account may not declare
([issue 049](../../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)).
So a module may serve tools and may not call them, and nothing today issues an account to anything
that wants to ask.
### Not yet true
The caller's half has no account. Until that is settled, the only thing that can ask a module a
question is the foundation's bootstrap admin, which is not a protocol so much as a way in.
---
## Capability: provisioning
A provider ships the provisioner that creates instances of what it offers
([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)).
| direction | carries |
|---|---|
| **grant, in** | the provision, who is asking — **a module on a machine**, not a machine — and what the consumer contributed |
| **credential, out** | the fields the provision promises a consumer |
**Who is asking is one thing with two parts.** A machine routinely runs several modules wanting the
same provision, so a grant addressed to a node alone does not name a consumer, and withdrawing one
would take another's away.
### Checked, and it agrees (2026-09-16)
This looked like the sharpest disagreement and was not one. The live wire is the contributions file
— `as`, `secret`, `node`, `at`, `values` — and it is the same on both sides. The types that
disagreed (`Grant`, `Interface` in the SDK's `contracts`) were dead: exported, imported by nothing,
describing fields the wire does not carry. They have been removed. The lesson kept: a type beside
the wire that has drifted from it is worse than none, which is why the wire is specified and
implementations are checked against it rather than trusted to still match a hand-kept shape.
---
## How an implementation is checked
Per capability, against fixtures rather than prose — a specification nobody can run is a document
two implementations drift from while both believe they conform.
| Rule | Checked by |
|---|---|
| The floor is the floor | Every implementation reads the same credential fixture, and refuses one whose fingerprint does not match what the broker presents. |
| Identity comes from the credential | A fixture sets an environment that disagrees with the credential; the emitted event carries the credential's. |
| The envelope is the envelope | An emitted event is compared header by header against a fixture; a missing required header fails, an unknown `x-` header is accepted. |
| Delivery is at-least-once | A fixture delivered twice is handled once. |
| A grant names a consumer | A grant fixture is read by every implementation and yields the same module and the same node. |
| A partial SDK is legitimate | An implementation claiming the floor and events passes those suites and is listed for them; a module using tools in that language is refused at build time with the reason. |
+179
View File
@@ -0,0 +1,179 @@
---
layer: to-be
status: proposed
code:
- mesh-catalog modules/showcase
- mesh-control internal/builder
- mesh-sdk src
updated: 2026-09-15
decisions:
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
- 02-DECISIONS/0040-what-a-module-is.md
- 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
---
# Writing a module
A worked guide. One module, four capabilities, four languages, and the packages it publishes.
The reference for *what can be said* is [`18-building-a-module`](18-building-a-module.md); the
reference for *what the code and the mesh say to each other* is
[`19-the-module-protocol`](19-the-module-protocol.md). This is how you actually write one.
## First: what is in the SDK, exactly
**The protocol, and nothing else** ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md)).
An SDK is an implementation of the module protocol in one language. If something is not in the
protocol it does not belong in an SDK, and that rule is what stops it becoming the 34,000-line
shared library this design exists to avoid ([ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md)).
Split the way the protocol is split:
| the SDK gives you | so that you can |
|---|---|
| **connection** — reads the sealed credential, pins the certificate, takes your identity from it | reach the broker as *this module on this machine*, and not be able to claim otherwise |
| **events** — emit, subscribe, the envelope, dedup | react to what happens in the mesh |
| **tools** — register and serve | be asked questions |
| **provisioning** — receive a grant, return a credential | give a consumer an instance of what you provide |
**What it does not give you, deliberately:**
- **No configuration loader.** Configuration arrives as files the mesh wrote and environment the
mesh set. Reading a file is not a thing an SDK needs to teach.
- **No API clients.** A Plex client belongs in the Plex module. It changes when Plex changes,
which has nothing to do with any other module — *frequent **and** cascading is the disease.*
- **No storage, no HTTP framework, no logging library.** Use the language's.
If you find yourself wanting to add something to the SDK, the test is ADR 0039's: **does editing it
recompile unrelated modules, and does it change often?** Both, and it does not belong.
## A module with four capabilities, in four languages
A module is **one piece of software** ([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)) and
may still be written in several languages — each artifact names its own, compiles alone and is
packed alone.
```
showcase/
module.json
events/ ← TypeScript: reacts to what the mesh does
tools/ ← Go: answers questions
provisioner/ ← Rust: grants instances of what it provides
ingest/ ← Python: a scheduled job
```
### The manifest
```json
{
"module": "showcase",
"build": { "artifacts": [
{ "name": "events", "kind": "bundle", "language": "typescript",
"entrypoints": ["index.js"] },
{ "name": "tools", "kind": "bundle", "language": "go",
"entrypoints": ["tools"] },
{ "name": "provisioner", "kind": "bundle", "language": "rust",
"entrypoints": ["provisioner"] },
{ "name": "ingest", "kind": "bundle", "language": "python",
"entrypoints": ["ingest.py"] }
]},
"resources": [
{ "id": "events", "type": "process", "name": "showcase-events",
"artifact": "events", "run": ["node", "index.js"] },
{ "id": "tools", "type": "process", "name": "showcase-tools",
"artifact": "tools", "run": ["./tools"] },
{ "id": "grants", "type": "process", "name": "showcase-grants",
"artifact": "provisioner", "run": ["./provisioner"] },
{ "id": "ingest", "type": "process", "name": "showcase-ingest",
"artifact": "ingest", "run": ["python", "ingest.py"],
"schedule": "0 3 * * *" }
]
}
```
**Four artifacts, four toolchains, four processes, one module.** Nothing here says how any of them
is hosted — that is the mesh's, and it is why these are `process` rather than four containers.
### Step 1 — say what it is, in each language's own terms
Each part is an ordinary project in its language, depending on the mesh SDK for that language the
way it would depend on anything:
```
events/package.json "@novox/mesh-sdk": "^1.2.0"
tools/go.mod require novox.example/mesh-sdk v1.2.0
provisioner/Cargo.toml mesh-sdk = "1.2"
ingest/pyproject.toml dependencies = ["mesh-sdk~=1.2"]
```
**Resolved from the mesh's own package registries**, which are the ordinary registries for each
ecosystem, hosted by the mesh. An author does nothing unusual: `npm install`, `go mod tidy`,
`cargo build`, `pip install` all work, and a developer's laptop resolves exactly what a build does.
### Step 2 — write each part against its capability
Each uses only the part of the protocol it needs. The event consumer never learns what a grant is.
```
events/index.ts on("module.builder.built", …) → the events capability
tools/main.go tool("showcase_state", …) → the tools capability
provisioner/main.rs grant → credential → the provisioning capability
ingest/ingest.py emit("module.showcase.ingested", …) → events, emitting only
```
### Step 3 — the mesh does the rest
You do not write a Dockerfile, a unit file, or a port number. The builder picks the toolchain from
the language, compiles each artifact alone, and publishes it. The controller assigns the machine
and the ports; the host writes the units.
### What it costs you to use four languages
**Honestly: four sets of dependencies to keep current, and four SDKs that must agree.** The mesh
makes it possible, not free. A module in one language is simpler, and the reason to use four is
that one of them genuinely suits a part better — not that you can.
## Publishing a package is a capability
A module may publish libraries as well as run code. **The SDK is not special; it is simply the first
module that did this**, and its own consumers are the mesh's modules.
```json
{
"module": "plex",
"build": { "artifacts": [
{ "name": "server", "kind": "upstream", "from": "plexinc/pms-docker@sha256:…" },
{ "name": "client-ts", "kind": "package", "language": "typescript", "from": "clients/typescript" },
{ "name": "client-rust", "kind": "package", "language": "rust", "from": "clients/rust" },
{ "name": "client-py", "kind": "package", "language": "python", "from": "clients/python" }
]}
}
```
A `package` artifact is built and **published to the mesh's registry for that ecosystem**, under the
version its own project file declares. Another module then depends on it the ordinary way:
```
"@novox/plex-client": "^2.0.0"
```
**Why this belongs to modules rather than being a separate thing.** A client for a piece of software
changes when that software changes, and the module that owns the software is the only thing that
knows. Putting the Plex client anywhere else is the shared-library disease with extra steps.
### What this leaves open
- **Which private registry.** That there *is* one is settled —
[ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) says each module consumes its dependencies
from the private registry, the mesh's own shared library included. Which software serves it is
not: the catalogue holds `verdaccio`, and a git host usually serves package registries too.
- **And the bootstrap does not have one.** ADR 0014 assumes a registry exists; on a fresh mesh
nothing has installed one when the first SDK is built. That is the same pivot as everything else
and it has not been designed.
- **Who may publish.** A builder pushing a package needs an account on that registry, which is a
credential in the bootstrap path and does not exist yet.
- **Versions and ranges.** Everything else the mesh delivers is pinned by digest, and a range is
resolved at build time from whatever the registry holds. A lock file records what was chosen, and
the builder records that as the build edge — but *a mesh that can rebuild a commit and get a
different library* is a real change from how everything else here works, and it should be a
decision rather than a consequence.
@@ -0,0 +1,178 @@
---
layer: to-be
status: proposed
code:
- mesh-host internal/bootstrap
- mesh-host cmd/mesh-bootstrap
- mesh-lab test/integration/one-node-mesh.test.ts
updated: 2026-09-15
decisions:
- 02-DECISIONS/0067-genesis-is-a-pivot.md
- 02-DECISIONS/0073-the-installer-carries-a-builder.md
- 02-DECISIONS/0071-where-genesis-gets-its-source.md
- 02-DECISIONS/0014-no-npm-workspace.md
---
# The installation, in full
Every step from a machine with nothing to a mesh that maintains itself. Written out explicitly
because it is the procedure everything else depends on, and because the parts that do not exist
yet are easier to see beside the parts that do.
[`17-raising-a-mesh`](17-raising-a-mesh.md) argues *why* it is shaped this way. This says *what
happens*, in order, with each step's name as the installer prints it.
## What must be true before anything starts
| | why |
|---|---|
| a container runtime | the foundation is containers, and the installer refuses without one |
| the host binary, where the installer expects it | it is what the machine becomes |
| a repository and a commit to build from | the installer carries a builder, not a controller, so it must be told what to make ([ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md)) |
| a way out to the internet | the store, the broker and the registry are pulled from it |
| a reachable git host, and a commit it serves | what is cloned is the trust anchor for everything this mesh will ever run ([ADR 0071](../../02-DECISIONS/0071-where-genesis-gets-its-source.md)) |
| the address other machines will reach this one on | a token carries it verbatim; the installer refuses to guess |
## Phase one — a machine becomes a mesh of one
Twelve steps, run by one program, each safe to run again.
| # | step | what happens | true afterwards |
|---|---|---|---|
| 1 | `preflight` | everything above is checked | the machine is not half-changed by a missing prerequisite |
| 2 | `load` | the carried builder image is loaded | the mesh has the one thing it cannot fetch — the thing that does the fetching |
| 3 | `build` | the builder clones the named repository at the named commit and **builds the controller** | what will run is something this mesh made and can make again |
| 4 | `bundle` | the foundation template is written out, with the built controller's id in place of the placeholder | the machine has a description of what it will become |
| 5 | `apply` | store, broker, schemas, and a **temporary** controller are raised | a mesh of one exists and answers |
| 6 | `verify` | the controller is asked, rather than assumed | it replies, and says it has no machines |
| 7 | `enrol` | the machine joins the mesh it is itself running | the mesh has one node, and an agent runs on it |
| 8 | `registry` | the mesh's own artifact store is installed | there is somewhere to keep what this mesh makes |
| 9 | `publish` | the controller's image is pushed into it | the image has a digest something other than itself assigned |
| 10 | `controller` | it is installed again, as an ordinary module pinned to that digest | what runs is a module like any other |
| 11 | `retire` | the temporary controller is dropped | **the pivot is complete** — what raised the mesh is gone |
| 12 | `builder` | the carried builder is published, installed as a module, and issued a broker account | the mesh can produce |
**Steps 8 to 11 are the pivot** ([ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md)). Before
them the controller is something the installer put there; after them it is something the mesh
holds a record of and can upgrade. The account in step 12 is issued *before* the machine is sent
anything, because a builder that arrives without its credential starts, finds nothing it may read,
and waits — which looks exactly like a builder with no work.
## Phase two — a mesh of one becomes a mesh that works
**Genesis ends with a mesh that runs, which is not the same as a mesh that works.** It has a control
plane, a store, a broker, a registry and a builder. It holds no module graph, has no private
network, no packet filter, and cannot resolve a name. Calling that "installed" is what let the
catalogue be missing from a test for weeks without anything complaining.
| # | step | what happens | why it is here |
|---|---|---|---|
| 13 | the shared base is built | the toolchain and runtime every module with code of its own stands on | nothing else with code can be built until it exists |
| 14 | a store module is built and run | a database **provider**, which the foundation's store is not | the foundation's store is the controller's own memory, and offers nothing to anything |
| 15 | the catalogue is built and run | the module graph | without it the mesh cannot say what it holds, what a change reaches, or what must be rebuilt |
| 16 | the catalogue asks for what it missed | the builds made before it existed are replayed | on a fresh mesh those are always the base, the store and the catalogue itself ([issue 050](../../04-ISSUES/050-the-catalogue-knows-nothing-built-before-it/00-report.md)) |
| 17 | the controller is rebuilt from its own repository | and rolled out through the module path | the moment the mesh stops depending on the installer for anything |
| 18 | `networking` is assigned **and the node placed** | a private network, and names | assigning installs the module; placing says where this machine is on it. Both, or the names file is written empty |
| 19 | the packet filter is assigned | rules generated from what modules declared | until this, every rule the mesh computes has never been applied to anything |
## Phase three — machines arrive
Only now. A machine joining a mesh that cannot build anything proves that enrolment works, which
was never the doubtful part.
| # | step | what happens |
|---|---|---|
| 20 | a node record is made and a token issued | one-time, carrying the broker's address and the fingerprint to pin |
| 21 | the machine enrols and runs its agent | being heard from once is not an agent running; both are checked |
| 22 | it is placed on the private network | or nothing can bind a consumer on it to a provider elsewhere |
| 23 | modules are assigned to it | and it pulls what it needs from the mesh's registry |
## What is not yet true
Stated plainly, because a procedure that implies otherwise is worse than none.
**Steps 13 to 19 are not the installer's.** They are things somebody types. The installer ends at
step 12, and everything that turns a running mesh into a working one is manual — which is why a
test had to be written to find out they were missing.
**Step 23 does not work for a second machine.** It has no account on the registry
([issue 042](../../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)) and
no reason to trust a registry serving plain HTTP over the network
([issue 048](../../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)).
Both are invisible on a mesh of one, where the registry is loopback.
**There is no private package registry**, and [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md)
assumes one: a module consumes its dependencies, the mesh's own shared library included, from it.
At step 13 nothing has installed one, so the first build of the shared base resolves the SDK some
other way — today by a git URL, which is
[issue 053](../../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md).
**The installer does not install the host's unit, though one exists.** `mesh-host/packaging/` ships
`nox-mesh-host.service` and two companions; the installer declines to place them because a unit file
is a packaging decision. So an install that does nothing further leaves a machine whose containers
come back after a reboot and whose agent does not — it runs the right things and can no longer be
told anything.
**This is a gap in packaging, not in the mesh**, and the distinction matters: the lab's
`--host-in-background` says in its own help that it does not survive a reboot, so a lab run failing
this is the lab being honest rather than the mesh being broken. What is missing is the step that
puts the shipped unit on the machine.
**Nothing asserts a mesh was installed this way.** A claim that a machine was brought up by this
procedure cannot be contradicted by anything afterwards.
## What a finished mesh holds
**Twelve, and after the pivot none of them is a specialty.** Every row is a module the mesh built,
holds a version of, and can upgrade — which is the whole claim, and is not true today for the first
two.
| # | module | provides | note |
|---|---|---|---|
| 1 | `postgres` | `postgres-database` | **the controller's own records and every module's.** One server, not two |
| 2 | `lavinmq` | `amqp` | the broker every node dials, and what modules are granted vhosts on |
| 3 | `mesh-control` | *claims* `the-controller` | decides what runs where |
| 4 | `distribution` | `artifact-store` | what the mesh built, pinned by digest — the module is the software (Distribution), the provision is the job |
| 5 | `builder` | — | turns source into artifacts |
| 6 | `mesh-tools` | build inputs | the base everything with code compiles against. **Runs nowhere** |
| 7 | `mesh-catalog` | the module graph | what is held, what a change reaches, what must be rebuilt |
| 8 | `networking` | `private-network`, naming | requirements only — assigning it brings `mesh-wireguard` and `mesh-names` |
| 9 | `dnsmasq` + one of `resolved-split-dns` / `resolv-conf` | `wildcard-resolution` | names that actually resolve, on top of `mesh-resolver`'s data |
| 10 | `step-ca` | `acme-ca` | certificates for `.internal` |
| 11 | `firewall` | *claims* `the-packet-filter` | rules generated from what modules declared |
| 12 | `gitea` | `package-registry` | where a module's dependencies come from ([ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md)), and the git host |
### Why it is twelve and not thirteen
**The foundation's store and the `postgres` module are the same module.** They were two rows while the
foundation was a different *kind* of thing: a store raised from a bundle cannot provide
`postgres-database`, so anything wanting a database needed a second server. That is visible on any
mesh built today — `mesh-store` and `postgres`, two containers, **the same image**.
The naming rule settles which name survives
([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)):
> Where the consumer **speaks a protocol** — a database's wire protocol and query dialect — the
> interface *is* the protocol. **"database" is not a capability; the protocol is.** Never false
> genericity: a name must not promise a swap the contract cannot deliver.
So there is no `store` module. The controller is coupled to postgres — its own queries use
`distinct on` and `on conflict`, which are not portable — and calling it `store` would advertise a
swap that would fail the first time somebody tried it.
**The same applies to the broker**, with a different outcome: `amqp` *is* a protocol and more than
one implementation speaks it, so `amqp` is a legitimate provision and `lavinmq` is one provider of
it. The foundation's broker and the `lavinmq` module collapse the same way.
### What this costs, and it is the last specialty
Adopting the store and the broker as modules is [issue 051](../../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md),
and it is the only part of this that has not been designed. The two hard parts:
- **upgrading a store the controller is reading from** — a rollout where the thing being replaced
is the thing holding the record of the rollout
- **upgrading a broker over the broker** — the instruction arrives on what it replaces, so the
machine must finish without being able to report progress
Both are operations with windows, and a machine rebooting inside one is an operator's problem during
an operation rather than a reason not to do it.
+118
View File
@@ -0,0 +1,118 @@
---
layer: to-be
status: in-progress
updated: 2026-09-15
decisions:
- 02-DECISIONS/0067-genesis-is-a-pivot.md
- 02-DECISIONS/0074-the-wire-is-specified-not-the-types.md
- 02-DECISIONS/0075-two-stores-and-which-provides-what.md
- 02-DECISIONS/0014-no-npm-workspace.md
---
# The work ahead
Everything decided this cycle and not yet built, in the order its dependencies allow. Each phase
ends at something provable on a running mesh, because a phase that ends at a claim is a phase that
went missing without anything complaining.
## How this is built, and when it is run
**Implemented as code with unit tests, committed per change, and run in the lab ONCE the pieces that
would change the outcome are in place.** The mistake this corrects: repeatedly running a 20-minute
lab against a mesh mid-transformation, debugging paths the next phase deletes. The base-build hang,
for instance, is almost certainly the SDK resolving from a git URL inside a docker build (issue
053) — which Phase 2 removes. Debugging it on the current shape is debugging deprecated code.
So the lab run is the acceptance test at the end of the assembled work, not the tool for finding
each bug. Where a fault can be reasoned out of the code path, it is — reading, not running.
## Phase 0 — folded in
The installer's own regressions (stale carried builder, a diagnostic on the parsed stdout) are
fixed and committed. Whether it reaches a full green run is answered by the final lab run below,
after the phases that change its build path are in — not before.
## Phase 1 — the protocol is one thing, and correct *(mostly done: the drift was dead types)*
**Why here.** The Go controller and the TypeScript SDK disagree about what a grant carries
(`consumer` is the module in one, the node in the other). That is exercised by the installer's own
provisioning — the catalogue's database — so it belongs before more is built on it.
- [x] 1.1 the drift was not live — inspection showed the wire agrees (contributions file; envelope
required headers). The dead types that disagreed are removed, ADR 0074 and doc 19 corrected
- [ ] 1.2 a conformance fixture for the two live cross-language contracts — the event envelope and
the contributions file — checked in both suites, as **prevention** rather than repair
- [ ] 1.3 (deferred) a full per-capability suite when a third language is actually added; not
needed to keep two honest
**Done when.** A fixture pins the envelope and the contributions file, and a change to either side
that breaks agreement fails a test rather than a mesh.
## Phase 2 — the private package registry, and the SDK in it
**Why here.** ADR 0014 says a module consumes its dependencies, the SDK included, from the private
registry. Nothing installs one, so the SDK comes from a git URL (issue 053). Needs the installer
(Phase 0) and gitea.
- [ ] 2.1 gitea provides `package-registry` in full — the endpoints, an account the builder may
publish with
- [ ] 2.2 the builder publishes the SDK there on build, by version
- [ ] 2.3 modules consume it by version; the git URL and the sibling-path lock are gone
- [ ] 2.4 a second language's SDK published the same way, proving the path is not TypeScript-only
**Done when.** A module builds against the SDK resolved from the mesh's own registry, and issue 053
closes.
**Where it stands (2026-09-16).** The decision the bootstrap turned on is settled and recorded
([ADR 0076](../../02-DECISIONS/0076-the-sdk-is-a-published-package.md)): the SDK is a published
package, built on a public base image and published before the toolchain that consumes it; mesh-tools
stays the thin toolchain base but `npm ci`s the SDK by version. On the code, the builder now resolves
a package-registry credential — from a binding the mesh writes or from the environment for a hand-run
or bootstrap build — and injects it into an image build as a buildkit secret, never a layer, so a
token is not baked into the toolchain image. Unit-tested. Still ahead: gitea serving the registry in
full with a provisioner that mints tokens (2.1), the SDK built and published by the mesh (2.2), the
mesh-tools manifest flipped off the git URL to `npm ci` by version (2.3), and the genesis step that
raises gitea and publishes the SDK before the base build. Those close together in one lab run.
## Phase 3 — nothing is special after installation
**Why last.** The hardest and riskiest, and it needs everything above: an installer that completes,
a protocol that agrees, a registry to publish to. Issue 051.
- [x] 3.1 the store is adopted — raised at genesis, then held as the `postgres` module; one server,
the controller's records and the database of each module that asks for one. *(Adopted in place: the module's
server names the same container and the same pinned upstream image the foundation runs, so the
applier reconciles it rather than raising a second postgres. Proven 22/22 in the one-node lab —
the catalogue, a `postgres-database` consumer, gets its database from it. Follow-up: the store
binds `0.0.0.0` from genesis so a mesh consumer can reach it, but the packet filter is
installed later — a brief pre-filter window where `mesh-store` is open before `from: mesh`
clamps it; bring the filter up earlier or bind narrower at genesis.)*
- [x] 3.2 the broker is adopted — one `lavinmq`, the `/` vhost for the mesh bus, a vhost per
consumer that requires `amqp`; the second server gone. *(Adopted in place like the store; the
module names mesh-broker with the foundation's TLS spec, the provisioner runs host-networked
as lavinmq's default guest, amqp-ping reaches it. Proven 22/22.)*
- [x] 3.3 an upgrade of each, proven: a store the controller reads from, a broker over the
broker, each with a stated window. *(Lab steps S1/S2: move each module's source, build, roll
out, then a full push applies the server change and recreates the container. The store's
window is a pool reconnect; the broker's is longer — recreating the bus the push travels over,
so the mesh reconnects to the one that returns. Data survives on the named volumes.)*
- [x] 3.4 `status` can say the foundation is behind its source, which today it cannot form.
*(Given by the adoption: postgres/lavinmq are ordinary modules with a source now, so
`module list`/`status` reports them behind or current like any other — the question could
not form when they were bundle containers.)*
**Done when.** A mesh built from bare metal runs one postgres and one lavinmq, and can upgrade
either — so the twelve-module floor has no specialty left in it.
## The through-line
Each phase leaves the mesh more able to describe and rebuild itself, and each ends at a run rather
than a paragraph. The order is dependency, not preference: the installer carries everything, the
protocol is what everything speaks, the registry is what everything is built from, and adoption is
what makes the last two things ordinary.
## The one lab run
After Phases 1–3 are implemented and unit-tested and committed: rebuild the images, **verify each
carries its change**, and run the one-node installer once. All 22 checks green, twice, is the
acceptance test for the whole cycle — not a debugging loop, a proof that the assembled thing works.
+7 -2
View File
@@ -15,8 +15,8 @@ document is written and this one's status becomes `implemented`.
| [`03-scenario-lifecycle.md`](03-scenario-lifecycle.md) | What happens to a scenario — raise, snapshot, restore, move, destroy | [ADR 0016](../../02-DECISIONS/0016-the-lab.md) | | [`03-scenario-lifecycle.md`](03-scenario-lifecycle.md) | What happens to a scenario — raise, snapshot, restore, move, destroy | [ADR 0016](../../02-DECISIONS/0016-the-lab.md) |
| [`04-lab-installation.md`](04-lab-installation.md) | Getting the lab onto a clean machine, and why it verifies capability rather than installation | [ADR 0010](../../02-DECISIONS/0010-delivery.md) | | [`04-lab-installation.md`](04-lab-installation.md) | Getting the lab onto a clean machine, and why it verifies capability rather than installation | [ADR 0010](../../02-DECISIONS/0010-delivery.md) |
| [`05-the-node-host.md`](05-the-node-host.md) | Tier 0 — the one thing installed by hand, and the only thing that changes a machine | [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) | | [`05-the-node-host.md`](05-the-node-host.md) | Tier 0 — the one thing installed by hand, and the only thing that changes a machine | [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
| [`06-the-control-plane.md`](06-the-control-plane.md) | Tier 2 — what the term means, and the test for what belongs in it | [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) | | [`06-the-controller.md`](06-the-controller.md) | Tier 2 — what the term means, and the test for what belongs in it | [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0048](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) | | [`07-the-foundation.md`](07-the-foundation.md) | Tier 1 — what the controller consumes and cannot grant itself | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0048](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) |
| [`08-connectivity.md`](08-connectivity.md) | One context in full — overlay, resolution, exposure, filtering, certificates | [ADR 0007](../../02-DECISIONS/0007-connectivity.md), [0050](../../02-DECISIONS/0007-connectivity.md), [0051](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0055](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) | | [`08-connectivity.md`](08-connectivity.md) | One context in full — overlay, resolution, exposure, filtering, certificates | [ADR 0007](../../02-DECISIONS/0007-connectivity.md), [0050](../../02-DECISIONS/0007-connectivity.md), [0051](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0055](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) |
| [`09-the-node-lifecycle.md`](09-the-node-lifecycle.md) | How a machine becomes a node, stays one, and stops being one | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0051](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) | | [`09-the-node-lifecycle.md`](09-the-node-lifecycle.md) | How a machine becomes a node, stays one, and stops being one | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0051](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) |
| [`10-delivery.md`](10-delivery.md) | Modules, the three edges, and how a change becomes a running thing | [ADR 0010](../../02-DECISIONS/0010-delivery.md), [0064](../../02-DECISIONS/0009-modules-and-the-graph.md), [0065](../../02-DECISIONS/0009-modules-and-the-graph.md) | | [`10-delivery.md`](10-delivery.md) | Modules, the three edges, and how a change becomes a running thing | [ADR 0010](../../02-DECISIONS/0010-delivery.md), [0064](../../02-DECISIONS/0009-modules-and-the-graph.md), [0065](../../02-DECISIONS/0009-modules-and-the-graph.md) |
@@ -27,6 +27,11 @@ document is written and this one's status becomes `implemented`.
| [`15-the-agent-session.md`](15-the-agent-session.md) | One mechanism started twice — a node's session and the mesh's | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) | | [`15-the-agent-session.md`](15-the-agent-session.md) | One mechanism started twice — a node's session and the mesh's | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) |
| [`16-module-coverage.md`](16-module-coverage.md) | What a module must be able to say, measured against 127 that exist | [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md), [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) | | [`16-module-coverage.md`](16-module-coverage.md) | What a module must be able to say, measured against 127 that exist | [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md), [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
| [`17-raising-a-mesh.md`](17-raising-a-mesh.md) | How a mesh comes into existence, and how a machine joins one that exists | [ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md), [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) | | [`17-raising-a-mesh.md`](17-raising-a-mesh.md) | How a mesh comes into existence, and how a machine joins one that exists | [ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md), [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
| [`18-building-a-module.md`](18-building-a-module.md) | How a build is modelled, and why a recipe that is always a Dockerfile does not fit what a module is | [ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md), [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md), [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) |
| [`19-the-module-protocol.md`](19-the-module-protocol.md) | What a module's code and the mesh say to each other; an SDK is an implementation of it | [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md), [ADR 0042](../../02-DECISIONS/0042-the-shape-of-an-event-on-the-wire.md), [ADR 0043](../../02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) |
| [`20-writing-a-module.md`](20-writing-a-module.md) | A worked guide: one module, four capabilities, four languages, and the packages it publishes | [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md), [ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md), [ADR 0039](../../02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md) |
| [`21-the-installation-in-full.md`](21-the-installation-in-full.md) | Every step from a bare machine to a mesh that maintains itself, and what is not yet true | [ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md), [ADR 0073](../../02-DECISIONS/0073-the-installer-carries-a-builder.md), [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) |
| [`22-the-work-ahead.md`](22-the-work-ahead.md) | Everything decided and not yet built, in dependency order, each phase ending at a run | [ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md), [ADR 0075](../../02-DECISIONS/0075-two-stores-and-which-provides-what.md), [ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) |
## Not yet written ## Not yet written
@@ -0,0 +1,80 @@
---
status: open
opened: 2026-09-14
located-in: []
fixed-by:
amended-design:
---
# 048 — Nothing makes a machine trust the mesh's own registry
## Symptom
The registry module says, in its own manifest, that its port is open to the mesh because *every
machine pulls images and artifacts from here*. A machine that tries to do so is refused by its
container runtime before the mesh is involved at all: the registry serves plain HTTP, and a
container runtime treats any registry that is not loopback as HTTPS.
Nothing in the mesh arranges otherwise. There is no resource that configures a runtime's view of
the registry, no field in a manifest for it, and no step in joining that establishes it.
**It has never failed, and the reason it has never failed is the finding.** Every proof that a
machine can fetch a mesh-built artifact has been a proof about the machine that built it, where the
reference was loopback and loopback is trusted by default. The one bed that pushes to a routable
address gets away with it because *the lab* writes the runtime's configuration before the mesh is
raised, naming the documentation ranges its scenarios use. That file is the lab's, not the mesh's,
and no machine outside a bed has one.
Observed while fixing a neighbouring fault: the builder recorded artifacts under a loopback address
(fixed — the binding states where the provider is, and it now uses it). Correcting the address is
what makes this reachable, and therefore what makes this visible.
## Why this matters
**A stated rule is enforced by nothing.** The registry declares that the whole mesh pulls from it
and opens a port to the mesh accordingly. That is the design. Whether any machine can act on it
depends on a file the mesh does not write, does not read, and has no opinion about.
**It is the other half of [042](../042-nothing-gives-a-node-an-account-for-a-registry/00-report.md),
and the two are easy to confuse.** 042 is about a machine not being *allowed* to pull — no
credential. This is about a machine not being *able* to, regardless of credential, because the
transport is refused. A fix for 042 alone would leave a machine holding a valid account for a
registry its runtime will not talk to, which fails with an error about certificates and reads as a
credential problem.
**It decides something about the private network that has not been decided.** Serving artifacts
over plain HTTP is defensible if the private network is the boundary, and indefensible if it is
not — and either way it is a position, not an omission. Today it is an omission that happens to
work in one place.
**It is a joining problem before it is anything else.** The first machine never meets it. Every
machine after it does, on the first module the mesh built rather than fetched — which is most of
them.
## Evidence
A machine raised by the installer, asked what its runtime trusts:
```
Insecure Registries:
127.0.0.0/8
<the documentation ranges this lab uses>
::1/128
```
Everything but the loopback entry was written by the harness. The mesh contributed none of it.
## Open questions
- **Should the mesh's registry serve TLS?** It would make the transport a property of the registry
rather than of every machine that talks to it, and the mesh already issues certificates. What
signs it, and what a machine checks it against, are the real questions.
- **Or should a machine's runtime be configured by the mesh** — a resource that states what this
machine trusts, applied like any other? That puts a node-wide setting in a module's hands, which
may be the wrong ownership.
- **What is the boundary the answer assumes?** If plain HTTP inside the private network is
acceptable, that assumption should be recorded with what it takes for granted about who is on
that network — and checked, rather than inherited from a harness.
- **How does this interact with genesis?** The installer publishes before the mesh can grant or
configure anything, and gets away with loopback. Whatever is decided has to leave that moment
workable.
@@ -0,0 +1,72 @@
---
status: open
opened: 2026-09-14
located-in: []
fixed-by:
amended-design:
---
# 049 — A module can serve tools, and nothing is allowed to call them
## Symptom
A module serves tools over the broker. Asking it one, from inside that same module's own container,
using its own credential, is refused:
```
Error: Channel closed by server: 403 (ACCESS-REFUSED) with message
"ACCESS_REFUSED - User '<node>-<module>' doesn't have permissions to queue 'amq.gen-...'"
```
A module's broker account is scoped to what it declares it emits and consumes. A tool call is a
request and a reply, and the reply arrives on a temporary queue the caller creates — which that
scope does not cover, and should not: a module that only publishes events has no business declaring
queues.
So the account is right and the request is reasonable, and there is no account that can make it.
## Why this matters
**A module's tools are its operator-facing surface, and nothing can reach it.** The catalogue serves
five: what this mesh holds, what a module is made of, what provides a given provision, what depends
on a module, and what is stale. Those are the questions the catalogue exists to answer, and today
the answer to "how does anyone ask one" is that they run a container joined to the broker's network
namespace and authenticate as the substrate's bootstrap admin.
**It is why a running module gets mistaken for a working one.** A test that cannot ask a module
anything checks that its container is up, and a container being up is not a claim about function.
That substitution has now hidden two separate faults in one week — a crash-looping runtime beside a
healthy container of a similar name, and a catalogue that was never installed at all.
**The mesh already has the shape for this.** An account scoped to a purpose, minted by the mesh and
sealed to a holder, is exactly what provisioning does. What is missing is the recognition that
*asking* is a use of a module, the way pulling is a use of the artifact store
([042](../042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)) rather than something
that happens beneath it.
## Evidence
The refusal above, from a module invoking its own tool with the credential the mesh gave it. The
working alternative, which is the measure of the gap:
```
docker run --rm --network container:<the broker> \
-e MESH_BROKER_URL=amqp://<bootstrap admin>@127.0.0.1:<port>/ \
<the module's image> invoke <module> <tool> '{}'
```
That is the bootstrap credential, host-local, with no scope at all. It works, and nothing about it
should be how a mesh is asked a question.
## Open questions
- **Who is the caller?** An operator at a terminal, an agent acting for one, and another module are
three different holders with three different scopes, and only the last resembles anything the
mesh mints today.
- **Should calling be a grant like any other** — a module declares it may be asked, and a consumer
is issued an account that may create a reply queue and publish to that module's request queue?
- **Should the control plane be the way in?** It holds an admin connection already, and a
`mesh-control` subcommand for asking a module a question would need no new account — at the cost
of routing every question through one process and its privileges.
- **What does a module declare about being asked?** Serving a tool is already declared. Whether it
may be asked by anyone who can reach the broker, or only by named holders, is not.
@@ -0,0 +1,66 @@
---
status: open
opened: 2026-09-14
located-in: []
fixed-by:
amended-design:
---
# 050 — The catalogue knows nothing that was built before it
## Symptom
A mesh raised from bare metal built six modules in order: the shared base, a store, the catalogue,
the control plane, a module, and a broker. Asked afterwards what it holds, the catalogue answered
with three — exactly the three built *after* it started running:
```
amqp-ping built from <the catalogue repository>
lavinmq built from <the catalogue repository>
mesh-control built from <the control plane repository>
```
Missing: the shared base, the store, and the catalogue itself.
The catalogue learns what exists by consuming the builder's announcement over the broker. Anything
built before it was running was announced to nobody, and nothing goes back for it.
## Why this matters
**The modules it cannot see are the ones a mesh is made of.** On a fresh mesh the things built
before the catalogue are, necessarily, the things the catalogue needed in order to exist: the base
it was compiled against and the store it keeps its records in. So the hole is not random — it is
always the foundation, on every mesh, at exactly the moment the graph is first populated.
**It breaks the question the catalogue exists to answer.** Build edges are derived facts between
versions, and the base is the node nearly every other module hangs off. A catalogue with no record
of the base cannot know that moving it makes everything standing on it stale, so `catalog_stale`
answers confidently and wrongly — the worst shape for a question about what to rebuild.
**Nothing reports the gap.** The catalogue does not know what it was not told, so it does not say
"three of six" — it says three, and a reader with no independent count believes it. This was found
by asking it a question and comparing the answer against what the mesh had just been watched doing.
**The record is not lost, only the catalogue's copy.** The control plane holds every build it
ordered, with commit, repository, path and resolved artifacts. So this is a gap between two stores
that should agree, rather than information nobody has.
## Open questions
- **Should the catalogue backfill on start** — ask the control plane for the builds it already
recorded and take them in? That fixes the fresh-mesh case and every case where the catalogue was
down while something was built, which is the same fault with a different cause.
- **Or should announcements be durable**, so a build announced while nothing was listening is
delivered when something is? That treats the catalogue as one consumer among several, which it
already is.
- **Or should the control plane be the record and the catalogue a projection of it?** Two stores
that must agree, kept in step by events, is the arrangement that produced this.
- **How would anyone notice next time?** The catalogue cannot compare itself to a source of truth
it does not have. Whatever is chosen, something has to be able to say "these disagree".
## How this would be checked
| Rule | Checked by |
|---|---|
| The catalogue holds every module the mesh built | A mesh is raised from bare metal and the catalogue is asked; its list is compared against the control plane's build records, and a module in one and not the other is a failure. |
| A change to the base makes what stands on it stale | The base is rebuilt on a fresh mesh and the catalogue names the modules that must follow — which requires it to know the base exists. |
@@ -0,0 +1,133 @@
---
status: fixed
opened: 2026-09-14
located-in: [mesh-host, mesh-catalog]
fixed-by: mesh-host 56124c3; mesh-catalog 5e4dc37; mesh-lab 440e265
amended-design:
---
# 051 — The mesh can update everything except what it depends on
## Symptom
A mesh raised from bare metal runs its store and its broker from a bundle the installer wrote once,
at install time, with the images pinned in it. Nothing can change them afterwards. There is no
build for them, no version for them to be behind, no `upgrade … roll-out`, and no way for the mesh
to report that either is out of date — because the mesh holds no record of them as modules at all.
Every other thing on that machine has all of it.
Observed on a one-node mesh, where the duplication makes it plain — the same image, twice:
```
mesh-store <the postgres image> the control plane's own records
postgres <the same digest> the postgres module's server
mesh-postgres <mesh-built runtime> that module's provisioner and tools
```
One of those two servers can be rebuilt from source and rolled out. The other cannot, and it is the
one holding the mesh's inventory, identity and licences.
## Why this matters
**It is exactly backwards.** The store and the broker are the components every other thing depends
on, so they are the ones whose security updates matter most, and they are the only ones the mesh
has no mechanism to deliver. A release that reaches every module's database does not reach the one
the control plane keeps its own records in.
**It is invisible rather than reported.** `status` says what is behind its source. The substrate
cannot be behind anything, because the mesh does not know it exists as something with a source. So
a mesh with a year-old broker reports itself entirely current, which is worse than reporting a
problem.
**The pattern that would fix it already exists and is proven.** The control plane is carried in,
raised, and then adopted as an ordinary module pinned to the image that is running — the pivot in
[ADR 0067](../../02-DECISIONS/0067-genesis-is-a-pivot.md). Nothing about that pattern is specific
to the control plane. Applied to the store and the broker it would make the substrate a *moment*
rather than a *kind*: how a mesh starts, not what it permanently is.
**It is also why the same image runs twice.** A store that cannot be a module cannot provide
`postgres-database`, so a module wanting a database needs a second server. Adoption removes the
duplication as a side effect, and the control plane already reaches its three contexts through
three separate credentials — which is the shape of a consumer, not an owner.
**And the module it becomes is `postgres`, not `store`.** The naming rule settles it
([ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md)): where a consumer speaks a protocol, the
interface *is* the protocol, and *"database" is not a capability*. The control plane's own queries
use `distinct on` and `on conflict`, so the coupling is to postgres and a `store` module would
promise a swap that fails the first time anybody tries it. The broker collapses the same way with a
different outcome — `amqp` **is** a protocol that several implementations speak, so `amqp` is a
legitimate provision and `lavinmq` is one provider of it.
So adoption is not only an upgrade path. It is two rows of a mesh's module list becoming one,
twice.
## And it is one SERVER each, not two
**The broker duplication is running today, not just latent.** The lavinmq module raises its own
`server` container and points its provisioner at `http://lavinmq:15672` — a second LavinMQ,
separate from the substrate's `mesh-broker`. A mesh with the module assigned runs both.
This is not how LavinMQ is meant to be used, and the module's own provisioner says so: a consumer
is given a *vhost* named for its login, isolated from every other consumer's by the vhost boundary
— *"the exact analog of postgres's database-per-login"*. One server hosts the mesh's own control
traffic on the `/` vhost and every consumer's broker as a vhost beside it. Two servers is the same
mistake as two postgres containers, wearing AMQP.
So adoption means the lavinmq module does not run a `server` of its own. Its server IS the
substrate broker, adopted; the module contributes the provisioner, the tools, the event consumer
and the run-once bootstrap that configures it — all against the one broker. Same for postgres: one
server, the control plane's records in their databases and every module's database beside them.
**The provisioner already assumes this** — it creates a vhost, not a broker — so the change is
removing the second server, not building a new isolation model. What has to be designed is only the
adoption itself: raising the one broker at genesis because nothing else can, then holding it as the
module, over the broker it is.
## What makes this harder than it looks
**The recursion is real, not incidental.** The control plane learns what modules exist by reading
its store. A store that is a module is a record inside the thing it is holding up. Adoption is what
resolves it — the store is raised by the installer because nothing else can, and only afterwards
becomes something the mesh has a record of — but the order of operations during an upgrade needs
stating, not assuming: a machine being sent a new store while the control plane is reading from it
is not a rollout, it is an outage.
**And the broker carries the rollout itself.** A declaration reaches a machine over the broker. A
broker upgrade is the mesh asking the machine to replace the thing the request arrived on.
## Open questions
- **Is adoption the right mechanism**, extending ADR 0067 to the substrate, or should the substrate
stay outside the module system and gain its own narrower update path?
- **What does a store upgrade look like** when the control plane is mid-read? Drain, quiesce, or a
window where the mesh accepts that it cannot be asked anything.
- **What does a broker upgrade look like** when the instruction travels over it? A machine that is
told to replace its broker has to complete the work without being able to report progress.
- **Does adoption also mean one server instead of two** by default, with a separate one for the
control plane as a choice for those who want the isolation — which is assigning a different
provider, a mechanism that already exists?
- **What reports it today?** Whatever is decided, `status` should be able to say the substrate is
behind. Today it cannot form the sentence.
## How this would be checked
| Rule | Checked by |
|---|---|
| The mesh can say its substrate is out of date | The store's source moves and `status` reports it behind, the way it does for any module. |
| The mesh can deliver a substrate update | A store or broker is upgraded on a running mesh and the control plane is answering afterwards, with its records intact. |
| Nothing runs twice without a reason | A mesh with one machine runs one postgres unless somebody asked for two. |
## Resolution
The foundation's store and broker are adopted in place as the ordinary `postgres` and `lavinmq`
modules (WBS Phase 3). Each module declares the container the foundation raised — the same name,
image, ports, volumes and args — so the applier reconciles it rather than raising a second server;
`mesh-host`'s `InstallStore` (phase3.go) carries in the genesis credentials the mesh cannot invent.
The servers bind mesh-wide so consumers reach them, and their provisioners run host-networked. An
upgrade of each is proven through its stated window — the store's a pool reconnect, the broker's the
harder case of recreating the bus the push travels over — and `status` now reports both as ordinary
modules that can be behind their source. A mesh built from bare metal runs one postgres and one
lavinmq, proven 22/22 in the one-node lab. Two follow-ups are tracked: [054](../054-the-adopted-store-and-broker-are-open-before-the-filter/00-report.md)
(the pre-filter exposure window) and [055](../055-the-adopted-store-and-broker-may-be-reachable-on-one-node-only/00-report.md)
(multi-node reachability).
@@ -0,0 +1,71 @@
---
status: open
opened: 2026-09-14
located-in: []
fixed-by:
amended-design:
---
# 052 — The firewall closes the port the mesh runs on
## Symptom
A one-node mesh with the packet filter assigned generates this input chain:
```
policy drop
ct state established,related accept
iif "lo" accept
icmp / icmpv6 accept
<mesh address> tcp dport 22 accept ssh
<mesh address> tcp dport 5000 accept the registry
<mesh address> tcp dport 20000 accept
udp dport 51820 accept the overlay
```
**The substrate broker's port is not there.** Every machine dials it to enrol and to receive every
declaration it is ever sent. Nothing opens it.
The rules are generated from what modules declare they listen on. The broker is not a module — it
is raised by the installer from the bundle — so it declares nothing, and the generator has nothing
to generate from.
## Why this matters
**It is invisible on the mesh where it is first assembled and fatal on the next one.** A mesh of one
never dials its own broker across the network, so the missing rule changes nothing and the firewall
looks correct. The first machine that tries to join is refused at the packet filter, during
enrolment, before the mesh can report anything about it — and the cause is several steps from the
symptom.
**It would have been met during the migration, not before it.** The intended order is to raise the
anchor, assign its modules, then join the other machines. Assigning the firewall before the second
machine enrols is both the natural order and the one that breaks.
**It is [051](../051-the-mesh-cannot-update-what-it-depends-on/00-report.md) in a second place.** That
issue records that the substrate cannot be updated because the mesh holds no record of it. The same
absence means the firewall cannot know the substrate exists. Anything else generated from what
modules declare has the same hole: the store, the broker, and their ports are outside every such
computation.
**The store is the same shape and has not been checked.** It binds on the machine and is reached by
the control plane; whether that survives a default-drop policy has not been established here.
## Open questions
- **Should the substrate declare its listens** — which means the substrate being something the mesh
holds a record of, i.e. 051's adoption?
- **Or should the firewall have a floor** that is not derived from modules at all, the way ssh
already is? Ssh is in the rules for exactly this reason: a machine nobody can reach is a machine
nobody can repair, and that is not a thing any module declares. The broker is arguably the same
class of fact — the mesh cannot manage a machine it cannot talk to.
- **What else is in that class?** Whatever the answer, the question "which ports must be open for
the mesh itself to work" should be answerable from one place rather than assembled from modules
that happen to exist.
## How this would be checked
| Rule | Checked by |
|---|---|
| A machine with the firewall assigned can still be joined | The packet filter is assigned to an anchor, and a second machine enrols afterwards. |
| The mesh's own ports are open | The generated ruleset is compared against the substrate's own bindings, not only against module declarations. |
@@ -0,0 +1,76 @@
---
status: open
opened: 2026-09-15
located-in: []
fixed-by:
amended-design:
---
# 053 — The SDK is pinned twice, and the two disagree
## Symptom
The module that carries the tool runtime names the SDK two ways, and they are not the same thing:
```
package.json @novox/mesh-sdk -> git+https://<the forge>/mesh-sdk.git#<a commit>
package-lock.json @novox/mesh-sdk -> ../mesh-sdk
```
The manifest names a commit in a repository any machine can reach. The lock names a **sibling
directory**, which exists on the workstation the lock was generated on and nowhere else.
It builds anyway, because the recipe runs `npm install` — which tolerates a lock that disagrees
with its manifest and re-resolves from the manifest. It is the one command that hides this.
## Why this matters
**A lock file exists to make a build reproducible, and this one describes one machine.** `npm ci` —
the command for exactly the case a lock is for — fails here, or worse, succeeds against whatever
happens to be at that path.
**It is the first thing a fresh mesh builds.** The toolchain image carries the SDK, and everything
with code of its own is compiled inside it. A dependency resolved differently on the build machine
than on a workstation is a difference in every module the mesh will ever build, arriving as a
compile error or a runtime mismatch far from here.
**And it is about to be copied.** Each language's toolchain will carry that language's SDK the same
way ([ADR 0074](../../02-DECISIONS/0074-the-wire-is-specified-not-the-types.md)). Whatever this
repository does, the Rust and Python ones will do, so the shape is worth getting right before there
are four of them.
## It is a violation, not an open question
[ADR 0014](../../02-DECISIONS/0014-no-npm-workspace.md) already decided this, and was accepted:
> Each module is an independent package that declares its dependencies and **consumes them from
> the private registry, including the mesh's own shared library.** A cross-package change is
> therefore two steps: publish the producer, then consume it.
So a git dependency at a pinned commit is not an alternative mechanism under consideration. It is
the mesh's own shared library being consumed by a means the record rules out, and the lock naming
a sibling directory is what that looks like when nobody publishes.
**Which makes the fix a direction rather than a discussion**: publish the SDK to the private
registry, consume it by version, and the lock stops being able to name a path that exists on one
machine.
## Open questions
- **Which private registry, and does the bootstrap have one?** The catalogue holds `verdaccio`;
a git host typically serves package registries too. ADR 0014 says *the* private registry as
though there is one, and today a fresh mesh has neither until something installs it — so the
first SDK build happens before the registry the record assumes.
- **Or should the SDK be a published package?** The catalogue holds a private registry module, and a
published package is how the rest of the world does this — at the cost of a mesh needing that
registry up before it can build anything, which is a bootstrap problem where there is currently
none.
- **What generates the lock, and on what?** A lock produced on a workstation with sibling checkouts
will keep saying this. A lock produced the way the image builds would not.
## How this would be checked
| Rule | Checked by |
|---|---|
| A build does not depend on the machine it runs on | The toolchain image builds with `npm ci` rather than `npm install`, which refuses a lock that disagrees with its manifest. |
| The SDK a module compiles against is the one named | The commit baked into the toolchain image is compared with the one the manifest pins. |
@@ -0,0 +1,48 @@
---
status: open
opened: 2026-09-16
located-in: []
fixed-by:
amended-design:
---
# 054 — The adopted store and broker are open before the packet filter exists
## Symptom
Adopting the store and broker as ordinary modules (issue 051) needs them reachable by
consumers across the mesh, so genesis now raises `mesh-store` and `mesh-broker` bound to
`0.0.0.0` rather than to loopback as the foundation used to. The packet filter is a module,
installed several steps after the store, the broker and the catalogue are already running.
Between each server coming up and the filter being applied, both listen on every interface
of the machine with only the genesis bootstrap credentials, and nothing drops traffic to
them. On a control-node that faces the network while it is being adopted, that is postgres
(bootstrap superuser) and a message broker open to anyone who can reach the machine, for the
length of the install.
## Why it matters
The design's rule is that what a port is reachable from is decided by the firewall, computed
from each module's `listens.from` — `mesh` for both of these. A rule enforced by nothing is
indistinguishable from a wrong one, and for the duration of this window that rule is enforced
by nothing: the thing that would apply it does not exist yet. It is the same window the ssh
rule already reasons about ("reached over the network, before the private network exists"),
but ssh is one guarded port and this is the mesh's whole store.
The foundation used to sidestep this by binding the store to loopback — only the co-located
control plane reached it — and adoption trades that away, because a module that adopts the
container in place must declare the same bind, and the module has to serve consumers.
## Open questions
- Can a default-deny base filter (drop everything but loopback, established, and ssh) be
applied at genesis, before the store and broker come up, and the mesh-scoped rules refined
once the private network has addresses to name? The consumers that must reach the store
(the catalogue) come up before the network step, so the `from: mesh` rule would have to be
in place by then.
- Or should the servers bind narrowly at genesis (loopback plus the container bridge) and
widen only once the filter that protects them exists — accepting that a bind change is a
recreate, so this would mean the store is recreated once during install?
- Is the window acceptable as-is, given the machine is mid-bootstrap and the exposure matches
what the pre-adoption `postgres`/`lavinmq` modules already had in steady state?
@@ -0,0 +1,40 @@
---
status: open
opened: 2026-09-16
located-in: []
fixed-by:
amended-design:
---
# 055 — The adopted store and broker may be reachable on the control-node only
## Symptom
The adopted `mesh-store` and `mesh-broker` bind `0.0.0.0` on the control-node. The
co-located provisioner and control plane reach them over loopback, and a co-located consumer
reaches them through the container bridge — which is what the one-node bed proves. A consumer
on ANOTHER machine reaches a provider by its `.internal` name over the private network, and
whether that path resolves to the control-node's bind is unproven: the one-node bed cannot
exercise it, and no multi-node bed installs the adopted store or broker.
## Why it matters
Phase 3's stated goal is a mesh — of any size — that runs one postgres and one lavinmq. The
whole point of a shared store and broker is that a module on any machine that is granted a
database or a vhost can open it. If the adopted servers are reachable only on the machine
they run on, a grant to a module placed elsewhere names an endpoint that machine cannot dial,
and the failure surfaces far from here as a consumer that cannot connect.
Before adoption this was a non-question: the store served the control plane alone and the
`postgres` module raised a second server that published mesh-wide. Collapsing to one server
means the one server has to be the mesh-wide one, reachable across the overlay.
## Open questions
- Does the mesh deliver the store and broker endpoint to a remote consumer as an address that
consumer can dial — the provider's overlay address — rather than a loopback or bridge
address meaningful only on the control-node?
- Is a `0.0.0.0` bind on the control-node reachable over the WireGuard overlay from a joined
machine, and is the packet filter's `from: mesh` rule enough to let it through?
- What is the smallest multi-node bed that would prove a database granted to a module on a
joined machine can be opened from there?
+6
View File
@@ -12,6 +12,12 @@ through them. Thin skills in `.claude/skills/` wrap these playbooks for invocati
`hq-handoff`, `hq-sync-constitution`, `hq-status`); each defers to its playbook as `hq-handoff`, `hq-sync-constitution`, `hq-status`); each defers to its playbook as
authoritative and adds only the mechanical scaffolding. authoritative and adds only the mechanical scaffolding.
## Words
One name per thing. [`00-META/glossary.md`](00-META/glossary.md) is the authority on
vocabulary — *controller* (not "control plane"), *foundation* (not "substrate"), *node* and
*control-node*, *seat* / *bench* / *claim*, *package* vs *artifact*. Use those words.
## Ground rules ## Ground rules
- **Markdown only.** No new top-level folders without explicit confirmation. - **Markdown only.** No new top-level folders without explicit confirmation.
+1 -1
View File
@@ -1,7 +1,7 @@
# Novox HQ # Novox HQ
The single source of truth for what Novox builds — what it **is**, what it is **becoming**, The single source of truth for what Novox builds — what it **is**, what it is **becoming**,
and why. Today that is almost entirely **Novox Mesh**, the substrate everything else runs on. Implementation lives in `modules/`; the reasoning behind it lives here. and why. Today that is almost entirely **Novox Mesh**, the foundation everything else runs on. Implementation lives in `modules/`; the reasoning behind it lives here.
## Structure ## Structure