novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge branch 'issue/021-provider-port-published-on-loopback' into design/bootstrap-is-a-pivot

Browse Source 
# Conflicts:
#	03-DESIGN/01-to-be/04-lab-installation.md
This commit is contained in:
jschoubben
2026-09-11 00:45:59 +02:00
parent e4a5d9b1d0 a4440c9acf
commit 84c95c53af
188 changed files with 14552 additions and 2968 deletions
Download Patch File Download Diff File Expand all files Collapse all files
03-DESIGN/01-to-be/00-work-breakdown.md
+402 -130
View File
@@ -1,158 +1,430 @@
---
layer: to-be
status: designed
code: [hal]
updated: 2026-08-23
decisions: [02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md]
code: []
updated: 2026-09-01
decisions:
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0009-modules-and-the-graph.md
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md
---
# Work breakdown — the decomposition
# Work breakdown — replacing what provisions the mesh
How [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md) gets built, in what order, and where a human must look.
*Rewritten 2026-08-31. The previous version planned a decomposition of the existing system in
place: extract contexts, convert modules to declared features, shrink its shared library. That is
not what is being done — a replacement is being built beside it, and the old plan's Phase 0 was
the only part that survived contact with it. So the document that was supposed to say what happens
next had been describing work on a system being retired.*
Ordering is not preference. Each phase removes a constraint the next one needs gone.
## The goal, in one sentence
---
**Modules move to the new mesh one at a time, until the old registry can be switched off.**
Everything below is ordered by what that requires. Nothing here is a rewrite of the old system;
its modules are the input.
## Phase 0 — a mesh that runs — **done**
Not *the code exists*. Twenty-two assertions on real machines in the lab, each confirmed to fail
when the behaviour is removed ([ADR 0016](../../02-DECISIONS/0016-the-lab.md),
[ADR 0017](../../02-DECISIONS/0017-a-test-defends-a-decision.md)).
| what is proven | |
|---|---|
| **a mesh comes into being** | a bare machine becomes one; others join with nothing but a token |
| **credentials** | delivered to both ends with the mesh holding neither; rotated so the old one stops working |
| **declarations survive reality** | a stopped machine is waited for; one that fell behind catches up unnamed; unassigning takes away exactly what it should; what the mesh says nothing about is left alone |
| **failure is legible** | a machine that cannot do what it was told is named, with why |
| **the mesh runs itself** | its own artifact store, and a builder that is a module the mesh assigns |
| **names and reachability** | internal names, wildcards under a machine, containers reaching other machines, certificates the mesh issued, filtering that matches exactly what was declared |
| **delivery** | a new commit reaches a machine already running the old one |
| **model access** | answered by a record, with a key the mesh cannot read |
**What Phase 0 does not prove, and it is the important sentence in this document:** every module
exercised above was written to test the mechanism. **No module from the existing system has ever
run on this.** The vocabulary was shaped by the things used to test it — the same fault as a
fixture agreeing with the code it checks
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)), at the
scale of a design.
## Phase 1 — the vocabulary a real module needs
Found by taking real modules and asking what they would require. Each is a gap in what can be
*expressed*, not a defect in what is built.
| # | task | done when |
|---|---|---|
| ~~1.1~~ | ~~An **object-store provision**~~ — **done 2026-08-31**, and it needed no change to the mesh: see below | seven assertions against a real store |
| ~~1.2~~ | ~~**A session as a consumer of a licence**~~ — **done 2026-08-31**, and it also needed no change: see below | two sessions on one machine, different licences, each its own key |
| ~~1.3~~ | ~~A **network** shape, and ordering within a module~~ — **done 2026-08-31** | the shape is created and removed; ordering was already there, and is now asserted |
| 1.4 | **Public certificate issuance** — **built; one gap** | ordering, the challenge and issuance are proven against a real authority; **collecting the issued certificate is not** ([`04-ISSUES/020`](../../04-ISSUES/020-a-certificate-is-issued-and-never-collected/00-report.md)) |
**1.3 and 1.4 block later ones** and are listed now so they are not met as surprises. 1.3 is what
a mail system needs and nothing else so far does.
**Phase 1 is closed with 1.4 partly open**, deliberately. Two of its four items needed no code at
all; the network shape was built; and certificates are configured correctly, order correctly, and
are issued correctly — the client does not collect what the authority issued, against a server
that exists to be a test server. That is filed rather than chased, because the remainder may say
nothing about a real authority and the next thing to learn comes from moving a module rather than
from a fourth lab run.
**Checkpoint:** each is demonstrated in the lab before the module needing it is attempted.
### 1.2, and the same surprise twice
**A binding is per module per machine, and the two sessions are two modules** — the same mechanism
in different context roots, and a context root is what a module delivers. So `(node, module)`
already names them apart, and nothing needed adding.
[`14-model-access.md`](14-model-access.md) had called per-module-per-machine *a step toward it and
not it*, which is true of a **worker** — many run on one machine from one module — and not true of
a session, of which there is one per node and one for the mesh.
### 1.3, and the first one that needed building
**Ordering was already there** — the apply loop sorts nothing, so a module says *this before that*
by writing it first. Untested until now, and the kind of property a later change breaks silently.
Worth separating from readiness: a container started is not a container ready, and nothing waits.
What needs something *usable* retries, which is what both provisioners do and is the better answer
anyway, because a dependency can restart long after everything was applied.
**The network was a real gap, and the first thing in Phase 1 that needed a decision.** Adding a
shape widens what a compromised control plane can express, so
[ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)
records why this one is worth it: an `action` could create a network and **nothing could ever
remove it**, because an action leaves no footprint the host can undo. The vocabulary is nine.
**Three tasks in a row that were already possible.** Both were written from the design rather than
from the code, which is the review's finding arriving in the plan: *a claim here is counted, not
reasoned.* The remaining Phase 1 items should be checked against the code before being started,
not after.
### 1.1, and what it turned out to be
*Done 2026-08-31. Worth recording because the task was not the one written down.*
**The control plane special-cases nothing.** `provides`, `requires`, `contributes` and `grants`
are name-agnostic — asking for a bucket needed no change to the mesh at all. What was missing was
a provider, and the last step where something on the machine turns a delivered secret into a key
that works. So "add an object-store provision" was never mesh work.
The provision is `s3-bucket`: a consumer's code is written against the S3 API and swapping one
store for another does not break it, so by
[ADR 0027](../../02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md) the name
says the protocol. A database is the other case, and names the engine.
**One assertion here that a database does not need.** One PostgreSQL server holds separate
databases and the product enforces the boundary; one object store holds every bucket behind one
endpoint, so *a consumer cannot reach another consumer's bucket* is a policy somebody wrote — and
a policy granting everything would pass every other test. **What is asserted is what the policy
does not say.**
## Data is the constraint, and it outranks the order below
*2026-08-31.* The modules being converted run live services — identity, mail — and **the data must
survive every step**. A data folder may move; it may never be lost.
**One thing was found by asking this and is fixed**
([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)): the host deleted
a directory and everything under it when the directory stopped being declared, which happens when
a module is unassigned or a manifest is edited to move a data folder — the exact operation this
plan needs. A directory holding anything the mesh did not put there is now kept and reported.
**That is not a backup and must not be read as one.** It stops the mesh destroying data. It does
nothing about a disk, a mistaken command, or a service corrupting its own store.
**So the rule for every step below:** the data is copied, the copy is verified by reading it back
through the service that owns it, and only then does anything point at the new location. Never
moved and then checked. **A backup nobody has restored is a belief, not a copy.**
## A module is adopted with the credentials it already has
*2026-08-31.* **Nothing is rotated during the conversion.** A service being adopted keeps the
password it is already using, because minting a new one is how a running service stops being able
to reach its own database in the middle of a migration.
The mesh has both paths and this needs the second:
| | |
|---|---|
| **generate** | a new secret, sealed to both ends. What a *new* module gets |
| **accept** | a value supplied from outside, sealed, plaintext discarded. **What an adopted module gets** |
**Rotation is a separate act, afterwards, once everything works.** The machinery for it is built
and proven — a credential moving at both ends with the old one ceasing to work — and it is exactly
the sort of thing to do deliberately on a quiet afternoon rather than as a side effect of moving a
service between systems.
**So there is a step before any of this: read the current environment out of the old system**, because
adoption means supplying those values and they live in its files today.
**And there is a failure worse than losing data, which is likelier.** A database image consumes its
password environment variable **only when its data directory is empty**. Everything here keeps its
data on a persistent directory, so the role holds whatever password it was created with, for ever.
Regenerate that variable and the application moves on while the database does not — permanently,
because nothing reconciles it. Eight modules are in that state today, working only because nobody
has regenerated their credential since their data directory was created.
*Where the detail lives:* this is operational and names machines, so it is in the mesh's own
knowledge base rather than here — `migration/where-service-data-lives`, which surveys where every
service's data actually sits and what each stop or removal would cost, and
`troubleshooting/db-password-frozen-at-first-init` for the lockout itself. **This document says the
rule; those say the specifics.**
*Corrected 2026-08-31 — an earlier version of this paragraph made that sound more dangerous than it
is.* A sealed secret is not unreadable; it is sealed **to the node**, which holds the private half
and writes the plaintext into the module's own file. The value is there, on the machine, as an
ordinary file. What does not exist is a way to ask *the mesh* what a secret is, and there is no
reveal command, because a mesh that can reveal a secret is a mesh that holds one.
## Where it starts, and what that costs
**On the node holding all the production data**, because that is where the services being
converted actually are.
Recorded plainly rather than argued with: this is the highest-risk order available. Everything
proven so far was proven on machines that could be destroyed and raised again, and the first real
exercise of the conversion will be on the one machine where a mistake is not recoverable. Nothing
about the lab work transfers automatically — a scenario proves the mechanism, not the state on
that machine.
**What makes it survivable is preparation rather than caution**: a restored backup before the
first step, one service at a time, and the previous arrangement left standing until the new one
has been read back. None of that is slower than the alternative, because the alternative includes
losing something.
## How the two systems hand over
*2026-08-31.* **The old system's brain is switched off; its services keep running.**
Not a migration and not a period of dual control. The old control plane — provisioning, the
coordinator, the pipeline, the things that *decide* and *write* — is stopped. Every workload it
was managing goes on running exactly as it is, because nothing is managing it. Then the new mesh
takes ownership of them one at a time.
**Nothing is ever unassigned in the old system.** Unassigning is how it removes things, and
removing is how data is lost. The old system is never asked to take anything away; it is asked to
stop having opinions.
| | |
|---|---|
| **stopped, and disabled** | provisioning, the coordinator, environment and configuration sync, the pipeline — anything that decides or writes a file |
| **left alone entirely** | the units running the actual services: identity, mail, databases, the forge. They keep serving throughout |
| **never used** | unassign, remove, delete — any operation whose job is to take something away |
**Disabled, not merely stopped**, and this is the part that is easy to get wrong: those units are
enabled, so stopping them lasts until the machine reboots. A reboot mid-conversion would bring the
old control plane back and it would resume regenerating managed files underneath the new one —
which is the one situation where two systems really would be fighting over the same machine.
**A service left running with nothing managing it is the safe state.** It has its data, its
configuration is already on disk, and nothing is going to change either. That is the whole trick:
the risk in a conversion is in the *managing*, not in the *running*.
**A brief interruption is acceptable. Losing data is not.** Where those two trade against each
other, the interruption wins every time — a service can be restarted, and there is no operation
that un-deletes a mail spool.
**The new host cannot remove what it did not put there.** Orphans are per-origin, so it only ever
removes resources it recorded itself. Services it has never been told about are not orphans to
it — they are simply not its business, which is what makes taking ownership one module at a time
safe.
## Phase 2 — the first real module
| # | task | done when |
|---|---|---|
| 2.1 | Port an **object store** module | it runs on the new mesh, serves a bucket to another module, and its credential rotates |
| 2.2 | Copy the data, and read it back through the service that owns it | the new location answers with what the old one holds |
| 2.3 | Point one dependent at it, old arrangement left standing | something real reads and writes through the new mesh's copy |
**Checkpoint, and it is a human one:** it runs for a week before anything else moves. The point of
going first is to find what Phase 1 missed, and a week is roughly how long that takes to show.
## Phase 3 — the modules that prove the shape
Each exercises something the first one does not.
| # | task | proves |
|---|---|---|
| 3.1 | An **identity provider** | a module that is itself a provider — the provides/requires chain, with consumers requiring it |
| 3.2 | A **forge** | a port claim against the machine's own daemon, and a module wanting both a database and an object store |
| 3.3 | A **mail system** | several containers as one module, a private network between them, and names that are not one-per-node |
**3.3 is the hardest thing in this document** and is deliberately last. If the declaration
language turns out to be insufficient, it says so here.
### Where Phase 3 actually stands — *2026-09-01*
All three have manifests. All three parse, resolve and plan. **None of them can start**, and the
two reasons are both filed rather than guessed at.
The **vocabulary held**. Nothing in 3.1–3.3 turned out to need a new shape: the identity provider,
the forge and the mail system are all expressible with what exists, including the mail system's
several containers on a private network — which was the one expected to break it. That is the
question this phase was designed to answer, and the answer is yes.
What did not hold was underneath the vocabulary:
- **[`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md)**
(fixed) — a credential belonged to a machine, so a node running several modules against one
database could not be planned. The refusal was loud on the provider and silent on the consumer.
- **[`023`](../../04-ISSUES/023-a-consumer-cannot-build-a-connection-string/00-report.md)** (open)
— a consumer gets its password and still cannot connect: the user name is invented by the
provisioner and recorded nowhere, and the bound values cannot reach a configuration file.
A third fault was in the manifests themselves rather than the design: each declared a secret at a
path named `.env` and read it as one, when a sealed file holds a password and nothing else. They
parsed and resolved and could never have worked, which is what a manifest checked only by a parser
buys. Two tests now refuse both halves of it.
### 3.1 needs a program, not a decision — *2026-09-01*
023 is fixed, and with it the two design faults are gone. What stands between 3.1 and a running
identity provider is now one concrete thing: **the realm provisioner does not exist.**
Its manifest named an image — `mesh-provision-keycloak` — that nothing builds and no program
backs. That has been removed rather than left standing, because a manifest describing a program
nobody wrote is the same mistake as the credential files that could never be read: it parses, it
resolves, and it could never work.
So Keycloak's manifest now says what is true today — a server the mesh runs, with its database
and its admin credential, both reaching it in a shape it can read. It no longer claims to provide
`oidc-client`, which means a consumer asking for one is **refused by name at plan time** rather
than resolving cleanly and waiting for a client nothing will create.
The provisioner is the same shape as the two that exist: it reads what the mesh granted and
reconciles a realm and a client per consumer. **It should be written against a real Keycloak in
the lab**, not from the API documentation — the object store's took three corrections that only a
running server produced.
The forge (3.2) and the mail system (3.3) need no provisioner and are not blocked on this.
### And they could not have run anyway — *2026-09-01*
Every one of the five named a container image that does not exist: sixty-four zeros where a digest
belongs, eighteen times over
([`025`](../../04-ISSUES/025-a-module-must-pin-a-digest-and-nothing-produces-one/00-report.md)).
They parsed, resolved and composed into a declaration a host accepts, and every one would have
stopped on the machine at the moment of fetching.
Nothing caught it because nothing could. A host checks the *shape* of a reference and no more —
verifying a digest exists means reaching a registry, which is the one thing a host must never have
to do. The refusal now sits where a declaration is composed instead, which is the last moment
before a machine sees one.
Twelve are pinned to real images. Two further faults surfaced only by pinning for real: the mail
system's seven images named repositories that **do not exist**, because it publishes to a
different registry than assumed, and one of the seven had been renamed upstream.
**The forge now runs**, on a database another module provides, with a password it did not choose
and a connection string it could not have written. That is the first of these descriptions to be
started rather than planned, and it exercises everything the credential work added.
What is still missing is the mechanism: nothing turns a tag into a digest as part of the mesh's
own work, so it was done by hand. Asking a registry takes about a second and pulls nothing, which
removes the main argument for leaving it undone.
## The conversion is done by hand, and that is a decision
*2026-08-31.* **Moving from the current system to this one is a person at a command line, working
through it.** Not a migration program, not a converter, not a period of dual-writing.
**What that removes from this plan is larger than what it adds.** Nothing below needs an importer,
a translation layer, a compatibility shim, or a mechanism for keeping two systems agreeing while
both are live — and every one of those is a thing somebody would otherwise reasonably build, use
once, and maintain for a year. The modules are the input; a person reads what one does today and
writes what it declares tomorrow.
**It also changes what "safe" means for the system being retired.** A fix to it has to be safe on
its own, because there is no careful rollout to sequence it into: the thing is being switched off
by hand, not managed into retirement. A change needing three steps in the right order is a change
that will be half-applied.
**And it is why the checkpoints below are weeks rather than gates.** Nothing enforces the order —
a person does — so the value of the sequence is entirely in what each step teaches before the next
one starts.
## Phase 4 — switch the old registry off
| # | task | done when |
|---|---|---|
| 4.1 | Move the remainder, by hand, a module at a time | nothing is assigned in the old system that is not assigned in the new one |
| 4.2 | The old one authoritative for nothing | a change to any module goes through the new mesh only |
| 4.3 | Switch it off | it is stopped, and nothing notices |
**4.3 is a day's work and the phases above it are not.** Naming it as a phase is what stops it
being mistaken for the goal.
## Sequencing
- **1 before 2.** Attempting a module without the vocabulary it needs produces a workaround, and a
workaround in a manifest is a design decision taken by whoever was in a hurry.
- **2 before 3, with the week.** Moving three modules before running one is how three modules
acquire the same defect.
- **3.3 last.** It is the only one that may send work back into the declaration language.
- **4 cannot start early, and there is no partial credit.** A registry still authoritative for one
module is still running.
## How this list is kept true
*This section exists because the document it replaces was wrong for weeks and nothing said so.*
**A claim here is counted, not reasoned.** The review of 2026-08-31 found a bundle described as
carrying two images that carries three, a bootstrap described as needing six shapes that uses
four, and ten documents calling themselves `designed` while naming lab-proven code. Each was
produced by describing the system from its design instead of reading it.
**A phase is done when the lab says so**, and the lab keeps a receipt of when it last ran and
against which commits. A phase marked done here whose assertions have not run is a claim about the
past.
**What is not proven gets said.** Phase 0 is done and its limitation is written into it. A list
that records only progress becomes a list nobody believes.
## Rules of engagement
These exist so the work can run largely unattended without accumulating the kind of
damage this refactor is meant to remove.
Unchanged from the previous version: they were about how work is done rather than what the work
is.
### Autonomous by default
An agent may, without asking:
- read anything, measure anything, query any database read-only
- create branches, write code and tests, open pull requests
- run the test suite and typechecks
- write and update `hq/` documents
Read anything, measure anything, query read-only. Create branches, write code and tests, run the
suites, and write or update documents here.
### Always stop and ask
- **destroying or overwriting data** — dropping a table, deleting a provision, rotating a
live credential, removing a module from a node
- **destroying or overwriting data** — dropping a table, deleting a provision, rotating a live
credential, removing a module from a node
- **merging anything** — every merge is a human checkpoint, without exception
- **a decision the ADRs do not already answer** — record the question in the relevant
research effort rather than picking and moving on
- **any change to `hq/00-META`** — it is stable by nature
- **anything touching a machine outside the lab**, including a configuration change that restarts
something people are using
- **a decision the records do not already answer** — record the question rather than picking and
moving on
- **any change to [`00-META`](../../00-META/)** — it is stable by nature
### Definition of done for every task
1. tests written **and failing first**, then passing
2. typecheck clean in every package the change touches
3. the local mesh (Phase 0) comes up, and the behaviour is demonstrated in it
4. `hq/` updated if the task changed or answered anything documented
5. deployed, and **delivery verified on every node** — not "the pipeline was green"
3. the behaviour demonstrated **in the lab, on real machines** — not asserted
4. documents here updated if the task changed or answered anything recorded
5. delivered, and the **effect** verified — not that a pipeline was green
### Non-negotiables carried from the current system
### Non-negotiables
- **Never edit mesh-managed files on disk.** Use the owning tool.
- **Never write to production databases directly.** Migrations for schema, application
code for data.
- **Every schema change ships twice** — consolidated schema *and* an incremental
migration.
- **Expand, then contract.** Add the new shape, migrate, verify, and only then remove the
old one — never in a single step.
- **A green pipeline proves transport, not effect.** Verify the effect.
---
## Phase 0 — A mesh that runs locally *(prerequisite)*
Nothing else starts until this exists. Every fault this refactor addresses was found in
production because there was nowhere else to find it.
| # | task | done when |
|---|---|---|
| 0.1 | Container image for a node runtime | a node process starts in a container and registers |
| 0.2 | Compose topology: broker, registry DB, object store, *n* nodes | `up` yields a mesh that elects a provider node and settles |
| 0.3 | Seed a minimal mesh: nodes, one module, one provision | a module deploys end-to-end with no external service |
| 0.4 | Run the pipeline inside it | a push-equivalent produces a cascade and a deployed artifact |
| 0.5 | Fixtures for the failure modes already known | credential rotation reaching a running session; a provider deploy rotating a shared credential; a migration that ships nothing — each reproducible on demand |
**Checkpoint:** a human confirms the local mesh reproduces at least one bug from
2026-08-22 before any decomposition begins.
---
## Phase 1 — Make the model expressible
The decomposition is impossible while a feature is a singleton per module.
| # | task | done when |
|---|---|---|
| 1.1 | Decision record — named features, per-node opt-in (next free number) | accepted |
| 1.2 | Manifest: declared `features:` with type + directory | a module declares two of one kind and both build |
| 1.3 | Selection: `always` / flavor-selected / `optional` | a node installs a subset; artifacts stay flavor-blind |
| 1.4 | `requires:` moves onto the feature | a schema feature's database is not provisioned where the feature is not installed |
| 1.5 | Assignment carries the opted-in feature set | opting a node in requires no rebuild |
**Checkpoint:** one existing module converted to declared features, deployed, verified —
before any others follow.
---
## Phase 2 — Draw the boundary the domain already has
Cheapest first, and each one proves the extraction pattern before the expensive ones.
| # | task | extracted from | risk |
|---|---|---|---|
| 2.1 | `hal/knowledge` — one store, review workflow ported | hippocampus + noxflow `knowledge_*` | low — additive |
| 2.2 | `hal/stream` — the record; notifications and messaging as views | axon, synapse, notifications, meetings, conversations | medium |
| 2.3 | `hal/agents` — identity, licence, runs, memory, thoughts | noxflow agents, `hal/thoughts` | **high** — touches credentials |
| 2.4 | `hal/work` — what remains of noxflow | noxflow tasks | medium |
| 2.5 | `hal/ai` — provider integration, flavored | `hal/claude*` | medium |
Each extraction is expand-then-contract: new context alongside, dual-write, verify, cut
over, remove. **Never a move commit.**
**Checkpoint:** after 2.1, a human confirms the extraction pattern before 2.2 begins.
After 2.3, a human confirms credentials still reach every agent on every node.
---
## Phase 3 — Reclaim the kernel
Only possible once domains have modules to own their code.
| # | task | done when |
|---|---|---|
| 3.1 | Move work-domain code out of `hal/sdk` | `workflow-engine.ts`, `task-commands.ts` live in `hal/work` |
| 3.2 | Move provider code out | `claude-credentials.ts` lives in `hal/ai` |
| 3.3 | Move delivery code out | feature handlers, artifact manager, build executor live in `hal/delivery` |
| 3.4 | Decide the residue | ADR: what `hal/sdk` keeps (open question 4) |
**Measure:** `hal/sdk` line count, tracked per task. Today: **34,636** across **155**
files.
---
## Phase 4 — Separate what the mesh runs from the mesh
| # | task | done when |
|---|---|---|
| 4.1 | Decide the destination (open question 3) | ADR accepted |
| 4.2 | Cross-repository dependency resolution proven | a catalogue module builds against a published `@hal/*` |
| 4.3 | Move the 91 catalogue modules | this repository contains only mesh contexts |
**Checkpoint:** move one application first and run it for a week before the rest follow.
---
## Sequencing constraints
- **0 before everything.** Unverifiable refactors are how this list got long.
- **1 before 2.** Extracting into contexts without per-node features recreates the module
count inside the new names.
- **2 before 3.** A domain can only own its shared code once the domain has a module.
- **2.3 after 2.1 and 2.2.** Agents touch credentials; do it once the pattern is proven on
cheaper contexts.
- **4 last.** It is the only phase that is pure movement, so it is the only one safe to
defer indefinitely.
- **Never edit mesh-managed files on disk.** Use the thing that owns the file.
- **Never write to a production database directly.** Migrations for schema, application code for
data.
- **Every schema change ships twice** — consolidated schema *and* an incremental migration.
- **Expand, then contract.** Add the new shape, migrate, verify, and only then remove the old one.
- **A green pipeline proves transport, not effect.**
## What "done" looks like
Eight contexts. `hal/sdk` holding only what is genuinely cross-cutting. A mesh that stands
up on a laptop. A module count that grows only when the domain does.
The old registry is off. Every module runs on the new mesh, declared rather than scripted. A
machine that fails says what it could not do. And the number of modules grows when the work does,
not when the platform needs somewhere to put something.
03-DESIGN/01-to-be/01-end-to-end-testing.md
+43 -9
View File
@@ -2,11 +2,10 @@
layer: to-be
status: in-progress
code: [mesh-lab]
updated: 2026-08-23
updated: 2026-08-31
decisions:
- 02-DECISIONS/0016-a-lab-node-is-a-virtual-machine.md
- 02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md
- 02-DECISIONS/0030-the-repository-structure.md
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0019-how-this-repository-works.md
---
# End-to-end testing
@@ -35,9 +34,9 @@ today, that is a gap in the vocabulary rather than a reason to privilege that sh
## Two classes of scenario
The design below describes a scenario as a complete mesh — forge, coordinator, delivery cascade
— because what it tests is a module. **That is the larger of two classes, and not the first one
built** ([ADR 0029](../../02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md)).
The design below describes a scenario as a complete mesh — forge (Gitea), coordinator,
delivery cascade — because what it tests is a module. **That is the larger of two classes, and
not the first one built** ([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
| | **Bootstrap scenario** | **Full scenario** |
|---|---|---|
@@ -127,7 +126,7 @@ drifts.
a mesh named by the request instead.
- **Scenarios must be concurrent and cheap.** Several agents working means several scenarios
at once, each needing its own network and nodes. A lab node is a virtual machine
([ADR 0016](../../02-DECISIONS/0016-a-lab-node-is-a-virtual-machine.md)), and snapshots are
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)), and snapshots are
what make repetition cheap — restoring a scenario costs far less than building one. The
earlier argument here, that only system containers made this affordable, was superseded: the
scale it assumed was invented rather than required.
@@ -337,7 +336,10 @@ credential rotation reaches every consumer, that delivery to an absent node is r
pending rather than done, that a returning node catches up. These are fewer and change
rarely, but they are where the known production faults get encoded so they stay fixed.
The known faults become mesh tests that fail today. That is the Phase 0 checkpoint.
The known faults become mesh tests that fail today. Phase 0 of
[`00-work-breakdown.md`](00-work-breakdown.md) is now complete on this basis — twenty-two
assertions on real machines — and what it does **not** cover is recorded there: no module
from the existing system has run against any of it yet.
---
@@ -390,6 +392,38 @@ Everything a node itself does is real, because a node is a real machine.
---
## A suite too expensive to run on every push says when it last ran
*Written 2026-08-31, from resolving [04-ISSUES/005](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md).*
This suite needs a machine with a hypervisor. It therefore cannot run on every push, and a suite
that does not run on every push runs **when somebody remembers**. Remembering is not a mechanism,
and the harness this one replaces proves it: it had not built for two and a half months, nothing
said so, and the coverage was assumed rather than checked.
**The danger is not that the suite breaks. It is that nobody notices it stopped running** — and
that danger belongs to *this* design, not to the harness it retired.
So three rules, each held by a test:
**A run leaves a receipt** — when, what passed, what it ran, and the commit each repository was
at. Kept **outside version control**: the question is *has this machine run it*, and a receipt in
git would be a claim about everybody's machine made by whoever committed last.
**A receipt says why it does not count.** Old, failed, taken against commits the repositories have
moved past, or a run that never raised a machine. Something can be asked, and answers non-zero.
**A receipt that says nothing about something is not a receipt that clears it** — including a
receipt written before it recorded a given fact, which claims nothing rather than everything.
**The run rebuilds what it tests.** The suite consumes artifacts from other repositories, and an
artifact rebuilt from memory is one rebuilt sometimes. A stale binary reporting success against
rules that have since changed is the same fault wearing different clothes.
**The general rule, which outlives this suite:** *silence and success must never look alike.*
It is the same rule the host follows about a service that does not exist
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) — absence must be distinguishable
from a failure to answer — applied to coverage instead of to a machine.
## Consequences
**Bringing a node into being is part of the framework.** A test creates its own nodes — one
03-DESIGN/01-to-be/02-scenario-declaration.md
+12 -12
View File
@@ -2,11 +2,11 @@
layer: to-be
status: in-progress
code: [mesh-lab]
updated: 2026-08-25
updated: 2026-08-28
decisions:
- 02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md
- 02-DECISIONS/0031-the-lab-provides-the-underlay.md
- 02-DECISIONS/0016-a-lab-node-is-a-virtual-machine.md
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0016-the-lab.md
---
# The scenario declaration
@@ -15,7 +15,7 @@ A scenario is a **declaration of an underlay**, plus what to put on it. It is th
everything in the lab hangs off, so it is worth getting small.
It states what a hosting provider and a home router would provide, and nothing the mesh is
responsible for ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
responsible for ([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
## Public networks are unrelated, and routed rather than bridged
@@ -87,7 +87,7 @@ Four consequences follow, and every one of them shapes this design:
address stop corresponding.
This is why the mesh dials outward and never inward
([ADR 0001](../../02-DECISIONS/0001-nodes-communicate-over-a-broker.md)), why a hub exists at
([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)), why a hub exists at
all, and why a node's endpoint is something a peer **learns** from arriving packets rather than
something anyone configures.
@@ -258,7 +258,7 @@ otherwise explicit declaration, and it exists because NAT has to run somewhere.
It is a **container, not a virtual machine** — a router is scenery rather than something under
test, so the fidelity argument that makes a node a virtual machine does not reach it
([ADR 0033](../../02-DECISIONS/0033-a-router-is-scenery-not-a-node.md)). What a router must
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)). What a router must
reproduce is kernel behaviour, and a container has the same kernel.
**`machines[].at`** — segment and addresses, or a **list** of them for a machine on several
@@ -299,7 +299,7 @@ belongs to a router it does not control, and asleep.
Whether the overlay survives that, re-forms, and is noticed to have changed endpoint is
**observed**, never arranged
([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
## Why the addresses are load-bearing
@@ -320,13 +320,13 @@ it must be.
The format should make getting this wrong hard rather than merely documented: a segment without
a `gateway:` is a public segment, and an address in it — including a gateway's `address:` — that
is not documentation space is a declaration error, refused before anything is raised. That is
[ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) applied to a configuration
[ADR 0010](../../02-DECISIONS/0010-delivery.md) applied to a configuration
file: the failure it prevents is silent, so the check has to be loud.
## The same declaration serves both classes
The bootstrap and full scenarios differ **only in `place:`**
([ADR 0029](../../02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md)). Everything
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)). Everything
about the underlay is identical, which is what makes one a strict subset of the other rather
than a fork.
@@ -354,7 +354,7 @@ not first.
## What a scenario deliberately cannot say
- **Overlay addresses, the hub, peer configuration.** Outcomes, not inputs
([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)).
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
- **What a machine is in mesh terms** — server or workstation, its site, its names. Mesh
configuration, established by the mesh.
- **A host's capability profile.** Detected, never declared.
@@ -543,7 +543,7 @@ cannot yet express.
Nothing here mentions overlay addresses, which node is the hub, who peers with whom, any name,
or any certificate. Research 004 recorded all of those for this topology, and **a scenario must
not state them** ([ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md)): they
not state them** ([ADR 0016](../../02-DECISIONS/0016-the-lab.md)): they
are what the mesh does, and a scenario that supplied them would be certifying its own work.
The absence is the point. Given the declaration above, whether a hub is elected, whether the
03-DESIGN/01-to-be/03-scenario-lifecycle.md
+8 -8
View File
@@ -2,18 +2,18 @@
layer: to-be
status: in-progress
code: [mesh-lab]
updated: 2026-08-25
updated: 2026-08-28
decisions:
- 02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md
- 02-DECISIONS/0031-the-lab-provides-the-underlay.md
- 02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0016-the-lab.md
---
# Scenario lifecycle
The first thing the lab must do, and the only thing it must do before anything else can be
written: **materialise a mesh, return it to a known state, and destroy it**
([ADR 0029](../../02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md)).
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
A [declaration](02-scenario-declaration.md) describes a scenario. This describes what happens
to one.
@@ -39,7 +39,7 @@ The order is not arbitrary — each step needs the one before it to exist:
1. **Segments.** Isolated links, one per declared segment, belonging to this instance and
joined to nothing outside it
([ADR 0032](../../02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md)).
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
2. **Gateways.** Derived, never declared as machines: a gateway is materialised for each
distinct `gateway:` declaration, sitting on both its segment and its parent, carrying the
translation, forwarding and mapping-expiry the declaration asked for.
@@ -60,7 +60,7 @@ habit.
## A failed raise leaves the wreckage
A step that fails stops the raise
([ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md)) — and **does not tear
([ADR 0010](../../02-DECISIONS/0010-delivery.md)) — and **does not tear
down**.
Tearing down on failure destroys the only evidence of what went wrong, which is precisely
@@ -100,7 +100,7 @@ made after it, and returning undoes it like any other change.
## Reaching in
Everything the lab does to a machine goes through the virtualisation layer, never over IP
([ADR 0032](../../02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md)). `exec` runs a
([ADR 0016](../../02-DECISIONS/0016-the-lab.md)). `exec` runs a
command on a machine and returns its output.
This has one consequence worth stating plainly: **a reachability question is asked from inside**.
03-DESIGN/01-to-be/04-lab-installation.md
+5 -5
View File
@@ -1,18 +1,18 @@
---
layer: to-be
status: designed
status: in-progress
code: [mesh-lab]
updated: 2026-09-11
decisions:
- 02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md
- 02-DECISIONS/0008-a-failed-step-fails-the-job.md
- 02-DECISIONS/0016-the-lab.md
- 02-DECISIONS/0010-delivery.md
---
# Installing the lab on a clean machine
The lab has prerequisites — a virtualisation daemon, copy-on-write storage, a pool, an identity
permitted to talk to it — and it cannot get them from the mesh, because it is where the mesh is
built ([ADR 0029](../../02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md)).
built ([ADR 0016](../../02-DECISIONS/0016-the-lab.md)).
So the lab needs an install path of its own. This describes it, and the shape it has to take is
determined by two failures observed while measuring
@@ -56,7 +56,7 @@ and unbounded at worst.
**The lab refuses to run degraded.** It does not warn and continue: a warning about a slow inner
loop is read once and ignored forever, and the loop stays slow. This is
[ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) applied where the failure is
[ADR 0010](../../02-DECISIONS/0010-delivery.md) applied where the failure is
performance rather than an error.
## Two ways the prerequisites arrive
03-DESIGN/01-to-be/05-the-node-host.md
+186 -28
View File
@@ -2,16 +2,19 @@
layer: to-be
status: in-progress
code: [mesh-host]
updated: 2026-08-26
updated: 2026-08-31
decisions:
- 02-DECISIONS/0030-the-repository-structure.md
- 02-DECISIONS/0036-a-node-is-a-managed-machine.md
- 02-DECISIONS/0037-the-host-applies-it-does-not-decide.md
- 02-DECISIONS/0038-a-node-joins-by-linking-first.md
- 02-DECISIONS/0039-the-link-is-the-security-boundary.md
- 02-DECISIONS/0008-a-failed-step-fails-the-job.md
- 02-DECISIONS/0041-the-host-depends-on-nothing.md
- 02-DECISIONS/0043-a-declaration-is-an-ordered-list-of-owned-resources.md
- 02-DECISIONS/0019-how-this-repository-works.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0010-delivery.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0005-the-node-host.md
---
# The node host
@@ -22,11 +25,11 @@ Tier 0. The one thing ever installed by hand, and the only thing that changes a
A **statically linked binary that requires nothing to be present** — copy it onto a machine and
run it, and that is the whole installation
([ADR 0041](../../02-DECISIONS/0041-the-host-depends-on-nothing.md)). Written in Go, because the
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). Written in Go, because the
job is system-level and because the host shares no code with any other tier.
A single binary with one job: **apply declared state on this machine**
([ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md)). Overlay
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). Overlay
membership, packet filtering, packages, services, containers and filesystems are not six
concerns it carries; they are six instances of the one.
@@ -58,7 +61,7 @@ returns it.
Three properties, each following a recorded decision:
**A failed step fails the apply.** Not "logs and continues"
([ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md)). A partial apply that
([ADR 0010](../../02-DECISIONS/0010-delivery.md)). A partial apply that
reports success is the mesh's most expensive shape.
**Each applier reads back.** Setting a value is not evidence the value took. The firewall is
@@ -66,7 +69,7 @@ asked whether the rule loaded; conntrack is asked what timeout it holds. This is
§5 as a component requirement rather than a review habit.
**What was applied is recorded after it works, never before**
([ADR 0035](../../02-DECISIONS/0035-a-picture-is-read-from-what-runs.md)). A failed apply leaves
([ADR 0018](../../02-DECISIONS/0018-a-picture-is-read-from-what-runs.md)). A failed apply leaves
the machine in whatever state it reached, and nothing must claim otherwise.
### store
@@ -75,17 +78,17 @@ Local, and **authoritative while disconnected**. Not a cache of the control plan
of what this node has applied and what it currently holds.
This is structural rather than convenient: if disconnection is an ordinary situation rather
than an exception ([ADR 0036](../../02-DECISIONS/0036-a-node-is-a-managed-machine.md)), the
than an exception ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)), the
store is what makes it ordinary. A laptop shut for a week comes back and reconciles; it does
not come back and ask what it is.
### link
The node's one connection to the control plane, and its security boundary
([ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)).
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
It is the broker connection that already exists
([ADR 0001](../../02-DECISIONS/0001-nodes-communicate-over-a-broker.md)) — outbound,
([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)) — outbound,
node-initiated, per-node addressed — carrying **per-node identity instead of a shared
credential**. The node owns no password. It owns an identity, and that identity is what it
presents.
@@ -105,7 +108,7 @@ architecture, a network position.
capability is real when it is present, running and working, and the difference is the whole
point of detecting it.
The profile is what makes [ADR 0036](../../02-DECISIONS/0036-a-node-is-a-managed-machine.md)
The profile is what makes [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
work: a node is a node, and what varies between them is here rather than in the definition.
### inventory
@@ -113,10 +116,24 @@ work: a node is a node, and what varies between them is here rather than in the
What this machine *is* — its identity, what it holds, what it has applied. Reported upward over
the link; never asked downward.
## What it is not
- It does not decide anything that needs another node.
- It never queries the mesh database.
- It has no listening surface.
- **It does not manage its own unit.** It manages `service` resources and its own unit is one —
the temptation is obvious and it ends with a host stopping itself half way through an apply,
leaving a machine with nothing running to fix it. The installation owns the host; the host owns
everything else.
**How it is installed, enrolled, run, upgraded and retired is
[`09-the-node-lifecycle.md`](09-the-node-lifecycle.md)**, in full and in one place. This document
is the component; that one is what happens to it.
## Where a declaration comes from
One behaviour, two sources
([ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md)):
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)):
| Situation | Source |
|---|---|
@@ -133,7 +150,7 @@ the mesh, and the full peer set arrives derived.
## What a declaration is
Settled by [ADR 0043](../../02-DECISIONS/0043-a-declaration-is-an-ordered-list-of-owned-resources.md).
Settled by [ADR 0005](../../02-DECISIONS/0005-the-node-host.md).
**JSON**, because the host has no dependencies to spend and the standard library carries no
YAML. **An ordered list of typed resources**, each with a stable identity — the order is stated
@@ -163,16 +180,57 @@ what it is. No control plane, no declarations, no network. Verifiable immediatel
the first node's path, and it is the claim the skeleton's Move 1 rests on and has never proved:
that one host can raise the substrate alone.
The first vocabulary is bounded by something the lab makes unavoidable: **a scenario is a
closed address space**, so a resource that must be fetched cannot be applied there at all. So
stage 2 begins with what needs no network — files, directories, service state — and the types
that need artifacts wait on where those come from, which is open in
[`02-scenario-declaration.md`](02-scenario-declaration.md).
Raising the substrate uses **four** shapes — `package`, `container`, `service`, `action` —
counted from the bundle that exists rather than reasoned about. `file` and `directory` are listed
below because they are the cheapest to be sure of and a substrate that needed them would find them
ready; the current bundle simply does not. **All of them are built:**
| | | |
|---|---|---|
| `directory`, `file` | **built** | no machine dependency at all |
| `service` | **built** | running/stopped **and** enabled/disabled at boot — a unit started but not enabled stops being true at the next reboot |
| `package` | **built** | present, never upgraded, and **never uninstalled** — the host cannot know what else needs it, so dropping one is *forgotten*, not *removed* |
| `container` | **built** | pinned by digest ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)); identified by a label carrying a digest of the declaration that made it, because a runtime normalises what it is given and that is indistinguishable from drift |
| `action` | **built** | bundle-only ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)); verify is mandatory and is the idempotency check as well as the read-back |
**A service says what it must reflect, and that is declared state rather than a command.**
`restart-on` names files whose change means the unit must be restarted — because a running service
does not re-read its configuration, and replacing a file, finding the service already running and
doing nothing leaves a machine behaving the way it did before while every check passes. A *command*
to restart would be an action, and the link may not carry one, so this is the shape that rule
leaves rather than a way around it.
**It may name a file another module put there**, written `<module>.<id>`. The case that needed it:
a resolver restarting when the mesh rewrites the names, which are computed by the mesh and belong
to its module rather than to the daemon's. Without it the daemon serves the names it started with
for ever — every machine that joined afterwards unreachable by name, and every check passing. An
unqualified name still means *my own*, so the common case reads as it always did.
**An action's verify is the definition of what the action is for**, and the action's own idea of
being finished must be the same one. *Written 2026-08-31, after this went wrong.* If an action
waits on one test and its verify reads back another, the two can disagree — and then the action
succeeds into a state its own verify rejects. The host says so accurately and uselessly: *the
action ran without error and its own verify still fails.* It is intermittent, it reads as a slow
machine, and the remedy people reach for is a longer timeout, which cannot help.
[04-ISSUES/017](../../04-ISSUES/017-an-action-succeeded-into-a-state-its-verify-rejects/00-report.md)
is that, in the one action the whole bootstrap depends on.
**The parser enforces the boundary rather than the caller remembering it.** `Parse` refuses an
action and is what the link uses; `ParseTrusted` permits one and is what the bundle uses. The
safe path is the default and the permissive one has to be named.
**The lab still cannot exercise the last three**, and that is now the only thing in the way: a
scenario is a closed address space, so nothing can be fetched there, and its machines carry no
container runtime. All three were instead verified against a real machine — a container created,
labelled, replaced when its declaration changed, exec'd into and removed; an action that exits
zero and satisfies nothing failing the apply. That is lab-installation work
([`04-lab-installation.md`](04-lab-installation.md)) rather than a constraint on the design, but
until it is done the substrate bootstrap has no end-to-end test.
**3 — link and store.** The node connects, receives declarations, and holds what it applied.
**4 — enrolment.** The one genuinely new mechanism in
[ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md); everything else there
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md); everything else there
is configuration of what already runs.
Stage 1 is deliberately the smallest useful thing. The lab currently raises **empty machines**
@@ -183,7 +241,7 @@ that, and every later stage is tested by a lab that already works.
**The lab is the harness.** A scenario places a host on a machine and asserts what it did —
against a real hypervisor, with the boundary never mocked
([ADR 0034](../../02-DECISIONS/0034-a-test-defends-a-decision.md)).
([ADR 0017](../../02-DECISIONS/0017-a-test-defends-a-decision.md)).
Each decision above owes a test:
@@ -195,6 +253,64 @@ Each decision above owes a test:
| 0036 — disconnection is a situation | a node cut off and returned reconciles without being re-adopted |
| 0008 — a failed step fails the apply | an apply with a failing step reports failure |
## What a machine says about itself, and what the mesh keeps of it
*2026-08-31, from finding that half of it was being discarded.*
**A capability is detected and never assumed** ([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)),
so the only account of what a machine can do is the one the machine gave. That account has two
halves and the mesh was keeping one:
| | |
|---|---|
| **the yes or no** | gates an assignment — *this machine has no seat, and nothing can be installed to fix that* |
| **the detail** | carries a value — `seat: card1-DP-1`, an architecture, an amount of memory |
They are **one fact read two ways**: *can this run here* and *what should it be configured as*. A
module that must not be assigned without an OLED panel and one that dims itself differently on one
are reading the same line. Keeping only the first read makes the second unanswerable, and there is
nowhere else to get it — the detector is the only thing that looked.
**The reason an absent capability is absent goes the same way, and it is the half a person needs
most.** *This machine has no container runtime* is the answer; *docker is not installed* is why.
The first is the mesh's to say and the second is only the machine's.
### The eight, and what each one is evidence of
*Written 2026-08-31 from `internal/profile/detectors.go`, because the set was implemented and
enumerated in no document. A vocabulary a module writes against, that exists only in code, is one
nobody can write against without reading the code.*
| capability | what a detection proves |
|---|---|
| `container-runtime` | a runtime is **running**, not installed |
| `package-manager` | the machine's own package manager works |
| `service-manager` | an init that can be asked for state — including *degraded*, which reports on stdout and exits non-zero |
| `firewall` | a filter this host can write rules into |
| `overlay` | the private network can be joined |
| `graphical-session` | a display server **is running** — state |
| `seat` | hardware where one **could** run — and assignment needs this one, not the row above |
| `privileged` | the host can change the machine |
**`seat` and `graphical-session` are the pair worth reading twice**, because collapsing them is
the obvious economy and it is wrong in both directions: a machine with a seat and no session can
be given a display server, and a machine with a session running is not thereby able to host a
second one.
**A detection runs something that only succeeds if the thing is *functioning*, never `--version`.**
A version string proves a binary is on disk, which
[`04-ISSUES/007`](../../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md)
records as false in the way that matters: the package was installed and the daemon was not
running.
**Never reported and reported nothing stay different.** One machine has not run the host yet; the
other ran it and can do nothing. Both refuse everything that requires a capability, and the
remedies are not remotely alike.
*Checked by recording a profile with a present capability carrying a value and an absent one
carrying its reason, and requiring both to survive — and by requiring a machine that never
reported to be distinguishable from one that reported an empty list.*
## Open
- **Whether one host can raise the substrate alone.** Move 1 assumes it. Stage 2 tests it, and
@@ -206,7 +322,49 @@ Each decision above owes a test:
package, a unit, a container and a dataset *are*. Nothing has measured that surface, and it is
the residue of the question [`host-size.md`](../../01-RESEARCH/006-mesh-from-scratch/host-size.md)
answered.
- **Rescue.** [ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md) suggests it is
- **Rescue.** [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) suggests it is
a node whose local state is discarded so the mesh re-derives it, and does not decide it.
- **What may expire.** An identity needing refresh to stay valid would make a laptop fail for
being a laptop ([ADR 0036](../../02-DECISIONS/0036-a-node-is-a-managed-machine.md)).
being a laptop ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
## What was added to the vocabulary, and why each cost was worth paying
*Written 2026-08-30. Every addition widens what a compromised control plane can express, so the
count is asserted by a test and a change to it is a decision rather than a convenience.*
Four shapes raise the substrate. Five more exist because most of what a person installs is not a
service:
| | why |
|---|---|
| **file**, **directory** | the substrate needs neither, and almost everything else does |
| **user** | a shell, a terminal, a chat client, a desktop are a package plus configuration **in somebody's home**. A mesh with no user owns `/etc` and nothing anybody looks at |
| **archive** | a theme is hundreds of files. Inlining them makes every declaration enormous and rewrites all of them when one changes |
| **network** | a module of several containers has to let them reach each other by name, and doing it with an action would create something nothing could ever remove ([ADR 0029](../../02-DECISIONS/0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md)) |
And `file` gained two fields: `bytes`, because a wallpaper is not a string, and `owner`, because a
file in a home belongs to somebody.
**`user` also makes a login shell declared state.** `chsh` is a command, the link may not carry
one, and a shell that could only be set by hand is a shell the mesh cannot manage — which is most
of the reason to manage a machine.
### The refusals that came with them
- **A file says what is in it exactly once.** `content`, `bytes` and `sealed` are exclusive, so
*what is in this file* is answerable by looking rather than by knowing which field wins.
- **Groups are added, never pruned.** The tool that sets them replaces the set unless told
otherwise, which would silently remove every group that makes a login able to use the machine.
A machine's own groups are not the mesh's to know about.
- **An archive is pinned by digest, checked before a single file is written.** This is the one
place the host reaches out on its own — everywhere else it holds one outbound connection and
fetches nothing — so the only thing making those bytes safe to unpack is that they hash to what
was declared.
- **An entry naming a path outside the archive is refused, not sanitised.** Rewriting it to land
inside would put a file somewhere nobody asked for and report success. The first implementation
quietly relocated it, and a test caught that.
- **Symlinks and device nodes are refused rather than skipped**, or an archive needing one arrives
silently incomplete.
**A partial host does archives and refuses users**: an archive needs a filesystem and a way to
fetch; a user needs a user database the host is allowed to write.
03-DESIGN/01-to-be/06-the-control-plane.md
+240
View File
@@ -0,0 +1,240 @@
---
layer: to-be
status: in-progress
code:
- mesh-control
updated: 2026-08-31
decisions:
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
- 02-DECISIONS/0008-a-context-owns-its-store.md
- 02-DECISIONS/0019-how-this-repository-works.md
---
# The control plane
Tier 2. The term appears seventy-nine times across this repository and was defined nowhere,
which is `how-we-build` §5 failing on this repository's own vocabulary.
This document defines it. It does **not** design the contexts inside it; those are open in
[research 006](../../01-RESEARCH/006-mesh-from-scratch/00-overview.md).
## The definition
> **The control plane is everything that needs to know about more than one node.**
That is the whole test, and it is not arbitrary — it follows from
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). The host applies and
does not decide *because deciding needs knowledge the machine does not have*. So the line falls
exactly there:
| Question | Whose |
|---|---|
| write this file, with this content, with this mode | the **host** — one machine |
| which nodes should run the store | the **control plane** — needs every node |
| is this unit running | the **host** — one machine |
| which peers belong in this node's overlay | the **control plane** — needs every node |
| what does this machine have installed | the **host** reports; the control plane **records** |
| has this node been unreachable for a week | the **control plane** — nobody else is watching |
A useful consequence: **anything a single machine could answer alone is not the control
plane's.** If it needs no second node, putting it here is a mistake, and the tier rule will not
catch it because the dependency direction is still correct.
## What is inside it
**Seven contexts and one interface**
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) —
each one earning its place by the test above rather than by being ours:
| | | needs to know about more than one node because |
|---|---|---|
| **inventory** | nodes, modules, assignments, versions | that *is* the mesh-wide fact |
| **config** | settings, secrets, and deriving them onto nodes | it derives **onto nodes** |
| **connectivity** | overlay, resolution, exposure, filtering, certificates — **specified in full in [`08-connectivity.md`](08-connectivity.md)** | who peers with whom; which node is reachable |
| **provisioning** | resource grants between modules | consumer and provider may be on different nodes |
| **delivery** | source to artifact to node | it targets nodes |
| **observability** | health, logs, metrics, alerts | *unreachable for a week* is nobody else's to notice |
| **identity** | agents, humans, services, authorisation | credentials follow an agent's node bindings and modality |
| **api** | the one interface every surface speaks to | — it is an interface, not a context |
**What is deliberately not here.** `work`, `knowledge` and `stream` are **mesh-hosted
applications** — first-party, shipped with everything else, and running on the mesh the way
anything else does. A task does not need to know a node exists, and *being ours does not make
something infrastructure*. `ai` is folded into `config`: a provider licence is an ordinary grant.
**`record` is an open question rather than an eighth entry.** Contexts integrate through it
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), which makes it load-bearing,
and [research 006](../../01-RESEARCH/006-mesh-from-scratch/skeleton.md) leaves *where it lives*
unresolved — putting it in the substrate risks recreating the circularity the tier design just
removed. Listing it here would settle by naming what has not been settled by arguing.
**One of the seven is built.** `inventory` owns a database of that name and holds the node records;
the rest do not exist. What it takes to run any of them — the language, and what must already be
running before it starts — is
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), which also records where the
build stopped and why: at **identity**, because what a node presents to prove who it is is not
decided anywhere, and a migration is the most expensive place in this system to guess.
**These are contexts, not services.** They are separate in the sense that matters — each owns
its own store, and they integrate through the record rather than by reading one another
([`how-we-build`](../../00-META/how-we-build.md) §4). They are not separate deployables, and
[research 011](../../01-RESEARCH/011-the-module-graph/00-overview.md) records why that
constraint is load-bearing: a single surface can compose them only while there is one interface
in front of them.
## Nothing outside a context touches its store
The question this answers: **can a node write to the registry database?** No — and not "only
through one node", which is the weaker arrangement it might be mistaken for.
> **No node holds a credential to any control-plane store, for writing or for reading.**
That is not a new rule here; it is four already taken, and it is worth seeing them together
because each one alone reads like a detail:
| | |
|---|---|
| [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) | the host never queries the mesh database |
| [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) | a node holds its own identity **and nothing else** — the shared database credential every node carries today is the exposure this exists to remove |
| [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) | a context is granted only what it **exclusively** owns: no shared writes, no read-only roles |
| [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) | there is no single mesh database, and nothing reads one |
### So how does anything get in
**Over the broker, as a message; the owning context writes.**
```
node ──event/report──► broker ──► the context that owns that data ──► its own store
```
A node reports what it applied, what it holds, and that it is alive. It **states**; it does not
**write**. The difference is the whole security boundary: a node that can write cannot be
prevented from writing anything, and a node that can only state has its blast radius bounded by
what the message vocabulary can say.
Reads work the same way in reverse — a node is *told*, in declarations. It never asks.
### Who actually consumes, and who writes
**The control plane is the consumer. There is one of it, and the context that owns the data does
the write.**
```
node ──► broker ──► the control plane, consuming
├─ a node reported what it applied ─► inventory writes the registry
├─ a node reported health ─► observability writes its own store
└─ a grant was requested ─► provisioning writes its own store
```
Seven contexts, **one deployable** — they are not separate services, so this is one process
consuming and dispatching internally, not seven consumers racing. Each context then writes only
the store it exclusively owns
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)).
**One consumer is a property worth having**, not just a consequence of
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). The as-is records that
*two consumers accidentally sharing one queue silently split the traffic between them, each
receiving half of what it expects* — which has happened, between a module's daemon and its
capability server. With one consumer that class of fault cannot arise.
**And the broker is the buffer while the control plane is down.** Nodes go on publishing;
messages queue; the control plane drains them when it returns. That is what makes
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)'s single control plane
tolerable — an outage delays the mesh's *knowledge* rather than losing it.
**With one consequence that must be bounded before it is discovered:** a queue with no limit
grows until the broker's disk is full, and the broker is the one component every node depends
on. Queues carrying node reports need a maximum length or a message lifetime, and losing the
oldest health report is obviously right where losing the oldest declaration acknowledgement is
not — so the bound is per queue and is not decided here.
### On volume, which is the real worry underneath
**Most high-frequency writes are not registry writes, and that is the first thing to check
before designing for throughput.** The registry is `inventory`'s store: nodes, modules,
assignments, versions. Those change when somebody changes something.
**Logs, metrics and health checks belong to `observability`**, which owns a different store
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)). Sending them to the registry
would be exactly the shared-schema mistake 0045 exists to stop, arriving through the back door
marked *performance*.
That leaves one genuine funnel: every context's writes go through the process that owns it, and
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) says there is one of it.
For a mesh of a handful of machines this is not a scaling problem, and **it should not be solved
by giving nodes database credentials** — that trades a bounded problem for an unbounded one. If
it ever binds, the answers are at the consumer: batch, apply backpressure, or move the highest
volume stream out of a relational store entirely.
**What observability actually stores its data in is not decided**, and it is the one place where
volume genuinely argues against a relational store.
## What it is not
- **Not the thing that changes machines.** It decides; the host applies. It never reaches into a
node except through the host.
- **Not a surface.** Tier 3 is how people and agents reach it. It has one interface; the
surfaces are what speak to that interface.
- **Not the substrate.** It *runs on* tier 1 — PostgreSQL, LavinMQ, an OCI registry
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) — and cannot start without
them, which is what makes them a lower tier.
- **Not privileged on a node.** It has no more access to a machine than the declaration
vocabulary allows ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
## It is also a consumer
The property that makes tier 2 unlike the others: **the control plane has requirements of its
own.** It needs a PostgreSQL database, an AMQP virtual host, and a bucket — the same things any
module needs, granted the same way.
That is the circularity the tiers exist to resolve rather than hide: the control plane cannot
provision its own database, because it is not running yet. So its **store** is raised from the
bundle the host carries, before there is a control plane to ask
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md),
[research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md)).
Its virtual host and its bucket are **not** in the bundle — by the time they are wanted there is
a control plane to grant them. Whether the bus must come first is
[open](07-the-substrate.md#open), and it turns on whether these contexts talk to each other over
it.
## Where it runs
**On nodes, like anything else.** It is not a place outside the mesh; it is modules the mesh
hosts, assigned to nodes by the same mechanism as everything else.
**One node runs it, and nothing takes over**
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The node is assigned,
never elected — no promotion, no quorum, no split brain.
That is sound rather than merely cheap, because the design already tolerates the control plane
being absent by construction: a node reconciles from **its own** store
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) and
never needed to ask anybody to hold the state it was last given. So the control plane being down
is not a new failure mode — it is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s ordinary disconnected
situation, happening to every node at once. **What is lost is change, not operation.**
The honest half: this node is a single point of failure, recovery is restore rather than
failover, and **certificate renewal is the clock** — an outage outlasting a renewal window expires
every public name.
## Open
- ~~**The contexts themselves.**~~ **Decided** — seven, by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
What remains open is narrower and named there: **where the record lives**, which research 006
leaves unresolved because the substrate is the one place it must not go.
- **How far it may be split.** One deployable today. Splitting a context out costs the single
interface a surface depends on
([research 011](../../01-RESEARCH/011-the-module-graph/00-overview.md)).
- ~~**How many run, and what a node does without one.**~~ **Resolved** by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md). What remains is
measurement: nothing reports how long the control plane has been unreachable, or how close a
certificate is to expiry — both needed for restore-not-failover to be a plan rather than a
hope.
- **What the interface is.** One interface is stated; its shape, and whether it is request,
subscription or both, is not
([research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md)).
03-DESIGN/01-to-be/07-the-substrate.md
+267
View File
@@ -0,0 +1,267 @@
---
layer: to-be
status: in-progress
code:
- mesh-host examples/substrate-first-node.lock
- mesh-host internal/apply
- mesh-lab test/integration/mesh.test.ts (a bare machine becomes a mesh)
updated: 2026-08-31
decisions:
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
- 02-DECISIONS/0007-connectivity.md
- 02-DECISIONS/0008-a-context-owns-its-store.md
- 02-DECISIONS/0019-how-this-repository-works.md
---
# The substrate
Tier 1. Defined the same way [the control plane](06-the-control-plane.md) is, because the same
gap applied: the word was load-bearing and unpinned.
## The definition
> **The substrate is what the control plane consumes and cannot grant itself.**
Every module that needs a database asks the control plane's provisioning for one. The control
plane needs a database too — and it cannot ask itself, because it is not running yet. That
circularity is not an awkwardness to work around; it *is* the definition. Anything on the wrong
side of it must be raised some other way, and the other way is the bundle the host carries
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
The test, applied:
| | control plane needs it | can it grant itself one? | |
|---|---|---|---|
| a relational store — **PostgreSQL** | its own state lives there | no — provisioning needs the store | **substrate** |
| a message bus — **LavinMQ** | it reaches nodes over it ([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)) | no — it cannot grant itself a virtual host | **substrate** |
| ~~an object store~~ | ~~artifacts and blobs it delivers~~ | — | **not substrate** — [ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) |
| ~~an image registry~~ | ~~images it delivers to nodes~~ | — | **not substrate** — needed to operate, not to start ([ADR 0033](../../02-DECISIONS/0033-the-substrate-is-a-store-and-a-broker.md)) |
| ~~an identity provider~~ | ~~only if it delegates authentication~~ | — | **not substrate** — it delegates to nothing ([ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md)) |
| ingress — **Traefik** | not to start; only to be reached by name | — it grants itself one afterwards | **not substrate** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) |
| anything else the mesh hosts | no | — | not substrate |
*The object-store row was wrong, and how it was wrong is worth keeping.* It answered *can it
grant itself one* — no, it cannot grant itself a bucket — while assuming the first column. **The
control plane does not need an object store**: it has no S3 client and never has, and artifacts
reach nodes as content-addressed blobs in the registry. The row was inherited from the system being
replaced, where an object store distributed module tarballs, and was never re-tested against the
definition above it. *Both columns must be answered, and the second is true of almost any service.*
**An object store is an ordinary module**, required through the module graph by whatever wants one.
A mesh with no workload needing one runs none.
**The role and the product are both written**, here and everywhere
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The role is what the argument
turns on — the test above works on roles, and would give the same answers for a different store.
The product is what actually gets installed and pinned, and a design that names only the role
does not record that the choice was ever made.
The dependency is on the **protocol**, not the product: AMQP for the bus, S3 for the object
store, the OCI protocol for the registry. That is what keeps the naming safe rather than a
commitment that cannot be revisited — replacing one is a substrate migration, not a redesign.
The store is the exception, and the exception matters: the provisioning model uses databases,
roles and schemas as PostgreSQL means them, so it is the one member that is not a swap.
## What that resolves
**Four or five?** [Research 006](../../01-RESEARCH/006-mesh-from-scratch/00-overview.md) asks
whether the identity provider is a substrate service, and the test answers it *conditionally* —
which is the honest answer rather than a number.
- If the control plane **delegates** authentication, it cannot serve anybody before the provider
exists, and it cannot grant itself a client. **Substrate.**
- If it **authenticates natively**, the provider is an ordinary hosted service like any other.
**Not substrate.**
So the count follows from a design decision that has not been taken, and the record should say
that rather than assert four.
**Why not "important infrastructure".** An identity provider, a mail server and an analytics
service are all infrastructure by any ordinary reading, and none of them are substrate — the
control plane starts and runs without them. *Important* is not the test; *the control plane
cannot obtain it* is.
## What the substrate is not
- **Not tier 0.** The host raises the substrate; it is not part of it. The host carries the
declaration that brings the substrate up, and depends on nothing.
- **Not the control plane.** These are services with no knowledge of the mesh. A store does not
know what a node is.
- **Not a place for logic.** The skeleton is explicit: tier 1 is *declarations only, no logic of
its own.* A substrate service is an upstream image, pinned, with configuration.
- **Not privileged.** The substrate is provisioned *from* by the control plane and grants
nothing on its own initiative.
- **Not the mesh's supply of anything**
([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)).
A substrate service and a module of the same product are **different instances**. The mesh's own
PostgreSQL and a PostgreSQL a workload was given are two servers, and a node hosting both runs
two containers — expected, not duplication to be tidied away.
The substrate is raised from the bundle before any mesh exists, so **it is not in the module
graph**: a workload depending on it would depend on something the graph cannot see, cannot rotate
a credential for, and cannot move. It would also put workload data in the store the control plane
keeps its own state in, where a workload that fills a disk takes down the one thing needed to fix
it.
## The pinned bundle
`substrate.lock` holds **what must exist before the control plane runs** — which is a smaller
set than the substrate, and the difference is easy to miss. It is the only place in the mesh
where versions are pinned by hand rather than resolved.
Being substrate and being in the bundle are two different questions:
| | is it substrate? | must it precede the control plane? |
|---|---|---|
| PostgreSQL | yes — the control plane's own state lives in it | **yes** — there is nowhere to put that state otherwise |
| LavinMQ | yes — it cannot grant itself a virtual host | **yes** — the control plane reaches a node only over the link, and the link is the broker ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
| the OCI registry | yes — it cannot grant itself a repository | no — the first node fetches upstream ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)) |
The registry is **substrate by role and ordinary by delivery**: by the time it is wanted there is
a control plane, and it provisions it the way it provisions anything. That keeps the bundle small
enough for the review [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) requires — one
substrate image until
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) established that the
broker has to precede the control plane, and two since.
*Corrected 2026-08-31, from counting what the bundle holds rather than reasoning about it.* **It
carries three images, not two** — PostgreSQL, LavinMQ, and the control plane itself, which the
sentence above had overlooked by counting only substrate services. The control plane is what the
substrate exists to start, and it is in the bundle for the same reason they are: there is nothing
to fetch it with yet. It also carries seven actions, a package and a service.
**Why pinned:** the bundle is applied when no mesh exists, so nothing can resolve a version, ask
a registry, or check a constraint. What the host carries must already be exact.
**Why references and not payload:** the bundle names images by **digest** and the host fetches
them ([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). A first node is
a real machine with a network; the sealed case is the lab, and the lab places images itself.
Reproducibility comes from pinning the identity of a thing rather than carrying its bytes, which
is what keeps the bundle small enough for a person to read and check.
## Raising it
The order, from [research 011](../../01-RESEARCH/011-the-module-graph/worked-provider.md):
```
0 a container runtime exists detected — docker or podman — or installed
1 PostgreSQL runs pulled by digest, from the bundle
2 a database per context is created an action, run locally — one today, `inventory`
3 each context's schema is applied an action, against its own database
4 LavinMQ runs pulled by digest, from the bundle
5 a virtual host, a credential, and actions, run locally
a self-signed certificate
6 the control plane starts and only now is there a mesh
7 the registry, and everything else the ordinary path
are provisioned
```
**Steps 4 and 5 are why the bundle is not one image**
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). The control plane cannot
provision the broker, because provisioning means telling a host, and telling a host happens over
the broker. The first node does not escape this by being local: it enrols the ordinary way, by
dialling the broker at the address in its token.
**Step 2 is one database per context and not one called `mesh`.** A context is granted only what it
exclusively owns ([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)), *the mesh
database* names a thing that will not exist
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)), and a separate
database is a boundary a cross-context join cannot casually cross where a separate schema is not.
Only PostgreSQL is raised from the bundle, for the reason in *The pinned bundle* above — the
rest of the substrate is wanted only once there is a control plane to provision it.
**Step 0 is easy to leave out and it is where several things meet.** A substrate service is a
container, so a container runtime must be working before anything else happens — and a runtime
is a *package*, not a container.
**Which runtime is detected, not chosen**
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)): a machine that
already has one keeps it. On a machine with none, the control plane names the package, because
what it is called differs per system. It is:
- what the host's capability detection already reports, and the first use of that report by
something other than a person;
- **adopted rather than installed** when the machine already has one with configuration somebody
chose ([research 012](../../01-RESEARCH/012-the-minimum-viable-node/00-overview.md));
- a package, which needs the machine's own package manager and a network — both permitted by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md).
So the bootstrap uses four shapes: **package**, **container**, **service** and **action** —
*counted from `substrate-first-node.lock`, which is the only bundle there is*. It had said six,
adding `file` and `directory`, which this bootstrap never asks for.
All four are built, as are the host's other five
([`05-the-node-host.md`](05-the-node-host.md) stage 2), so nothing in this bootstrap is blocked
on the host any longer — which is the claim that mattered, and it was true either way.
**Steps 2 and 3 happen before there is a mesh to do them**, which is why provisioning is part of
the bootstrap rather than a service consumers use later. They are **actions** the bundle
declares and the host runs
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) — so the
host's vocabulary grows by one shape rather than by one resource type per substrate service.
## Open
- ~~**Whether identity is the fifth.**~~ **Closed 2026-08-31** by
[ADR 0031](../../02-DECISIONS/0031-the-control-plane-authenticates-nobody.md): the control
plane delegates authentication to nothing, so identity is an ordinary module. With the object
store gone ([ADR 0028](../../02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md))
the substrate is three, and no member is conditional.
- ~~**Whether the bus must precede the control plane.**~~ **Resolved** by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) — it must, and the question as
posed here could not have answered it. This asked whether the control plane's contexts talk to
each other over the bus; they do not, being one process, which under this framing would have
kept LavinMQ out of the bundle. What decides it is how the control plane reaches a *node*, which
is only ever over the link.
- **What issues the broker's certificate at bootstrap.** New, and created by the row above. A
token pins the fingerprint a host must expect before it sends anything
([`09-the-node-lifecycle.md`](09-the-node-lifecycle.md)), so the broker needs a certificate at a
moment when there is no mesh to issue one and no public name to obtain one for. Self-signed and
pinned is the shape that fits; how it is later replaced by the certificates in
[`08-connectivity.md`](08-connectivity.md) is not decided.
- **How a context added later gets its database.** By then there is a control plane — but one
holding a credential that can create databases holds more than what it exclusively owns
([ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)).
- ~~**Whether the host can do step 2.**~~ **Resolved** by
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md). A service
running on this machine is part of this machine, so the scope was never in question — the real
question was whether the host must learn what a database is, and it must not. The bundle
declares an **action**; the host runs it and verifies it, and what a database means stays with
the module that provides one.
- **Whether one host can raise all three.** The claim under stage 2 of
[the node host](05-the-node-host.md), never proved. If it is false, the tier boundary moves.
- **How the substrate is updated once a mesh exists.** Pinned by hand at bootstrap; afterwards
the control plane could deliver it like anything else, and nothing says whether it does.
## Raised, and observed
*Written 2026-08-30, the first time a bare machine became a running mesh and something joined it.*
**It works, and what that means precisely:** a machine with a container runtime and nothing else
applied the bundle its host carries and ended with a store, a database per context, those
contexts' schemas, a broker holding a certificate it generated itself, and the control plane
serving on top of them. Eleven resources, one command, no mesh to ask anything of.
**Then it joined itself.** The same machine took a token, checked the broker against the
fingerprint pinned in it, generated three keypairs, and enrolled — which is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *the first node is a node whose
mesh is not up yet*, observed rather than argued. Its specialness lasted one command.
**And a credential crossed.** With a second node recorded, the machine was declared the provider
of a database and pushed to over the broker. What arrived and what did not is the whole of the
[secrets argument](../../02-DECISIONS/0009-modules-and-the-graph.md), measured on a real machine:
| | |
|---|---|
| the password, in plain text | **on the machine only**, one file, mode 0600 |
| in the declaration that crossed the broker | absent |
| in the control plane's database | absent |
| in what the node reported back | absent |
**One fault, and it was in the joining.** The token did not say what the mesh calls the machine,
so enrolment needed a flag its own help said it did not — and failed at the broker with an empty
username. Recorded in ADR 0004 as the fifth thing a token carries.
03-DESIGN/01-to-be/08-connectivity.md
+574
View File
@@ -0,0 +1,574 @@
---
layer: to-be
status: in-progress
code:
- mesh-control internal/catalogue/filtering.go
- mesh-control examples/route-proxy
- mesh-control internal/identity/authority.go
- mesh-host internal/identity/serving.go
- mesh-host internal/apply (the service that reflects a rule set)
updated: 2026-08-31
decisions:
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0007-connectivity.md
- 02-DECISIONS/0007-connectivity.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0007-connectivity.md
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
---
# Connectivity
One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most
moving parts: **overlay, resolution, exposure, filtering, certificates.**
It is written as a whole because the five are one design. They share inputs, they must agree, and
every one of them today is computed in a different place by a different module from a different
copy of the same facts.
## Why it is control-plane work
Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one
node* — to each responsibility:
| | needs to know | whose |
|---|---|---|
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane |
| **resolution** — which name is which node | **every node** | control plane |
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | control plane |
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies |
| **certificates** — who may present which name | which name belongs to which node | control plane |
**Not one of the five can be answered by a machine on its own.** That is the whole reason this is
a context rather than a set of node-local modules — and it is exactly what the current
arrangement gets wrong, by computing all five on the node from a direct database connection.
## The shape: decided centrally, delivered as files
Every one of the five resolves the same way, and it is worth stating once rather than five times:
> **The connectivity context computes the configuration. It arrives over the link as `file`
> resources. The service reads files and knows nothing about the mesh.**
This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already
exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*.
It is also what removes the last two upward dependencies.
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and
they are the reason every node permanently holds a credential to it
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity
modules. **Closing this context closes that set.**
### And they are modules, not a second mechanism beside the module system
*Written 2026-08-29, from building it. The first version was code beside the module system doing
the module system's job, and the fault it produced is the point of writing this down.*
**A machine was on the private network because it had an address.** Every node that had been
placed got a peer list, whether or not anybody wanted it there, and there was no way to say a
machine should stay off. That is what "special-cased" cost, and it was invisible until somebody
wanted the exception.
**What made it look unavoidable:** a peer list cannot be written in a manifest. It is derived from
every other machine, so it differs on each one and changes when any of them changes. So the
manifest says its resources are **computed** — it names something in the control plane that works
them out per node — and it is a module in every other respect: assigned, resolved, configured by
settings, and absent from a machine nobody gave it to.
**What that made possible immediately** is the arrangement below, which the code has:
| module | provides | requires | claims |
|---|---|---|---|
| the WireGuard one | a private network, **and the mesh's own addressing** | | *the* private network, one per node |
| the names one | name resolution | the mesh's own addressing | |
| `networking` | | both of the above | |
**Three rather than one, because WireGuard is one VPN of several.** Naming the module after the
job — `networking` — and putting WireGuard inside it is the retired *flavor* idea wearing a
generic name: the second VPN has nowhere to go. So a module is named for what it *is* and declares
what it *does*, and `networking` is the third row — requirements and no files
([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)).
**Names left the WireGuard module for their own.** They had been delivered inside it, on the
argument that a machine with peers and no names is half on the network. True, and the wrong place
to fix it — names are identical over a *different* private network, so bundling them made one
module out of two things. They require the mesh's **addressing** rather than a private network in
general, because that is what they are computed from: over a VPN that hands out its own addresses
the mesh has nothing to write, and refusing is what stops a machine being given a hosts file that
means nothing on it.
**And the claim is not decoration.** Choosing a different VPN still installed WireGuard — dragged
back in by the names, which needed addresses only WireGuard hands out — and nobody was told.
Running two VPNs is not always wrong; being *the* one the mesh runs over is singular. So it is a
claim, and the collision is refused by name.
**And the proxy's half, which was the other module reaching into the database.** A web application
requiring a reverse proxy has to say *which name, which port*, and there was nowhere to put it —
`requires` says a thing must exist and never said what to do with it. A module now contributes to
a requirement, the control plane collects every contribution on a node, and the provider is given
them as a file at a path it named. It reloads when that file changes, by the same `restart-on` the
private network needed when a peer list changed under a running interface.
**The proxy's configuration is not written by the mesh.** It is given the facts and turns them
into whatever it runs, which is why swapping Traefik for something else touches nothing that
publishes through it. See [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) for the
other direction — handing a credential *back* — which is the larger half and is not built.
**What is still not a module, and why that is correct.** The host needs none of this. It has an
address and a route before the mesh exists — that is the machine's own networking — and the
broker's address is carried in the token rather than resolved
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). **The one connection that
carries modules cannot itself be one.** Everything above it can be, and now is.
## The order it comes up in
The one thing to get right, because everything else depends on it:
```
0 the node has an underlay address the machine's own — DHCP, or a provider gave it one
1 the node dials the mesh OVER THE UNDERLAY, at the address in its token
2 it proves itself, and is proved to the link exists (ADR 0004, ADR 0004)
3 the mesh grants it an identity and an overlay address
4 the overlay comes up peer graph delivered as files
5 names resolve resolver config delivered as files
6 filtering is applied derived from what is assigned here
7 routes and certificates once this node has something to expose
```
**Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not
be created: the overlay is configured by the mesh, so a link that required the overlay could
never be established on a new node. The link stays on the underlay permanently — it is
outbound-only and carries its own identity, so it needs nothing the overlay provides.
**Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address*
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Today this is
patched with an `/etc/hosts` floor written underneath the resolver; under this design there is
nothing to patch.
**Step 1 has a precondition this document treated as a fact to record rather than a requirement:
the broker's node must be dialable by every node, at a stable address, and so must the hub**
([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Across the internet that means publicly
reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and
a broker node whose address moves invalidates every token issued for it.
**Whether the link should later move onto the overlay, with the underlay as fallback, is
[open](../../02-DECISIONS/0007-connectivity.md).** It is a decision rather than a derivation: the
gain is which network carries bytes, not what an attacker can reach, since the link is already
encrypted against a pinned fingerprint.
## 1 — The overlay
**What is decided:** the peer graph. For every node: its overlay address, which peers it holds,
which of those it may dial, and which must dial it.
**Inputs, all declared:**
- **reachability** — an endpoint, or none
([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Not
inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and
wrong for a routable address behind a closed firewall.
- **site** — where the machine physically is, or nothing if it roams.
- **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a
renumbering is an outage and nothing can be asked which node is the hub.
**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
public key is published to the mesh. This is already true and it is already right — it is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own
identity* applied to the overlay, and it means the control plane computes a graph it cannot
itself impersonate.
**Shape: a hub, with direct peering between co-located nodes.**
| | |
|---|---|
| two nodes at the same site | peer **directly**, host-routed, with a keepalive |
| everything else | routes through the **hub** |
| a node with no site — it roams | **hub only** |
**Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a
preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does
not fall back to the general one. So a node whose location changes gets exactly one path, because
two paths would mean one of them silently swallowing traffic.
**What the host receives:** an interface configuration and a peer list, as files. It does not
compute them, and after this it holds no credential to the mesh's database.
### Four things the lab found, none of them visible from the mesh's own state
*Written 2026-08-29, on the first three machines to actually run this.*
Each looked like a working network from every angle the mesh can see: the graph was right, the
files were right, the services were up, and every node reported success.
- **A running interface does not re-read its configuration.** A node joins, every existing node's
peer list changes, each file is replaced — and the service is already running, so nothing
reloads it. Every existing node keeps a network that no longer exists. The declaration has to
say the service must *reflect* the file, which is declared state; a command to restart would be
an action, and the link may not carry one
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)).
- **A hub that shares a site with a spoke was emitted twice** — once as a direct peer and once as
the route of last resort. WireGuard takes one entry per public key, so the interface refuses the
file. The ordinary shape of a small mesh, and in none of the tests written before it ran.
- **Two nodes at one site that neither can be dialled must not peer directly.** Nobody opens the
path, and the direct route is more specific than the hub's, so it wins and blackholes. This
document's own warning, arriving in its implementation: *a more specific route to a dead
endpoint blackholes; it does not fall back to the general one.*
- **The container runtime closes the door the overlay needs.** Docker sets the FORWARD policy to
DROP, so a hub with `ip_forward` enabled still carries nothing between its spokes. The substrate
at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The
hub inserts its own rule above those chains and removes it on the way down.
**The pattern in all four:** the mesh's picture of the network was correct and the network did not
work. That is the argument for the lab in one line — none of these is reachable by reasoning, and
each was found within minutes of a real machine trying it.
## 2 — Resolution
**Two name spaces, and they do not mix:**
| | resolves to | certified by |
|---|---|---|
| **internal names** | overlay addresses | the **mesh CA** |
| **public names** | whatever the outside world must reach | a **public authority** |
A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact
used by things outside the mesh — and the separation carries two lessons that were learned
expensively enough to be worth restating:
- **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces
a delay and a failure mode that appears on one node and not others — the worst shape a fault
can have.
- **A node must not pin its own public name locally.** The duplicate record breaks resolution of
that name for everything else that needs it.
**What the host receives:** the resolver's configuration, as files, listing every peer's internal
name and overlay address.
**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.
### Names, and what a container can see
*2026-08-31, from a container that could not resolve a name every machine could.*
Internal names are `<node>.internal` — the suffix is the one IANA reserved in 2024, so a name that
leaks into a public resolver fails rather than reaching a stranger's machine. They are computed
centrally, because a name set needs every node at once, and written to each machine's hosts file.
**A file rather than a resolver**, and the reasoning holds: it works on every Linux, needs no
package, and has no failure mode of its own. The stated trigger for a daemon was *names that are
not one-per-node* — service names, wildcards.
**But a container does not inherit the machine's names.** It gets its own hosts file holding only
its own hostname. So every name the mesh wrote was invisible to the majority of things that need
one — and *on the machine it always worked*, which is exactly what made it easy to miss. It was
found by a database client on one node failing to resolve another node, on a mesh where both names
were correct and present on both machines.
**So the mesh gives its names to the containers it declares**, written into each container's own
hosts file by the runtime. That extends the file decision rather than overturning it. Given by the
mesh and not chosen by a module: a module that listed the machines would go stale the day one
joins, and a module that did not would be one whose containers cannot reach anything by name.
**The boundary, which is deliberate and worth stating:** *declared* containers. A container
somebody starts by hand is not the mesh's to configure, and reaching into every container on a
machine — declared or not — is what a nameserver in `resolv.conf` would be for.
### The resolver, built
*2026-08-31.* **A service is reached at `<service>.<node>.internal`** — the first label is the
service, the rest is the node — so what resolves is *anything under a node's name*, going to that
node. What routes it once it arrives is a proxy's, and stays separate.
**The mesh writes the data and runs no daemon.** One wildcard per machine, from the same set that
writes the hosts file. A resolver is third-party software and runs *on* the mesh rather than being
*of* it: the mesh has no business shipping one, choosing which one, or knowing its configuration
language. Swapping dnsmasq for unbound changes that module and nothing in the control plane.
**Two roles, two claims, because they are different things.** systemd-resolved cannot answer a
wildcard at all — it routes the mesh's suffix to something that can. Treating serving and asking
as one role produces a module that cannot work.
| | claims | |
|---|---|---|
| serving | `the-dns-port` | answers the wildcards |
| asking | `the-resolver-configuration` | decides what the machine asks |
So *which* resolver is not a mesh-wide decision. One machine can use what systemd already owns and
another can run dnsmasq, and two of either on one machine is refused rather than fought over.
**Two things a resolver must not do**, both found by a machine rather than by reasoning:
- **Take an address something else holds.** systemd-resolved holds `127.0.0.53` *and* `127.0.0.54`.
- **Read `resolv.conf` for its upstreams.** Whatever points a machine at the mesh writes the
resolver's own address there, so it becomes its own upstream and every query it cannot answer
loops until its receive queue fills. It needs no upstream: only the mesh's suffix is routed to
it.
*Checked on two machines, through the path an application takes — nsswitch, files, then DNS —
because the module deciding what the machine asks is half of what is being tested and only that
path goes through it.*
**That is now the second reason to want a resolver**, and it is a different one from the trigger
above:
| | |
|---|---|
| names that are not one-per-node | a service named under a machine — `postgres.novox.internal` |
| containers the mesh did not declare | anything a person or another tool starts on a node |
### Which resolver is not a question the mesh answers
*Written 2026-08-31, after treating it as open when it had been decided two days earlier.*
**A resolver takes over `/etc/resolv.conf`, which is a singular resource, so it is a claim** —
[ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) lists it in the table beside the seat
and pid 1. Choosing between resolved, dnsmasq and unbound is **assigning a module**, per machine,
and two of them cannot both be assigned there:
> `resolved-config and dnsmasq both claim "/etc/resolv.conf", and only one thing may hold it per node`
So there is nothing global to settle and nothing for the mesh to guess. One machine can use what
systemd already owns and another can run dnsmasq, and neither has to know about the other.
**What the mesh contributes is the part only it can know**: which machines exist and where. That
is `mesh-resolver`, which writes one file and holds no claim, because writing a file takes nothing
over. A daemon module requires that data and claims the resolver — so swapping the daemon changes
that module and nothing else.
**This was recorded on 2026-08-29 and reopened as an unanswered question on the 31st.** Which is
the argument for the table in ADR 0009 being a table: the pattern is only obvious once seen, and
the cost of not seeing it is inventing a mechanism that already exists.
## 3 — Exposure
Settled by [ADR 0007](../../02-DECISIONS/0007-connectivity.md); summarised here because
this is where it belongs.
**A route is a grant.** A module that must be reachable declares it needs one; the proxy provides
it and hands back the public name. Ordinary
[ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)
vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a
name rather than supplying nothing and receiving credentials.
**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.
### What was built
*2026-08-31.* Nothing new in the vocabulary, which was the claim and is now the fact: a route is a
provision, a proxy provides it, and a module that must be reachable requires it. The consumer
contributes the name it wants and the port it listens on; the proxy receives every consumer that
asked; the consumer is told what the provider serves, which is how it knows its own name.
**One field was missing, and it is the one anything reaching back needs.** A contribution now
carries **where the mesh says that machine is**. A database is reached *by* its consumer, so the
mesh never had to tell a provider where anybody was; a proxy is the other direction — it is told
to send traffic to a consumer and has to open a connection. Without it every provider implementing
a provision would have to know how the mesh names machines, which is a convention leaking into
every module.
**Exposure and filtering are different questions and a module answers both.** A workload says what
it listens on and who may reach it; separately, it says it wants a route. A module that asked for a
route and not for the port is unreachable by the proxy it just asked for — which the lab
demonstrates, because the machine is already filtering by the time this runs.
**Withdrawal, which was open above.** The file the proxy is given is the whole truth about who has
a route, so a proxy replaces its table rather than merging. Merging would keep serving a name whose
module was unassigned — and *a stale public name pointing at nothing fails more visibly than a
stale grant* is the reason it must not survive, not a reason to tolerate it.
**A name a proxy does not serve is refused by saying which it does.** A route that was withdrawn
and a name that never existed are different things, and a bare 404 makes an operator go and read
the mesh to tell them apart.
*Checked in the lab by a request to the name reaching the workload across the private network and
returning the workload's own answer, then by unassigning the module and requiring the same request
to stop working.*
## 4 — Filtering
**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a
consequence of what runs on it and who must reach it, not an independent declaration to keep in
step by hand.
**A rule names its source** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
A rule with no source is open, and must say so rather than appear to restrict something. `scope:`
is removed rather than implemented: five manifests carry it today, it is referenced by no code,
and it is the clearest instance in the repository of *an unenforced rule is indistinguishable
from a wrong one, and costs more, because people believe it.*
**Unknown keys are refused** — the discipline the host's declaration parser already has
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and
the one manifests lack. `scope:` survived because nothing rejected it.
### What was built
*2026-08-31. Everything above was the intention; this is what exists, and how each part is
checked. [04-ISSUES/003](../../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) is
resolved by it.*
**A module says what it listens on**, as a port, a protocol and a source — `mesh`, `anywhere`, or
`machine`. The source is required and there is no default, which is the whole of *a rule names its
source*: a manifest that omitted it would read as a restriction and be none. *Checked by a manifest
with a port and no source being refused, and by one naming a source the mesh cannot render being
refused as well — the second is what stops a source becoming a comment.*
**The set is derived per node**, from every module assigned to it, not from the module asking for
it. Where two modules want the same port, the wider source wins and both are still named, because
removing one of them must not read as a reason to close a port the other needs. *Checked by
rendering a node whose firewall module has no ports of its own and asserting another module's port
is in the result; and by giving one port two modules and one source each, and asserting the
narrower rule disappears while both names survive.*
**What is not declared is closed.** The rule set drops by default. *Checked by naming the input
chain in the assertion rather than the policy alone — the first version of that test passed while
input accepted everything, because another chain in the same file also said `policy drop`.*
**From the mesh means the machines the mesh has**, as their addresses on the private network, not
as a subnet. A subnet is a guess that stays wrong quietly; the address set shrinks when a node
leaves and nobody edits anything. A machine that asks for `mesh` where the mesh knows no addresses
is **closed and told so in the file** — widening it would open a port nobody asked to open, and
dropping it silently would close one somebody did.
**Three things it deliberately does not do**, each of which looked right and would have broken
something:
| | why not |
|---|---|
| decide what the machine **forwards** | the container runtime writes its own forwarding rules and a second policy is consulted alongside them, so a drop here stops every container on the node — the control plane included. Nothing in a manifest says what a machine routes, so there is nothing to derive it from either |
| **flush the ruleset** when loading | that empties every table on the machine, the runtime's among them. Only the mesh's own table is replaced, and it is declared empty first so the replacement works on a machine loading one for the first time |
| carry a **command to load itself** | the link may not carry an action ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A service is declared to reflect the file instead, so replacing it restarts what loads it — the shape that rule leaves, used here for the first time for its real purpose |
**A module that wants a rule set brings the unit that loads it.** Found the hard way: the unit a
distribution packages for nftables runs, applies the rules and exits, so it is neither running nor
stopped — and a host asked for a service that is "running" reports, quite correctly, that it is
stopped. Every packet was filtered exactly as declared and the machine was marked as not doing what
it was told.
**The vocabulary has no word for "ran, did its job, and exited"**, and that is a real gap rather
than a wording problem: the whole class of configuration-applying units — packet filters, sysctl,
tmpfiles — is shaped that way. Until there is one, a module ships a unit that stays, which is also
the better shape: how a machine enforces rules is a fact about the machine, and the mesh has no
business depending on what a distribution happens to package.
**One rule is derived from the overlay's shape rather than from what is assigned: a hub's own
listening port.** A hub accepts inbound connections from every node at other sites; a machine that
is not a hub dials out and needs nothing open, because a reply to a flow it started is already
accepted. The two want different rules on an *identical module*, so `listens` — a static field —
cannot say it. The machine a static answer gets wrong is the one facing the public internet, which
is the machine that most needs filtering.
*Recorded as a gap on 2026-08-31 and closed the same day.* **A computed module now contributes
listens the way it contributes resources.** The port comes from the endpoint, which is where the
interface takes its `ListenPort` from — one source, so a rule set cannot open a port the interface
is not on. It is open to *everywhere* deliberately: a node at another site is not on the private
network until this port lets it on, so restricting it to the mesh would be a rule that can never
be satisfied by the thing it exists for.
**A generator that cannot say what a machine opens is refused, not read as silence.** Closing a
port on the evidence of a failure to look is how a machine is severed by a fault somewhere else —
and the machine it would sever is the hub, whose only route to being repaired is the network it
just closed.
*Checked by filtering the hub and then requiring the mesh to keep working: a declaration still
reaches the other machine, and the other machine still reaches the hub. A rule file that looks
right and a mesh that has stopped are exactly what that guards against.*
**And it is enforced, which is what separates this from `scope:`.** Checked on two real machines:
two ports opened, one declared, and from the other machine the declared one answers and the
undeclared one does not — then the module is removed and the port closes with nobody editing a
rule. *A rule set that is written but never loaded passes every check that reads the file, which
is why the check reads packets.*
## 5 — Certificates
**Two authorities, kept separate on purpose.**
| | issued by | for |
|---|---|---|
| **public names** | a public ACME authority | anything outside the mesh reaches |
| **internal names** | the **mesh CA** | node-to-node, over the overlay |
**The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living
in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA
unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)).
**Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered
at the name being certified, so issuance happens through a publicly reachable node regardless of
where the workload runs — the same asymmetry as exposure, for the same reason.
**The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore
defaults to the public authority's *production* endpoint. Two consequences, and the second is
worse than the lab problem that found it — every certificate experiment on a real node consumes
production issuance quota, and a retry loop can exhaust it for a week.
**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the
fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)),
so nothing needs the CA before membership. It certifies internal names afterwards, and that is
all it does.
### What was built
*2026-08-31.*
**A node generates a fourth key**, and the reason is the one the other three already give: *a key
used for two purposes is one rotation away from breaking the other.* The identity key would work
for TLS and reusing it would mean rotating a node's identity every time its certificate is
replaced. The private half never leaves the machine; the mesh is told the public half at
enrolment.
**So there is no certificate request and nothing to seal.** The mesh signs a statement binding a
public key to a name it alone assigns, which is the whole of what a certificate authority does.
It issues rather than stores: the node's key does not change, so signing again produces an equally
valid certificate and there is nothing to keep in step.
**A machine with no name inside the mesh is refused**, not given a certificate for nothing. A
certificate for a name nothing resolves is a certificate nothing can check.
**And the key is stored in the format a server reads** — PKCS#8 PEM, not the host's own encoding.
That is not an implementation detail of whoever writes the file: the file exists *because
something else reads it*, so the format is the interface
([04-ISSUES/014](../../04-ISSUES/014-a-key-that-is-present-and-unusable/00-report.md)).
*Checked by a real handshake between two machines: one serves on its internal name with the key it
generated, the other verifies against the mesh's authority and nothing else. Every cheaper check
passed while the server could not start — the key was present, the certificate was valid, and
nothing read either the way a server would.*
## What this removes
The list is worth having in one place, because it is most of the argument:
- **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only
two, both connectivity.
- **Therefore the database credential on every node**, and the object-store credential beside it.
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s central claim becomes
true rather than aspirational.
- **The `/etc/hosts` floor**, and the bootstrap circularity it patched.
- **Hub election by address prefix**, and the silent no-hub failure when nobody knew the
convention.
- **The RFC1918 inference**, and the lab substitution that existed to satisfy it.
- **`scope:`**, and the class of manifest key that means nothing.
## Open
- ~~**What happens when the hub is down.**~~ **Resolved** by
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s
matching question — they were one question. Nothing takes over. WireGuard has no failover, the
hub is declared rather than elected, and non-co-located paths stop while co-located direct peers
and every already-assigned workload keep running. The recovery path is restore, and its deadline
is certificate renewal.
- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from
an address, but no procedure exists, and a graph delivered node by node has an ordering problem
while it is half-applied.
- ~~**Revoking a route** when a module is unassigned.~~ **Resolved** 2026-08-31 — see §3. The file
a proxy is given is the whole truth about who has a route, so a route does not outlive the module
that asked for it.
- **IPv6.** [ADR 0007](../../02-DECISIONS/0007-connectivity.md) makes
it expressible; nothing here says the overlay or the resolver handle it.
- **Reporting declared-versus-observed.** ADR 0007 makes the disagreement detectable and does not
say who looks or what they are told.
03-DESIGN/01-to-be/09-the-node-lifecycle.md
+696
View File
@@ -0,0 +1,696 @@
---
layer: to-be
status: in-progress
code:
- mesh-host internal/link/run.go
- mesh-host internal/link/enrol.go
- mesh-host packaging/nox-mesh-host-resume.service
- mesh-host packaging/nox-mesh-host-network.sh
- mesh-control internal/token
- mesh-control internal/inventory/nodes.go
updated: 2026-08-31
decisions:
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0010-delivery.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0005-the-node-host.md
---
# The node lifecycle
How a Linux machine becomes a node, stays one, and stops being one.
[`05-the-node-host.md`](05-the-node-host.md) describes the host as a component. This describes
it as something that runs for years on a machine somebody else also uses — which is where the
questions that were not being asked live.
## The states
```
unmanaged ──install──► hosted ──enrol──► enrolled ⇄ disconnected
▲ │
└─────release──────┘
```
| State | Has | Can |
|---|---|---|
| **unmanaged** | nothing of ours | — it is a Linux machine |
| **hosted** | the host, no identity | apply a local file, apply its bundle |
| **enrolled** | identity, link, store | everything; this is *a node* |
| **disconnected** | identity, store, no link | hold its machine in the last state it was told |
**Only `enrolled` and `disconnected` are nodes**, and they are the same node in two situations
rather than two kinds of thing
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). **`hosted` is not a
node** — it is a machine with a program on it that has not been told which mesh it belongs to.
There is no state for *the first node*. That is the point of
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md): the first node walks the
same path, in an unusual order.
---
## unmanaged → hosted: installing
In the machine's own idiom, because the package manager and the init file are the system's
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)):
```
# Alpine — the intended first node
apk add nox-mesh-host
rc-update add nox-mesh-host && rc-service nox-mesh-host start
# Arch
pacman -S nox-mesh-host
systemctl enable --now nox-mesh-host
```
Two lines each, and the init file behind them is four
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)) — it says
*run the launcher at boot* and nothing else, so a third system is transcription rather than a
port.
Or, where there is no repository to install from:
```
curl -fsSL https://<release>/mesh-host-<system>-<version>-x86_64.tar.gz | tar -xz -C /usr/local/bin
```
**The binary is per system as well as per architecture**, because two of its appliers are.
**The tarball must never acquire a dependency**, because the mesh's own package repository is
hosted on the mesh. Any route that needs the mesh in order to install the thing that joins the
mesh is a circle — unusable on a first node, and unusable by whoever is repairing a mesh that is
down, which is exactly when it is wanted.
### The unit it installs
```ini
[Unit]
Description=Novox Mesh node host
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/lib/nox-mesh-host/launch
Restart=always
RestartSec=5s
StateDirectory=mesh-host
[Install]
WantedBy=multi-user.target
```
**Two lines of policy, and that is deliberate**
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). The init is
asked to *start this at boot* and *start it again if it exits*, and nothing else. Both are
expressible in OpenRC, runit, s6 and an Android `init.rc`, so porting this file is transcription
rather than design.
**`Restart=always` and not `on-failure`**: the host restarts onto a new binary by exiting
*cleanly*, so a supervisor that only restarts on failure would leave every upgraded node stopped,
having successfully upgraded.
**What the init does not do is decide when to give up.** Counting failed starts and rolling back
lives in the launcher, where it can be tested — `OnFailure=` in a unit file can only be read and
hoped for, and it is the one thing that has to work on a machine where nothing else does.
**The package owns this file. The host never does.** It manages `service` resources, and its own
unit is a service — the temptation is obvious and it ends with a host stopping itself half way
through an apply, leaving a machine with nothing running to fix it. A declaration naming the
host's own unit is **refused**, and that refusal is a test rather than a convention.
The line to hold: **the installation owns the host; the host owns everything else.**
At this point the host is running and **doing nothing**. It has no identity, so there is nobody
to link to and nothing to apply. It answers `profile`, `inventory` and `version`, and waits.
---
## hosted → enrolled: the ordinary case
```
nox-mesh-host enrol --token <one-time token>
```
The token carries **four** things and is carried by a person
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)): the broker's
address, the fingerprint to expect, **the control plane's signing identity**, and the right to
join once.
**The fourth is the one this document listed three of.** A node connects to the broker and takes
instruction from the control plane behind it, and those are two different identities. Pinning only
the broker would make the control plane's authority *transitive* — a compromised broker could then
forge declarations, which, since the host applies whatever the link delivers, is the whole machine.
So the transport is verified once at connect, and **each declaration is verified by its signature,
every time**.
What happens, in order:
1. the host dials the broker at the address in the token, **over the underlay**;
2. it checks the broker's certificate against the pinned fingerprint — *before* sending anything;
3. it presents the one-time secret **and its own public key**, which the mesh records;
4. it reports its `profile` and `inventory` upward;
5. the control plane decides what this machine should be, and sends a declaration;
6. the host applies it, reads back, and reports.
**Step 4 is the one that is easy to miss and is what makes step 5 possible.** The control plane
cannot decide what a machine should run without knowing what it *can* run — a graphical session,
a container runtime, an architecture. The profile is not a diagnostic; it is the input.
**The node computes nothing about the mesh.** It needs one peer to reach; the whole overlay is
derived centrally and pushed down
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md),
[`08-connectivity.md`](08-connectivity.md)).
### The first declaration is the overlay, and nothing else
**The mesh makes a node reachable before it makes it useful.** Step 5 is not one declaration
carrying everything the node will ever run. It is two, in order:
```
first the overlay — this node's address, its keys, its peers, its names
then everything else — packages, containers, services, files
```
Three reasons, and the third is the one that matters when something goes wrong:
- **It is forced.** A node cannot join the overlay before contacting the mesh, because its
address and peer set are *assigned* — it generates a keypair, publishes the public half, and
receives the rest ([`08-connectivity.md`](08-connectivity.md)). So the overlay is the first
thing the mesh can give it, and it should be.
- **It is what [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) already
says:** *a joining node does the minimum to be reachable, and nothing else.*
- **It is the way back in.** Once the overlay is up, the node is reachable over it — by SSH, by
anything. If a later declaration breaks the machine, there is a route to it that does not
depend on the mesh's control path working. **Sending a large first declaration risks a node
that is broken and unreachable at the same time**, and those two failures are much worse
together than separately.
### Reachable is not the same as having a control surface
Worth stating plainly, because the two rules read as a contradiction and are not.
| | |
|---|---|
| **every node reaches every other node** | over the overlay — SSH, services, ordinary traffic. This is the point of having one |
| **every node consumes from the broker** | its own queue, over its own outbound connection ([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)) |
| **nothing dials a node to control it** | the host has no inbound control surface ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) |
**[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) is about the control
channel, not about network reachability.** What it forbids is a listening thing that accepts
instructions and changes the machine. A node being reachable on the overlay — the whole purpose
of the overlay — is untouched by it, and so is a person opening a shell on it.
The distinction is *who can tell this machine what to be*: only the control plane, only over the
link the node opened, only in declarations of known shape.
---
## hosted → enrolled: the first node
The same path, with the mesh built in the middle of it.
```
# 1 — raise the substrate and the control plane from the carried bundle
nox-mesh-host reconcile
# 2 — the control plane now exists, and issues the first token
mesh-control token issue
# 3 — the machine joins the mesh it just raised
nox-mesh-host enrol --token <token>
```
Step 1 is the bootstrap from [`07-the-substrate.md`](07-the-substrate.md): a container runtime,
then PostgreSQL, then the database, then the schema, then the control plane. It needs no identity
because nothing is being asked of anyone — the host is applying a declaration it already
carries, to the machine it is already on.
**After step 3 the first node is not special in any way**, which is the property `adopt.sh` and
the bootstrap script never had. Its specialness lasted two commands.
**And enrolment is exercised on node one.** The path every other node depends on is walked
immediately, against a control plane on the same machine, rather than being written and first
used months later on node two.
---
## Two kinds of host
Everything above assumes a machine with an init that runs the host at boot. Not every machine
has one ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)).
| | **resident** | **episodic** |
|---|---|---|
| examples | Alpine, Arch | Android |
| started by | an init, at boot | whatever the platform allows |
| supervised by | the launcher | nothing — the platform decides when it runs |
| the link | held open | opened while it runs |
| being stopped | shutdown, or a failure | **ordinary** |
| shapes | all six | `file`, `directory`, `action` |
| can be the first node | yes | **no** |
**An episodic host being killed is disconnection, not failure.** That is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) doing the work it was written
for: reachability is state, not class. Everything the design already does for a laptop that
closes — an authoritative local store, reconcile on start, *last heard from* reported without an
alarm — is what an episodic host needs, at a shorter period.
**It cannot be the first node**, and that is not a limitation to work around. Every step of
raising a substrate is a `package`, a `container` or an `action` against one, and a partial host
refuses the first two. So `mesh-host-android bundle` returns a file that says so rather than an
empty placeholder waiting to be filled in.
**Two things this changes for anything reading the mesh.** *Last heard from* is a much weaker
signal on an episodic host — a healthy phone looks like a dead server — so a reader has to know
which kind it is looking at. And a declaration may take a long time to land, which makes
[ADR 0010](../../02-DECISIONS/0010-delivery.md)'s separation of
*outstanding* from *failed* load-bearing rather than tidy.
**Still open:** how an episodic host is started in practice — an APK with a foreground service,
or Termux with its boot addon — and, first, **what an Android node is for.** A device that can
write files and run commands is not a workload host; it is a presence, or somewhere an agent
runs. Building the start mechanism before deciding that would be building it for nobody.
## Adoption: what happens to what is already there
Adoption is not a state. It is what the **first apply** does when it is told to own something a
machine already has ([research 012](../../01-RESEARCH/012-the-minimum-viable-node/00-overview.md)).
A candidate machine is not empty. It has a package manager, probably a container runtime,
configuration somebody chose. [ADR 0005](../../02-DECISIONS/0005-the-node-host.md)
says the host never touches what it did not create — adoption is the deliberate act of taking
ownership of exactly that, so it is a companion to that rule rather than an exception:
> *never, unless adoption made it the host's* — with adoption **explicit, recorded, and visible
> in what the host says it owns.**
Three rules, all earned:
**The original is kept before anything is written.** A one-way door on a working machine is not
an installation. This is a *never* rule, and it earns that from the worst loss in this record —
a tool acting on a path it did not own.
**On conflict, the machine's configuration wins.** Adoption always completes; the conflict is
flagged and reconciled afterwards. A machine in use keeps working exactly as it did.
**Adoption produces a briefing**, not just a result: what it found, what it took over, and what
it could not resolve — with each line marked `ok`, `kept`, `unknown` or `failed`, and the overall
outcome **derived** from the worst line rather than stated alongside it.
---
## enrolled: what running actually looks like
**Changes are pushed, not polled.** A declaration arrives as a message on the link and the host
applies it then. The link is already open and outbound
([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md),
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) — asking it
repeatedly whether anything has changed would be slower to land *and* constant traffic to learn
nothing.
| Trigger | Kind | |
|---|---|---|
| **a declaration arrives** | **pushed** | the ordinary path — this is how a change lands |
| **start** | event | the machine may have changed while nothing was running |
| **reconnect** | event | declarations may have been missed |
| **every ten minutes** | periodic | **drift, and only drift** |
**Why the timer cannot be an event.** Drift is change the *mesh did not make* — somebody edited
a managed file, a distribution upgrade replaced a config, a container was stopped by hand.
Nothing will ever publish a message about it, because whatever did it is not part of the mesh.
Only looking finds it.
So the two periodic things do different jobs and should not be conflated:
| | direction | answers |
|---|---|---|
| **reconcile timer** | local, looks at the machine | *does this machine still match what it was told?* |
| **heartbeat** | upward, reports to the mesh | *is this node still here, and what is it running?* |
**The heartbeat is what makes silence mean something.** A node with nothing to do sends nothing;
without a heartbeat that is indistinguishable from a node that stopped. With one, *last heard
from* is a fact beside every node — which is what
[how long disconnected](#how-long-disconnected-and-who-is-told) reports and what
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md) exists
because a stuck node cannot send.
**Rebooting mid-apply is safe by construction.** The store records each resource *after* it
worked ([ADR 0018](../../02-DECISIONS/0018-a-picture-is-read-from-what-runs.md)), so a host that
dies half way through comes back, finds the completed ones already matching, and applies the
rest. The rule that exists to stop the host lying about what it did also makes it crash-safe.
## Updating what the node holds
An ordinary declaration. Someone assigns a module; the control plane recomputes what that node
should be and sends it; the host applies the difference and removes what is no longer declared.
**Removal is not symmetric, and the asymmetry is the design:**
| | on being undeclared |
|---|---|
| file, directory | **removed** |
| container | **removed** — the host created it |
| service | **stopped**; the unit file is not the host's to delete |
| package | **left installed** — *forgotten*, not removed |
| action | **forgotten** — it left nothing the host owns |
The host removes what it *made* and leaves what it merely *configured*. Uninstalling a container
runtime because a declaration changed would stop every container on the node.
---
## enrolled ⇄ disconnected
Not a failure. Not degraded. A situation
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
A disconnected node **keeps reconciling against its own store**, so it goes on holding its
machine in the last state it was told to hold. A laptop shut for a week comes back and
reconciles; it does not come back and ask what it is.
What it cannot do: receive new declarations, be granted anything new, or have its certificates
renewed — which is the clock on the whole arrangement
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)).
**How long it has been disconnected is a fact the mesh must hold**, and nothing holds it today.
Without it, a node running last month's assignments looks exactly like one that is current.
---
## Rescue
The host is still a command-line tool, and that is what rescue is:
```
nox-mesh-host owned # what do you think you own?
nox-mesh-host apply repair.json # apply something by hand, locally
nox-mesh-host profile # what can this machine actually do?
```
`apply FILE` accepts actions, because someone who can write that file and run this binary as
root can already do anything it can. The bound in
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md) is on what a
**remote** party may push, not on what a person at the machine may do.
This replaces the three hand-run scripts that exist today — first node, joining, rescue — with
one binary that has always been the same binary.
---
## enrolled → hosted: retiring a node
Two cases, and they are genuinely different.
**Graceful.** The control plane sends a final declaration that names nothing. The host removes
what it owns by the table above, reports, and drops its identity. The machine keeps the host
installed and is back to `hosted`. Nothing is left behind that anybody has to remember.
**The node is gone.** Stolen, dead, or simply unreachable. The mesh cannot tell it anything, and
by [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) it will go on reconciling
its last declaration **forever**.
That is the honest consequence of making disconnection ordinary, and the answer is not to make
the host expire. It is that **the node holds nothing that outlives revocation**: its identity is
its own, and every grant it holds is a per-node credential at the provider
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md),
[ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md)). Revoking is done at the
database, the broker, the object store — not on the machine.
So a lost node keeps *running* and stops being able to *reach* anything. That is the best
available outcome and it is worth stating plainly rather than implying the mesh can reach out and
switch a machine off, which it cannot and should not be able to.
---
## Losing the store
Worth its own section because the failure is quiet.
If `/var/lib/mesh-host/state.json` is lost — a reinstall, a replaced disk — the host loses
**its record of what it owns**, not its ability to work. It re-enrols, receives the declaration
again, and re-applies it.
**Without help, what does not come back is removal.** Resources applied under an older
declaration, whose record is gone, become unowned: the host will not touch them, because it
never touches what it did not create. They would sit there, unmanaged, indefinitely.
**So the mesh keeps a copy of what each node reports it owns**, refreshed on every apply report,
and hands it back on a rebuild — see [Protecting the store](#protecting-the-store). The store
remains locally authoritative for *operating*; the copy exists only for this.
---
## Upgrading the host
The host is delivered like anything else
([ADR 0010](../../02-DECISIONS/0010-delivery.md)), and this is worth
walking through because tier 0 looks like it should be special and is not.
```
push to mesh-host
│
├─ build go build → one static binary
├─ publish packaged, into the mesh's own package repository
└─ deploy each node's declaration now names the new version
│
└─ pushed to each node; the host applies it on arrival
(a node that is offline gets it on reconnect)
```
**Compared with today.** The current pipeline's third silo runs *once per node* and sends each
one a command to install and start. That is where the as-is records a package install that
404ed while the job went green. Here deploy is **one write** — the declaration changes — and the
installing is the host's ordinary work, which reads back before it records anything.
**The repository is reachable because a declaration made it so.** A `file` resource writes the
package manager's configuration pointing at the mesh's repository; a `package` resource names
the version. Both ordinary shapes, applied by the same host. **No new resource type**, which is
the test of whether this is really uniform.
### The restart
```
1 pacman installs the new binary the running process is untouched —
Unix keeps the running executable's inode
2 the host verifies the new binary runs `nox-mesh-host version`, as a subprocess
3 it finishes the apply and reports never mid-way
4 it exits 0 having finished, not having been stopped
5 the launcher starts it again on the new binary — it supervises the host
rather than exec'ing it (ADR 0005), so this
needs nothing from the init
6 the new host reconciles on start trigger 1, confirming the machine still matches
```
**Step 2 is the one to insist on.** A package can install a binary that does not execute here —
wrong architecture, a libc that is not present. Running it once before committing to a restart
turns "the node never came back" into "the apply failed and said why". It is the same read-back
rule the rest of the host already follows, applied to the one resource that is the host.
**The host never asks the service manager to restart it.** That is the host stopping itself
part-way through an apply. It stops by finishing.
**A fleet upgrades over an interval, not at an instant**, because each node restarts when its
own apply completes. A node must therefore report the version it is **running**, not the one
installed — otherwise the mesh believes an upgrade landed at step 1.
**A version that crashes on start rolls itself back**
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)).
What the init starts is not the host but a **launcher**, and the launcher is where the policy
lives:
```
init ──► nox-mesh-host-launch ──► nox-mesh-host
├─ halted? say so and stop; a person has to look
├─ count this start attempt
├─ too many, not yet rolled back? roll back, then start
├─ too many, already rolled back? halt — the machine is the problem
└─ otherwise start the host
```
It reinstalls the version recorded in `known-good`, which the host wrote the last time it
completed a reconcile — and the host clears the attempt counter at the same moment, for the same
reason.
**The launcher rather than the init's own features**, because this is the one thing that must
work on a machine where nothing else does. A shell script with a counter can be run against a
stub package manager and asserted; `OnFailure=` in a unit file can only be read and hoped for.
It also means the init is asked for nothing but *start* and *restart*, which every init can
do.
**It rolls back once.** If the previous version also fails, the node stops in a failed state
rather than flapping between two binaries. A second failure is a different diagnosis: the
machine is the problem, not the binary.
**Why this matters more than it looks.** A host that will not start cannot link, and a node that
is not linking looks exactly like a machine somebody switched off — which is the one condition
this design has deliberately decided not to alarm on. Without rollback, a bad release reaches
every node, each one goes quiet, and the mesh reports a fleet of sleeping laptops.
---
## Details that are easy to get wrong
Each of these has a wrong answer that looks reasonable, which is why they are written down
rather than left to be worked out.
### Re-enrolling as the same node
**A token is issued *for* a node record**, and that is where a re-enrolment is decided.
```
mesh-control token issue --node workstation # this machine is that node again
mesh-control token issue --new # a machine the mesh has not seen
```
The host does not need to know which it is. It presents a token and receives an identity; what
that identity is bound to was decided when the token was made.
**Issuing a re-enrolment token revokes the previous identity for that node**, and that is not
housekeeping. Two live identities for one node record is the stolen-laptop case with the thief's
credentials still valid — the case
[`retiring a node`](#enrolled--hosted-retiring-a-node) says is answered by revocation.
### Protecting the store
**The host reports what it owns, and the mesh keeps the last report.**
The store stays locally authoritative — a node operates from its own copy and needs nothing to
do so ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). What changes is that
the mesh holds a **copy for recovery**, refreshed on every apply report.
So a node that loses its state file re-enrols, receives both the declaration *and* the record of
what it previously owned, and can then remove what is no longer declared. The orphans that used
to be permanently stranded are recoverable.
**This is a backup, never a source.** The host never reads it to decide anything; it is handed
back only on a store rebuild, and a node that disagrees with it wins, because the node is the
one that can see the machine.
### How long disconnected, and who is told
**The mesh records last contact per node; the node records time since it last linked.** Both,
because they answer different questions — the mesh's is *have I heard from it*, the node's is
*how stale am I*, and a node reporting the second on reconnect is how a long absence gets
noticed at all.
**No threshold and no alarm.** A laptop switched off for three weeks is doing nothing wrong, and
a mesh that alerted on it would train people to ignore the alert. It is a **reported fact** —
`last seen 4 days ago` beside every node — and what counts as too long is a judgement for
whoever is looking, not a constant in the design.
### Whether a failed adoption line blocks
**Adoption always completes. A node with a `failed` line is a node, and it is not eligible for
assignment until the failure is resolved.**
*Flags inform, they do not block* holds for **conflicts** — where the mesh chose deliberately and
the machine still works. A **failure** is different in kind: not *we chose* but *we could not*,
and it gets different treatment for that reason.
The distinction is between **joining** and **being given work**. Refusing to join makes a
machine in use unadoptable, which is the outcome that rule exists to prevent. Placing work on a
machine where something the mesh needed never happened produces a module that is installed and
does not work — [04-ISSUES/007](../../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md)
arriving from the adoption side.
### What a briefing is
**A structured document with prose in it**, held in the node's state and reported to the mesh.
It is the first thing a session on a new node reads, which makes it an interface.
```
outcome kept derived from the worst line below, never stated separately
node workstation
adopted 2026-08-27T14:02Z
ok container runtime docker 27.0, adopted; original config kept at <path>
kept storage driver machine has overlay2, the mesh wanted btrfs — machine wins
unknown firewall ruleset could not be parsed
failed package database locked by another process
what to look at
The storage driver disagreement is preference, not requirement, so nothing is broken.
The package database was locked; nothing was installed. Re-run adoption when it is free.
```
**The outcome is computed from the lines**, so a briefing cannot read *fine* while carrying a
failed line. Two independently written fields drift, and that drift is the fault this repository
keeps cataloguing.
### Where the enrolment token comes from
**`mesh-control token issue` prints it once**, to the person running it. Single-use, and it
expires whether used or not ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)).
It is carried by hand — read off a screen, pasted into a terminal. That is the design rather than
a gap in it: its authenticity comes from the channel it travelled, which is what
lets a node verify a mesh it has never spoken to
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). A token emailed,
committed, or dropped in shared storage has lost the only property that makes it worth carrying.
**On the first node it comes from the control plane that was raised two commands ago**, which is
the same command against a mesh that is one machine old.
---
## Still open
- ~~**Automatic rollback of a bad host version.**~~ **Resolved** by
[ADR 0005](../../02-DECISIONS/0005-the-node-host.md): a launcher
counts failed starts and rolls back — shipped by the package, not the host binary, because a
binary that will not start cannot recover itself. It rolls back once; a second failure means
the machine is the problem, not the binary.
- **How a previous declaration is retained and chosen**, which is what rollback of anything else
would use ([ADR 0010](../../02-DECISIONS/0010-delivery.md)).
- **A node returning after months** applies a very large jump in one go. Correct, and untested.
## Going away and coming back
*2026-08-31, from being asked whether a machine that drops off has to be adopted again.*
**It does not, and nothing about it expires.** A disconnected node is the same node in a different
situation ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) — reachability is state,
not class. The node holds its own identity and the mesh holds the public half; there is no lease,
no timeout, and nothing that lapses while a machine is shut. A laptop closed for a week comes back
and reconnects, and a declaration sent while it was away is waiting for it.
**The only thing that forces re-enrolment is a machine losing its own key** — a reinstall, an image
re-cloned. That is deliberate: the mesh then reports what it can no longer seal to rather than
delivering blobs the machine cannot open.
### The gap that was left, and why it mattered
**A machine that suspends does not know it has been disconnected, and neither does anything else.**
After a resume the socket looks perfectly healthy from inside the process: no error, no close,
because nothing has tried to send anything yet. Heartbeats discover it twenty or thirty seconds
later.
For those twenty or thirty seconds the node believes it is in the mesh and is not — and *absence
must never be indistinguishable from a failure to answer* is the rule this whole design is built
on. It recovered on its own, which is why this was a quality gap rather than a fault. It was still
the machine waiting to be told something it already knew.
**So the machine says so.** Waking, and changing network, both rouse the host.
| | |
|---|---|
| **it ends the current attempt** | not the wait after it — the process is not waiting, it is sitting inside a connection that will never return |
| **by signal, not by anything listening** | a socket for this would be a control surface on every machine, in exchange for saving twenty seconds, and the security argument rests on there not being one |
| **two rouses at once are one** | a machine suspending and resuming repeatedly must not build a backlog of reconnections to work through |
| **the backoff is not reset** | being roused says the machine changed, not that whatever refused the connection has stopped. A laptop woken on a network with no route would otherwise retry at full speed for as long as somebody keeps opening the lid |
| **`down` does not rouse** | the link is already gone, reconnecting will fail, and the backoff exists for exactly that |
*Checked by holding a link that never returns on its own — which is precisely what a suspended
connection is — rousing it, and requiring the attempt to end and another to begin. And by running
the dispatcher against every event a network manager emits, requiring it to act on the ones that
change where packets go and on no others.*
03-DESIGN/01-to-be/10-delivery.md
+217
View File
@@ -0,0 +1,217 @@
---
layer: to-be
status: in-progress
code:
- mesh-control internal/builder
- mesh-control cmd/mesh-control (build, build --behind, push, status)
- mesh-control internal/inventory/builds.go
updated: 2026-08-31
decisions:
- 02-DECISIONS/0010-delivery.md
- 02-DECISIONS/0009-modules-and-the-graph.md
- 02-DECISIONS/0009-modules-and-the-graph.md
- 02-DECISIONS/0010-delivery.md
- 02-DECISIONS/0010-delivery.md
- 02-DECISIONS/0009-modules-and-the-graph.md
- 02-DECISIONS/0009-modules-and-the-graph.md
---
# Modules and delivery
How a change somebody makes becomes a thing running on machines.
This is the whole of it, current, in one place. Where a decision record is cited it is for the
reasoning behind a choice, not because the answer is somewhere else.
## A module
The unit of delivery: assignable to a node, versionable, replaceable on its own.
**Not a grouping.** There is no `networking` module containing four things — there are four
modules, named individually, with edges between them. Folders assert relationships; edges record
them, and only edges can be queried or kept true automatically.
**When several modules always change together**, that means they share an *authority* — one place
that decides for all of them. It does not mean they should be one artifact. Connectivity is the
worked example: one context decides the overlay, names, routes, filtering and certificates, and
`wireguard`, the resolver, the proxy and the firewall remain four modules, because they are
deployed to different sets of nodes.
> **Coherence is a context. Delivery is a module.**
## The three edges
A module's relationships to other modules. Two are declared; one is read from the code.
| edge | means | declared? | satisfied |
|---|---|---|---|
| **presence** | that thing must exist and be reachable here | yes, in the manifest | at provisioning |
| **instantiation** | that thing makes something for me and hands back credentials — a database, a bucket, a route | yes, in the manifest | at provisioning, and again whenever it must be |
| **build** | I was compiled against that artifact | **no — derived from imports** | **at build, once** |
**Why the build edge is derived and the others are not.** A runtime edge is an *intention*
somebody has about how the mesh should be wired, and only a person can state it. A build edge is
a *fact about code that already exists* — and a declared list of dependencies drifts from the
imports it describes, so the imports are what is read.
**Why the build edge is a different kind rather than a variant.** It is fixed inside an artifact
rather than negotiated when something runs, and its only remedy is a rebuild. Nothing can
re-provision it.
## The core library
One module everything is allowed to depend on, holding **the mesh's own domain**: a module, a
node, an assignment. Those three are what every context talks about and none of them owns.
The test for whether something belongs: *would this still mean the same thing in a context that
had never heard of the one it came from?* A node would. A pipeline stage would not — that is
delivery's. A grant would not — that is provisioning's.
**Types ship with the module that owns them**, not here. A consumer needing `inventory`'s types
depends on `inventory` — one narrow, visible edge — rather than everything depending on a hub
where the relationship cannot be seen. A library everything depends on is expensive to change
whether it holds types or code; what makes it expensive is the fan-in.
**This stays small on its own**, which is the point of choosing a domain rather than a drawer. A
domain model changes when what the mesh *is* changes, which is rare. *Shared code* changes
whenever anybody writes something reusable, which is constantly.
## Delivery is a comparison, not a pipeline
The control plane holds two facts and builds the difference:
```
what source exists ─┐
├─► differ? ─► build ─► judge ─► declare ─► nodes converge
what has been built from it ─┘
```
**A change becomes a build because source is ahead of artifacts.** Not because a message arrived.
An event makes it fast; nothing makes it necessary — so a missed webhook costs latency and cannot
cost correctness.
That is the same shape the host uses on a machine, one layer up:
| | reconciles | against |
|---|---|---|
| the control plane | artifacts | source |
| the host | machine state | declarations |
**There is no pipeline as a state machine.** No stage list something can be omitted from, and no
run to lose.
### An artifact is current, or it is not
> An artifact is out of date when **its source moved, or anything it was built against moved**.
So what is recorded against an artifact is a commit **and the identity of every artifact it was
built against** — its input closure. That is what makes *is this current?* answerable without
building anything, and what makes the rebuild set computable: take the changed module, follow
inbound build edges transitively, and that is what is stale. In order, because the edges are
directed.
**A shared change is a cascade, and that is inherent.** One change to the core library
invalidates nearly everything. The ordering comes from the graph, not from a hand-written list of
levels.
### The verdict
An artifact may not be declared until something has judged it fit. Two tiers, because one gate
would be both slow and unreliable:
| | judged by | when |
|---|---|---|
| **the module's own tests** | the build | **always** — this is most of it |
| **the lab** | a raised scenario | when an assertion genuinely needs a mesh |
**A run that failed for environmental reasons is not a verdict.** A machine that would not boot
says nothing about the artifact, and recording it as *unfit* is the same untruth as recording a
dispatch as a deploy. *Outstanding* and *failed* are different results.
### Declaring, and converging
Deploy is **one write**: the affected nodes' declarations now name the new artifact. It is not
once per node, and nothing is pushed to a machine.
Each host applies what it is told, reads back, and reports. A node that is switched off does it
when it wakes.
**What a delivery result means:**
```
meshboard source X · built from X · fit · declared on 5 · applied on 3, 2 outstanding
```
Not *the job went green*. **Outstanding is not failure** — a node that has not applied yet is a
fact with a timestamp, and it resolves itself when the node comes back.
## What this is designed against
Every property above answers something that has actually gone wrong, recorded in
[`00-as-is/04`](../00-as-is/04-delivery.md):
| what happened | what prevents it |
|---|---|
| a merge created no pipeline, and nothing said so | a change is found by comparison, not by an event |
| a package install 404'd from every mirror while the job went green | the applier is the reporter, and it reads back |
| a verify stage was built and never scheduled | verification is not a stage that can be left off a list |
| a service was reported started when the command merely returned | *green proves transport, not effect* — so nothing reports transport |
| the build node parked forever while every other node deployed | there is no fan-out to be asymmetric about |
## What must exist before this can be built
Not aspirations — things without which the above does not work:
1. **The module graph, with build edges.** No graph, no rebuild set and no ordering.
2. **A recorded input closure per artifact**, so currency is answerable without building.
3. **Something that notices a reconciler is not converging.** Below.
## Open
- **Does a fit artifact declare itself?** Nothing above says who moves the declaration. If it is
automatic, merging to main deploys to production — which may be wanted, and is far too large a
property to acquire by omission.
- **A reconciler that cannot reach its target retries forever.** A failed job stops and names its
step; a loop is silent. Without something that notices *this has been trying for an hour*, this
design reintroduces the fault it removes. **The largest open risk here.**
- **Reproducible builds.** If rebuilding unchanged source against unchanged inputs produced the
same digest, a cascade would stop at the first module whose output did not move. Without them,
one core-library commit redeploys the fleet with no behavioural change.
- **How a module publishes its own types**, which differs per language.
- **How the control plane upgrades itself.** It declares its own new version and the host applies
it — but if the new one is broken, the thing that would fix it is the thing that is broken. The
host has a launcher for exactly this; the control plane has nothing.
## What "behind" means, and what it used to mean
*2026-08-31.*
The risk this record names is losing **did my change go out?** — answerable today by opening a
pipeline, and something has to replace it or the comparison is worse to live with whatever its
other properties.
**It was answerable only for the machines that broke.** `push --behind` meant *failed or refused*,
so a machine that applied cleanly and whose declaration has since changed was not behind. For every
machine that worked, the answer was silence — and silence meant both *your change is running there*
and *your change has not been sent*, which is the question unanswered rather than answered.
**So the mesh records a digest of what it last sent each machine.** A digest rather than the
declaration: what a machine should be is recomputable at any moment, and a stored copy would be a
second account of it, able to disagree with the first. What cannot be recomputed is what was
*actually sent*.
**Recorded after the send.** A digest kept for something that failed to send would make the machine
look current for a declaration it never received — the failure mode this is meant to remove,
arrived at from the other side.
**Three situations, kept apart**, because they read differently to whoever is looking even where
the remedy is the same push:
| | |
|---|---|
| **out of date** | it was sent something, and the mesh would now send something else |
| **never told** | nobody has ever asked this machine to be anything |
| **not worked out** | the mesh cannot say what it should be — not reported here at all, because saying "waiting" about it would invent a comparison. `plan` is where that is answered |
**`status` says it and `push --behind` acts on it**, and both because the alternative is a flag that
knows something the person reading the status does not.
03-DESIGN/01-to-be/11-a-board.md
+142
View File
@@ -0,0 +1,142 @@
---
layer: to-be
status: in-progress
code:
- mesh-control cmd/mesh-control/board.go
- mesh-control cmd/mesh-control/readable.go
updated: 2026-08-31
decisions:
- 02-DECISIONS/0008-a-context-owns-its-store.md
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
---
# A board
**A place to see the mesh.** Read from a survey of the one that exists, so what is proposed here
is a shorter list than what is there, deliberately.
## What the existing board does, and what of it belongs here
Eight sections. Four are about work and workers and are held back with the rest of that domain;
the other four are about the mesh itself.
| | what it shows | where it stands here |
|---|---|---|
| **the mesh** | every node, what each runs, what each takes from another, module versions | **everything behind it exists** — it is a reader, not a second source |
| **what is wrong** | machines not doing what they were told, machines not answering | **the page nobody had thought to ask for**, and the one a person opens first |
| **builds** | a build, its stages, its log | **everything behind it exists** — every result is kept, failures included |
| **sessions** | model sessions, their usage, and switching between accounts | [ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md) |
| **channels** | nodes messaging each other | the agent layer |
## The constraint that matters, and it is not a feature
**The existing board is one service that reads every context's database.** It joins nodes to
provisions to modules to sessions by querying each store directly, because that is the shortest
path to a page that shows all of them at once.
That is [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md) violated by the one
component with a reason to violate it, and the cost is not hypothetical — it is the same cost the
shared library has: **a boundary nothing may cross is a boundary that can move; one thing crossing
it is enough to freeze it.** A board that reads the provisioning tables directly is a board that
breaks when provisioning changes its tables, and the change then gets weighed against the board.
**So a board reads through interfaces and holds nothing.** Everything on the mesh page above is
already answerable by asking the control plane — what nodes exist, what each resolves to, what it
takes from elsewhere, which module came from which commit. A board that asks those questions is a
client. A board that queries `inventory` is a second control plane with a worse contract.
**It stores nothing of its own.** No cache that can disagree, no table of "what the mesh looked
like last time". If a question is slow to answer, the answer belongs in the context that owns it,
where everything else asking gets it too.
## Where it is reachable from, which decides everything else about it
*Written 2026-08-31. It was assumed throughout and stated nowhere, which is the wrong way round
for the most consequential fact about this component.*
**The board is published on a public name.** Not reachable only over the private network — on the
internet, behind the reverse proxy, like any other published workload.
**So its login is a perimeter, not defence in depth.** A board on the overlay alone would sit
inside the boundary that [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) already
calls the security boundary, and a login there would guard a room whose door is inside the
building. This one faces everybody.
**Which makes the identity provider the mesh's outermost gate.** The board is a presentation layer
over the control plane and the control plane's networked surfaces can change the mesh
([ADR 0035](../../02-DECISIONS/0035-one-implementation-several-surfaces.md)), so **whoever that
provider admits can assign modules, from anywhere.** Said flatly because it is easy to arrive at
one reasonable step at a time and then be surprised by.
Three things follow, and none of them are the board's own design:
- **Who may log in, and how, is a decision about the mesh** rather than about an application. A
realm that lets somebody in has let them into the mesh.
- **A public name needs a public certificate**, from an authority the world trusts rather than the
mesh's own — which is why that exists at all.
- **The provider failing is not only "the board is down".** Its configuration going wrong in the
other direction — a realm that admits too much — is a mesh-wide exposure with no local symptom.
**The command line is unaffected and is the reason this is tolerable.** It authenticates through
nothing, needs no network, and answers to the machine's own login — so the mesh remains operable
by somebody standing at it whatever happens to the gate. *That is the property to protect if the
rest of this is ever traded away.*
## What it is not
- **Not the way to change things.** Reading is the whole of it to begin with. Every action the
board could offer already exists as a command, and a button that does something no command does
is a second implementation of a decision.
- **Not a dashboard of graphs.** What a person needs from a mesh is *which machine is not doing
what it was told*, and that is a list, not a chart.
## The three questions, in order
Written down because the order is the design. A person opens this when something is wrong, and a
page that led with the third would bury the first:
1. **Is anything broken?** A machine whose last declaration was refused or partly failed. It has
consequences now.
2. **Is anything not answering?** A machine not heard from. It may be new, switched off or
unreachable — **which is not the same as tried and could not**, and collapsing the two sends
somebody to debug a machine that was never sent anything.
3. **Is anything out of date?** A module behind its source, and the machines running the old one.
A plan for later rather than a problem now.
**Refused and failed stay distinct all the way to the page.** Refused means the machine is exactly
as it was and what is wrong is in what was sent; failed means it is in a state nobody declared and
what is wrong is on the machine. They are fixed in different places, so a page that said "error"
for both would send half its readers to the wrong one.
## What was built
*2026-08-31.*
**One reading, three ways of saying it.** The questions are asked once, by one function, and
answered as a person's `status`, as its JSON, and as this page. Three implementations of *which
machine is not doing what it was told* would be three chances to disagree about it — and the
disagreement would surface as two people looking at two screens arguing about which machine is
broken.
**It holds nothing and changes nothing.** Every request reads the mesh now. There is no cache to
go stale, no table of what the mesh looked like last time, and no button: every action a board
could offer already exists as a command, and one that did something no command does would be a
second implementation of a decision.
**It never touches a context's store.** That is the whole constraint above, kept: the board is a
client of the same functions the commands use, so provisioning can change its tables without the
change being weighed against a page.
**A board that cannot read the mesh says so.** An empty page says *nothing is wrong* in the one
situation where nobody can know that, so the failure is rendered instead — and it says explicitly
that it is a statement about the page rather than about the mesh.
**A machine's own words are shown, and are not markup.** They are the whole reason the page is
useful — a board that said only *failed* would send a person to ask the thing they opened the
board to avoid asking. They are also the only text on the page that nobody in this repository
wrote, which is why the escaping is a test rather than an assumption.
*Checked by giving a machine a declaration it cannot apply and requiring the page to name that
machine, say `failed` rather than `error`, and quote what the host said — then by comparing the
page's own JSON against the command's, because two answers to "which machine is broken" would be
worse than either alone.*
03-DESIGN/01-to-be/12-a-module-repository.md
+347
View File
@@ -0,0 +1,347 @@
---
layer: to-be
status: in-progress
code:
- mesh-control internal/builder
- mesh-control internal/catalogue/build.go
- mesh-control internal/inventory/secrets.go
- mesh-control cmd/mesh-builder
updated: 2026-08-31
decisions:
- 02-DECISIONS/0009-modules-and-the-graph.md
- 02-DECISIONS/0010-delivery.md
- 02-DECISIONS/0005-the-node-host.md
---
# A module repository, and what builds it
**Designed from what the mesh needs, not from what came before.** The system this replaces has a
concept of *features* — several independently-deployable units inside one module — and it is
deliberately absent here.
## Features are unnecessary, and that closes an open prerequisite
[ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) lists *named features
with per-node opt-in* as a prerequisite, on the grounds that without it "every independently
deployable unit inside a context becomes a module again and the count returns."
**The premise was right and the remedy already exists in another form.** What features were for is
three things the mesh now does separately:
| features did | what does it here |
|---|---|
| several deployable units in one thing | **several modules**, which is what they are |
| turning one on for one node | **assignment**, which is per node already |
| keeping related things together | **`requires`**, and a module with requirements and no files of its own |
`networking` is exactly that last row: it ships nothing, requires a private network and name
resolution, and assigning it brings both. So the module count does not return, because the thing
that made it return — *a module is expensive, so put several things in one* — is gone. A module
here is cheap: a manifest and, usually, nothing else.
## One file at the root
`module.json`, and a convention somebody can look for beats a setting somebody has to find. It
says what the module is, what it provides and requires, what it claims, what capabilities it
needs, what it puts on a machine — and, if anything must be produced from the source, what to
build.
## The manifest in the repository is not the manifest the mesh holds
A resource names an artifact:
```
{"id": "dotfiles", "type": "archive", "artifact": "config", "path": "…"}
```
and the built manifest names the thing:
```
{"id": "dotfiles", "type": "archive", "source": "…/blobs/sha256:…", "digest": "sha256:…"}
```
**Two documents on purpose.** A digest is not knowable until something is built, so a repository
carrying one is a repository whose file is wrong the moment anybody edits anything — and the mesh
would be pinning a value nobody could have checked. The built manifest is derived, and the record
of *which commit it was derived from* is what makes "is this current?" answerable without building
it again.
The word `artifact` never reaches a machine. The host's decoder is strict and would refuse it, at
the worst possible moment.
## The builder runs on a node
**Not in the control plane, and this is the same boundary as everywhere else.** Building needs a
container runtime and a working tree; what the control plane may send a machine is bounded by the
declaration language ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and *run this build*
is not in it. The alternative — the control plane holding a container socket — would make it the
one component that can do anything on any machine, which is the property the whole design is
arranged to avoid.
So the builder is a program a machine runs, given work over the broker like anything else, holding
its own credential and nothing more.
**A build is work, not state**, and that is why it does not travel as a declaration. Everything
else the control plane sends a node is *what you should be*, reconciled forever. A build happens
once and is finished; as a declaration it would either rebuild on every reconcile or carry "and I
already did this" — state about an event rather than about a machine.
So it has its own queue, and the answer comes back correlated. **One queue**, so several build
machines share the work and each request is done exactly once, which a routing key per machine
would not give.
**A build machine has its own credential**, and it is not a node's. It may read the build queue
and write to the mesh exchange, and that is all — a node's queue carries that node's declarations,
and a build machine has no business reading them.
**The answer goes through the exchange, never the default one.** Permission on the default
exchange is granted per *exchange*, not per queue, so anything allowed to use it can publish into
any node's queue. That is the privilege a build machine most obviously should not have. So an
asker binds its own reply queue to the same routing key and filters by correlation; every asker
sees every result, which is the price of the builder never needing that permission.
Three properties of the builder that are decisions:
- **a request is acknowledged only once the answer is away.** A builder that dies mid-build then
leaves the work for another machine rather than losing it with nobody ever hearing why
- **one build at a time.** Five at once against one runtime finishes all five slower than it would
have finished the first, and the queue is what shares work between machines
- **a failure is a result.** A build that fails silently is indistinguishable from a builder that
is not running, and those want completely different responses — the same rule the host follows
about a service that does not exist
### And it is a module the mesh assigns
*2026-08-31. Written after `builder issue --node`, which is the part that makes the sentence
"holding its own credential" true rather than aspirational.*
A build machine is a machine that runs the builder, and there is exactly one honest way to say
which machines those are: **assign it**. So the builder is a module like any other — an image, a
container, a working directory, and a claim so a machine does not end up running two.
The one thing that could not be a module in the ordinary way is the credential. It is not
generated, because the broker has to have been told about it, and it is not written in a manifest,
because a manifest is public and the same file goes to every machine that ever runs it. So the
mesh **creates the account, seals the URL to the machine that will use it, and discards the
plaintext** — the "given, not generated" case above, and its first user.
Nothing is printed. A credential shown on a terminal is a credential in a scrollback buffer, and
the copy that matters would then exist in two places, one of which nobody is guarding.
**What this replaces:** a builder started by hand with whatever credential was to hand, which in
practice meant the broker's administrative account. *A program documented as holding its own
credential and given somebody else's is worse than one with no story at all* — the documentation
is what stops anybody checking.
*Checked in the lab by assigning it and then asking the mesh to build a module: the credential
file arrives readable only by that machine, names the scoped account rather than the broker's own,
and the build completes — which is the only proof the credential authenticates, because a
container that is up holding a credential it cannot use looks identical from outside.*
**And it is told what to check the broker against**, not only who to connect as. A mesh's broker
presents a certificate of the mesh's own, which is in no public trust store, so a URL alone reaches
only a broker somebody else vouches for — which is no mesh broker at all. The credential carries
the fingerprint beside the URL: **the same two facts a node's token carries**
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)), for the same reason, arriving by
a path other than the thing being trusted.
**A builder that is a module cannot see the machine's filesystem.** It runs in a container, so a
local path exists for the machine and not for it. That is not a limitation to work around — it is
the arrangement working: a build machine shares the runtime it was given rather than the machine
it sits on. **A module is cloned from the forge over a URL**, and "build this directory" is a
convenience for a builder somebody started by hand.
### And the loop is closed
*2026-08-31.* [ADR 0010](../../02-DECISIONS/0010-delivery.md) replaced a pipeline with a comparison
and named the risk: **losing the question "did my change go out?"**. The mesh could already answer
which modules were behind their source — and then a person read that list and retyped each
repository, which is a person being the loop, and the loop is the thing the pipeline was doing
before it was taken away.
`build --behind` is the other half, and it is the mirror of `push --behind`: the mesh knows what is
stale, so it builds it. The two forms are deliberately not combined — naming a repository and
asking which need building are different requests, and guessing which was meant would sometimes
build something nobody named.
**One failing does not stop the others**, for the same reason one broken module no longer blocks a
machine's whole declaration: a mesh where one bad repository holds back nine good ones is a mesh
where nobody dares add the tenth.
**Each is built from its own recorded ref**, not from the commit the mesh happened to notice.
Pinning to that would quietly turn a tracked branch into a pin — a change of meaning nobody asked
for, arrived at by an implementation detail.
**Building is not delivering, and the two stay separate.** A machine keeps running what it has
until it is told otherwise; the mesh changing its mind is not a machine acting on it, and
collapsing the two is how a mesh comes to report success for something that has not happened.
*Checked end to end: a commit, a build, a catalogue entry, and a machine that ends up running what
the source says — with both halves that make the answer trustworthy. It is still running the old
one until it is pushed, and it stops being reported as behind once it has caught up, because a
status that says "behind" for ever is one nobody reads.*
## What is kept
**Every result, including the failures.** A failed build that leaves no trace is indistinguishable
from one nobody asked for, and the difference is the whole of whether somebody should be looking
at something. A build that failed before it knew what it was building keeps the repository, which
is what a person goes and looks at.
Recording is idempotent on the correlation, because a result arrives twice — once as the answer to
whoever asked and once on the exchange, where the control plane is also listening. Two rows would
show one build as two, and which is real is not answerable afterwards.
That is what a builds view reads, and until it existed there was nothing to read: a result was
answered to the asker and kept nowhere.
## Three properties that are decisions
- **A fresh clone every time.** A build reusing a working tree can succeed because of something a
previous build left behind, and that is a build nobody can reproduce.
- **Archives are packed deterministically** — sorted, and carrying no timestamps, ownership or
original paths. Two builds of one commit must produce one digest, or nothing downstream can tell
*this changed* from *this was built again*, and every rebuild looks like a change to every
machine holding it.
- **Nothing is published until everything is built.** Half a module in the store, under a digest
the mesh never records, is reachable, unreferenced, and indistinguishable from something in use.
## What a module may build, and what it may only borrow
| kind | is |
|---|---|
| **image** | built from a Dockerfile in this repository |
| **archive** | a directory in this repository, packed |
| **upstream** | an image somebody else built, mirrored into the mesh's own registry |
**The third exists because a module usually runs software it did not write.** A database module
ships configuration and a provisioner and does not build a database. Naming the upstream reference
directly would need every machine to reach a public registry, and would pin to a tag its owner can
move — which is what pinning exists to prevent
([ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)). Mirroring is what the
bootstrap already does by hand; this makes it something a module can say.
An upstream reference with **no tag or digest is refused**: what gets mirrored would be whatever
`latest` means today, and a module pinned to that is not pinned.
## A module's own secret
A database has a superuser password, a broker an administrator, a registry an account. **None of
them is *for* anybody** — they are not the credential a consumer is given, and the mechanism that
hands those out has a consumer in the middle of it.
So a module says what it needs and where to put it — `own-secrets`, keyed by a name of the
module's choosing — and the mesh generates one **per node**,
seals it to that machine and reads it no more than it reads any other secret. Per node
deliberately: a module running on three machines has three passwords, where one in the manifest
would put the same secret on every machine that ever runs it, in a file anybody can read, for ever.
Made once and kept, or a running database would be handed a password it was not started with.
Remade when the machine's sealing key changes. **Declared and not made is refused**, because a
module whose own credential is silently absent starts, fails to authenticate, and the reason is
three layers from the machine reporting it.
### It is named for whose it is, not how secret it is
*2026-08-31, from an audit asking whether the manifest format was becoming hard to hold in the
head.*
The field was called `needs`, beside `secrets` — which is where a **provision's** credential lands
on a consumer. Both were name-to-path, both held something secret, and the names distinguished
them not at all. **Reaching for the wrong one parsed cleanly and failed somewhere else entirely**,
which is the shape of fault this whole design exists to prevent, sitting in the manifest format.
The axis that separates them is not how secret they are — both are — but **whose**:
| | keyed by | belongs to |
|---|---|---|
| `secrets` | the provision it is for | the relationship with another machine |
| `own-secrets` | a name the module chose | this module, and nobody else |
**A manifest using the old name is told the new one** rather than refused with "unknown field".
Whoever wrote it knew what they meant, and the mesh knows what it is called now.
### Some of them the mesh cannot make
*2026-08-31, from making the builder a module — the first thing to hold one.*
A generated secret is the mesh's, and remaking it costs nothing: **nothing else ever knew the old
one.** That is the assumption the paragraph above rests on, and it is not true of every secret a
module needs.
A broker account's password exists because **the broker was told about it**. A licence key exists
because somebody bought it. The mesh's job with these is to carry the value to the machine that
will use it and then be unable to read it — the same sealing, from the other direction: **given,
not generated.**
Treating the two alike is wrong in exactly one place, and it is the place nobody looks. When a
machine rejoins it has a new sealing key, and everything sealed to the old one is remade. Remaking
a *given* secret puts thirty-two random bytes where a working credential was, and every visible
signal says it worked: the mesh sealed a secret, the machine applied it, the file is there with
the right permissions. What fails is a program authenticating to something else, hours later,
with an error that names neither the mesh nor the secret.
So **where the value came from is recorded, and a given secret is never regenerated.** A rejoined
machine asking for one is refused, naming the remedy — issue it again — because the remedy is a
command somebody runs and no amount of pushing will produce a password the broker has never heard
of.
*Checked by taking a given secret, changing the machine's sealing key, and asserting the mesh
refuses rather than answers; and by asserting that two ordinary pushes hand back the same value,
without which the refusal would be a secret that never survives at all.*
## What one assignment gets you
A database module, written to see whether it could be:
```
directory /var/lib/mesh/postgres
directory /var/lib/mesh/postgres/grants
container the database pinned by digest, mirrored
container the provisioner pinned by digest, mirrored
file the superuser password sealed to this machine
file what its consumers asked for
```
**The provisioner watches** rather than being invoked. That is what lets it be a module: run once,
it needs something to run it after every declaration — a timer, or a unit wired to a file.
Watching, it is an ordinary long-running service the host already supervises. It polls rather than
watching the filesystem, because the host writes atomically: the file is replaced, so a watch on
the path stops seeing anything after the first replacement, and a watcher that silently stops
working is worse than a poll.
Writing it found one thing wrong, and it was the manifest rather than the host: a container
declared `restart-on`, which is a service field, and the host refused it by name. **It is right
to.** A container whose own definition changes is recreated, and a file it mounts is read by the
process inside, which is that image's business.
## Where artifacts go
**The registry the bootstrap already pulls from**, for both images and archives. An OCI registry
is a content-addressed blob store that also understands images, and an archive is a
content-addressed blob.
An object store beside it is the right answer for objects that are *mutable*, need per-reader
access, or are not build output. None of that describes a digest-pinned archive, and running a
second service for one kind of immutable blob is two things to run, two to back up, and two ways
for an artifact to be missing. **Overturnable without touching anything else**: a manifest carries
a URL and a digest, and neither says what served it.
### And the mesh runs it
*2026-08-31.* Which registry is a **provision**, mesh-scoped: a build machine requires
`artifact-store` and is told where it is, the same way an application is told where its database
is. Nothing is configured with an address.
This closes the last thing the mesh depended on and did not run. The registry a bootstrap pulls
from belongs to whoever raised the machine; from the moment the mesh has one of its own, an
artifact's home is somewhere the mesh can move, replace and back up.
**The chicken and egg is the bootstrap's, resolved the same way.** A registry module is an
`upstream` artifact — mirrored from a registry that already exists into the one being started. The
first copy comes from outside, exactly once, and every copy after it is the mesh's.
*Checked in the lab by assigning it and then asking for `/v2/` — on the machine, and from a second
machine across the private network, because a mesh-scoped provision that only answers locally is
not one. A container that is running is not a registry that replies, and this project has paid for
that distinction once already.*
03-DESIGN/01-to-be/13-credentials-and-their-rotation.md
+107
View File
@@ -0,0 +1,107 @@
---
layer: to-be
status: in-progress
code:
- mesh-control internal/inventory/secrets.go
- mesh-control cmd/mesh-control/rotate.go
- mesh-control examples/postgres-provisioner
updated: 2026-09-01
decisions:
- 02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md
- 02-DECISIONS/0009-modules-and-the-graph.md
---
# 13 — Credentials, and moving them
*Written 2026-08-31, when rotation was built. The delivery half was already proven; this is the
half [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) records as
unowned, and it was **measurably false** in the system being replaced.*
## What went wrong before, precisely
`provision_ensure`, documented as *NEVER rotates an existing secret*, minted a new password on
every adoption and updated **only the provider's row**. Consumers on three nodes held dead
credentials for two days. Two rows for one provision were written 216 ms apart, so at most one
could have matched the live role. The mesh reported success throughout.
Three separate faults, and it is worth naming them apart because they have different fixes:
| | |
|---|---|
| **one credential, many holders** | rotating it was necessarily a fan-out, and nothing enumerated who held it |
| **the record moved and the consumers did not** | the change and the delivery were different acts, and only the first happened |
| **nothing said so** | the mesh could not tell a rotated credential from a working one, so nobody looked |
## What replaces it
**Every pair has its own credential.** A provision between one consumer and one provider is one
password, made once and kept. So rotating a credential touches one role and leaves every other
consumer alone — and *who holds this* is a query rather than an assumption. That alone removes the
first fault: there is no shared secret to fan out.
**A consumer is a module on a machine, not a machine.** This was written as though a pair were two
machines, and built that way, and it was wrong in a way that only shows on a real node
([`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md)):
a machine running several services against one database server had one credential between them.
The provider refused to plan at all, and the consuming node did not refuse — it gave the first
module a credential and the rest nothing.
Two modules on one node are as separate as two on different nodes. They are different containers,
with different data, and one login opening both is the thing this page exists to prevent. It is
also what makes withdrawal possible: one role per machine cannot express *this module no longer
has a login and the others still do*.
**The change and the delivery are one command.** `rotate` discards the credential and sends both
ends, and it does the sending itself. Leaving that to whoever remembers is the second fault
exactly, and the interval in which it goes wrong is unbounded — two days, in the recorded case.
**It is all-or-nothing.** If any affected machine cannot be resolved, nothing is sent and the old
credential keeps working. A mesh that has not rotated is far better than one that has
half-rotated, and the difference is that the first is obvious.
**The window is stated rather than hidden.** A role's password changes on the provider and the file
changes on the consumer, and those cannot be simultaneous. So there is an interval in which a
consumer cannot authenticate, and the honest thing is to make it as short as the broker allows and
to say it exists. `status` names who is still behind.
## The provider makes it true, and the mesh cannot
The mesh generated the password, sealed it to the machine that must accept it, and **discarded the
plaintext** — so it cannot tell a database to start accepting it. Something on that machine reads
what the host wrote and makes it true.
That something is part of the module, not part of the control plane. **The control plane decides
and never touches a machine; a provisioner runs on the machine and touches it.** Two files, because
the mesh could not compose a document containing a value it does not have:
| | |
|---|---|
| a manifest | every consumer, what it asked for, and where its credential is |
| one file per consumer | that consumer's password, alone |
**Both, or neither works.** A provisioner given the passwords and not the manifest finds a
directory of unexplained secrets and reports that nothing has been granted — which is true, and
reads exactly like a credential that was never delivered. That has now happened once, here.
**The password a provisioner uses is itself a file the mesh wrote.** Passing it through the
environment needs a person in the middle of the one path that exists so there is not one, and puts
a superuser password where `docker inspect` prints it.
## How it is checked
Not by comparing two files. **Two ends holding a matching string proves they agree, not that either
is right** — the recorded fault produced two ends that agreed with each other and not with the
database.
So the check is three logins against a real PostgreSQL, from the consumer's own machine, over the
private network:
1. the delivered credential authenticates
2. after rotation, the new one authenticates
3. **the one that was rotated away does not**
The third is what makes it a rotation rather than an addition. Without it the check passes against
a provider that added a password and removed nothing.
**Not over loopback.** `pg_hba` trusts anything there, so every password looks correct — a
deliberately wrong one returned a row for an afternoon before that was noticed.
03-DESIGN/01-to-be/14-model-access.md
+184
View File
@@ -0,0 +1,184 @@
---
layer: to-be
status: in-progress
code:
- mesh-control internal/licences
- mesh-control cmd/mesh-control/licence.go
updated: 2026-09-05
decisions:
- 02-DECISIONS/0024-model-access-is-a-provision.md
- 02-DECISIONS/0009-modules-and-the-graph.md
---
# 14 — Model access
*[ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md) decided it and listed four
things the mesh did not have. Written 2026-08-31, when two of them were built. **The other two are
still gaps and are still written as gaps** — the record's own warning is that pretending otherwise
is how a plan becomes a surprise.*
## What was built
**A licence is a record, and the first provision no machine answers.** Everything else the mesh
brokers is answered by something running on a node. A hosted model is on nobody's machine and is
reached over the public internet, so the rule that refuses two ends sharing no private network —
correct everywhere else — must not apply to it. A machine on no private network at all can hold a
licence, and that is not a special case to remember: it falls out of the answer not being a
machine.
**The name is the operator's.** *The personal account*, *the organisation's account*. Those are
names a person uses, and the mesh uses them too, because the whole point is saying **which one** a
consumer uses — and an anonymous credential hanging off a provider cannot be said. Many to many,
so deliberately **not a claim**: two machines sharing an account is the ordinary case rather than
a collision.
**One provision name for all of them.** A module requires `model-access`, never `anthropic`. A
module that named a provider could not be moved onto a model the mesh runs itself without editing
it — and moving it is the point.
**A model in a machine's own set answers it locally**, and no record is consulted. That is what
makes *the mesh's own model* an ordinary answer rather than a parallel arrangement.
### Accept: taking a value the mesh did not make
Every other credential here the mesh generated, sealed to both ends and discarded. An API key
arrives from a person, and the missing verb was *accept*: **take a value, seal it to each holder,
discard the plaintext.** A mesh that kept operator-supplied keys readably is the arrangement this
project measured and rejected.
**It seals to the holders that exist at that moment**, and this has a consequence that must be
said out loud rather than discovered:
> A consumer put on a licence *after* the key was supplied has no key, and **the mesh cannot make
> one** — it discarded the only copy.
So that state is reported at every point a person could meet it: when the consumer is put on the
licence, in `licence list`, and — decisively — **the declaration is refused** rather than written
without the file. A machine that resolves cleanly and receives nothing fails later, somewhere that
names neither the licence nor the mesh.
**A key is read from a file or standard input, never an argument.** A key on a command line is a
key in shell history and in every process listing taken while it ran. It is never echoed back:
what is stored is unreadable by whoever holds it, the control plane included, and printing it
would put the one copy that matters on a terminal.
## Refusing is felt, and that is the design working
ADR 0024 predicted it: *a mesh holding three ways to reach a model refuses every consumer that has
not said which — which is correct and is a great deal of saying-which the first time.*
It is correct, and correct is not the same as usable. So the refusal names **the candidates and
the exact command**. The difference between a mesh that refuses helpfully and one that merely
refuses is whether anybody can act on it without going and reading something else.
## Still gaps
Unchanged from [ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md), and deliberately
not half-built:
**A consumer that is not a machine.** *This worker uses that licence* is a binding to an agent, not
to a node. What is delivered still lands on a machine; what is **chosen** is chosen per agent, and
the provisions model has no consumer identity other than a node. What exists today is per module
per machine, which is a step toward it and is not it.
*2026-08-31: this gap now has named consumers rather than hypothetical ones.*
[ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) puts two sessions on the
control-plane node — the node's own and the mesh's — each bound in its own right. See
[`15-the-agent-session.md`](15-the-agent-session.md).
**And for sessions the gap is already closed, which was not obvious.** A binding is per module per
machine, and this document called that *a step toward it and not it* — reasoning that a machine
cannot name an agent. It cannot; but the two sessions are **two modules**, because they are the
same mechanism started in different context roots and a context root is what a module delivers.
So `(node, module)` tells them apart, and asking for a licence per session needed no new consumer
identity. Checked rather than argued: two sessions on one machine hold different licences, each is
given its own key, and releasing one leaves the other.
**What is still open is the rest of the gap, and it is the harder half.** A *worker* is not one
per machine — many can run on one, from one module — so `(node, module)` cannot name them apart
and this reasoning does not extend to them. That belongs with
[ADR 0003](../../02-DECISIONS/0003-agents-are-persistent-employees.md), which is unbuilt, and it
is the reason this section stays open rather than being struck out.
**Switching is a reaction, not a declaration.** A licence that hits its limit and must be swapped is
a response to something observed. Expressing it as a declaration would make the declaration mean
*whatever is working right now*, which is not a thing anybody declared. It belongs with
observability, changing a binding — and the binding is then declared as usual. **Saying this
plainly is what stops the declaration language growing a conditional**, and nothing built here
grew one.
## The vendor-agnostic generalisation
*[ADR 0050](../../02-DECISIONS/0050-model-access-is-vendor-agnostic.md) is accepted; this section
describes what it decides. The section above stands as what is built today.*
What runs is one vendor — the mesh's Anthropic feature. A read-only trace asked whether the
`model-access` provision is Anthropic-shaped or genuinely general, and found that the general layer
already exists: a licence is a record with a `vendor` and a non-secret `serves`, a holder's
credential is sealed per holder, `accept` takes an operator-supplied value and discards the
plaintext, and a locally-run model answers at node scope with no licence. None of that names
Anthropic. What is Anthropic's is a thin band: the credential is a subscription OAuth grant — an
hourly access token and a refresh token — and that shape alone drags central rotation, a
refresh-token-stripping delivery, an identity guard and a `utilization%` usage reading behind it.
**A vendor is an adapter.** The vendor-specific lifecycle moves into a per-vendor adapter selected by
the licence's `vendor` field — the same shape as a registrar-scoped `public-dns` provider behind the
neutral `public-dns` interface ([ADR 0044](../../02-DECISIONS/0044-a-public-name-is-provisioned-like-any-capability.md)).
A consumer still requires `model-access` and never names a vendor; the interface is drawn at the
consumer's real coupling — *reach a model* — per [ADR 0040](../../02-DECISIONS/0040-what-a-module-is.md).
The field is `vendor` rather than `provider`, because "provider" already means *which node answers a
brokered provision* and the two facts must not share a word.
The adapter declares a `shape` — `static-key` or `refreshable-grant` — and, optionally, the verbs a
vendor happens to need: `accept` a supplied key, `refresh` a grant, read an `identity` off the
credential to catch a mis-binding, report `usage`, and `deliver` the value. **A static-key vendor
implements almost nothing** — a supplied key, sealed to its holders, delivered. The abstraction is
built so the common vendor is small and the rare one carries its own weight.
```
requires: model-access (the consumer, vendor-blind)
│
┌──────┴───────┐
│ a licence │ vendor: … serves: base URL, model
└──────┬───────┘
selected by │ vendor
┌─────────────────┼──────────────────────────┐
▼ ▼ ▼
anthropic-api-key anthropic (another vendor)
shape: static-key shape: refreshable-grant
accept, deliver accept, refresh, identity,
usage, deliver
```
**The crux is one relaxation, stated plainly.** A refreshable credential cannot be both sealed so the
mesh cannot read it *and* rotated centrally — rotation needs a readable refresh token, and the working
central rotation is the half [ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md) keeps
on purpose. So for `refreshable-grant` vendors only, the **manager node holds the refresh token
encrypted at rest** — a bounded, declared exception. Access tokens stay sealed per holder, and the
refresh token is stripped on delivery, so *a node never holds a refresh token* remains true for every
node but the one manager. **Static-key vendors keep the full guarantee**: there is nothing to rotate,
so `accept` discards the plaintext and the carve-out never fires — and static-key is the majority. The
exception is written down because a relaxed guarantee that is not stated is indistinguishable from a
broken one, and it is narrow on three axes at once: refreshable-grant only, the refresh token only,
the manager node only.
Usage is normalised to `(licence, consumer, period, metric, value)` with the raw response kept beside
it; the metric is vendor-defined, so no false common unit is forced. Anthropic becomes the first
`refreshable-grant` adapter, and `anthropic-api-key` — the same vendor's plain keys — is the
`static-key` case that proves the abstraction is more than one vendor in disguise.
## How it is checked
In the lab, on real machines, in the order a person would meet it: a consumer is refused with both
candidates named; put on one and still refused because no key exists; the key is given on standard
input and not echoed; the public half arrives saying it came from a record rather than a machine;
the key arrives readable only by that machine — and it is **nowhere in the control plane's own
database**, nor in anything that crossed the broker.
For the generalisation ([ADR 0050](../../02-DECISIONS/0050-model-access-is-vendor-agnostic.md)): a
second vendor — `anthropic-api-key`, static-key — is bound to a consumer and exercises the whole path
with the carve-out switched off, its key sealed per node and absent from the control plane's database.
For a `refreshable-grant` licence, the refresh token is asserted to exist (encrypted) **only on the
manager node**, to be **absent from every holder's delivery**, and the delivered credential to be
access-token-only; a `static-key` licence stores no refresh token anywhere. A scenario with two
vendors confirms each licence's lifecycle runs its own adapter, selected by `vendor`.
03-DESIGN/01-to-be/15-the-agent-session.md
+226
View File
@@ -0,0 +1,226 @@
---
layer: to-be
status: designed
code: [mesh-control, mesh-host]
updated: 2026-08-31
decisions:
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
- 02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md
- 02-DECISIONS/0024-model-access-is-a-provision.md
- 02-DECISIONS/0025-the-design-record-is-read-not-copied.md
---
# The agent session
**One mechanism, started twice.** A node's session and the mesh's session are the same thing
pointed at different context. This document describes the mechanism; where the two differ it says
so, and the differences are few enough to list here:
| | node session | mesh session |
|---|---|---|
| **address** | the node's name | the mesh |
| **context root** | the node's | the mesh's |
| **engram** | that node's | the mesh's |
| **licence** | bound in its own right | bound in its own right |
| **runs on** | that node | the node holding the control plane |
| **how many** | one per node | one |
Everything below applies to both unless it says otherwise.
## What a session is, restated for what is being built
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) settled the behaviour: permanent,
remembering across callers, its own tools, reachable over the broker, and — switched off — still
answering *I am switched off* rather than falling silent.
**Nothing said how one is set up**, and that gap is what this document closes. It was invisible
until [ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) needed to describe
a second instance, because *the same as that one, elsewhere* means nothing until the first has
been written down.
## The context root is the whole of the difference
**A session is defined by the directory it starts in.** That directory holds the engram, the
session's tools, and whatever standing instruction it works under. Two sessions differing only in
their root are two different agents, and nothing else has to differ to make them so.
This is deliberately a **small** definition. The alternative — a session type, with the mesh
session as a distinct kind — would mean two implementations of one mechanism, and
[ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) records what that
costs: two things nearly the same, built twice, until neither word means one thing.
**The root is delivered as declared state, not carried by the session.** It is files on a machine,
which is precisely what the host applies
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A session's context therefore changes the
way anything else changes — the mesh declares it, the host writes it — and there is no second
mechanism for shipping an engram.
**Changing the engram is changing a file.** So it is versioned, reviewable, and rolled back like
any other declared state; and a node whose engram was changed reports having applied it, the same
as it reports anything else.
## Where each one runs
**A node's session runs on that node**, and cannot be moved. Moved, one machine is answering as
another (ADR 0004).
**The mesh's session runs on the node holding the control plane.** The reasoning is in ADR 0026
and is worth carrying here because it is easy to get backwards: this is not *the important agent
goes on the important machine*. It is that the control-plane node is already the one place
excepted from *compromise of a node is compromise of that node*, and an agent able to reach
everything, placed anywhere else, would create a second such place.
## Messages
**A session is reached over the broker**
([ADR 0002](../../02-DECISIONS/0002-nodes-communicate-over-a-broker.md)). There is no second
transport, nothing is dialled at a session, and the mesh session is not a hop: node-to-node
messages continue to travel directly, and nothing is routed through it.
**A message becomes a prompt; a reply travels back the same way.** Whoever asked — a person at a
surface, another session, a worker — is a caller, and the session remembers what each of them
asked, together, over time.
**Being asked something it does not have, a session may ask another.** How it does so is its own
business and follows from its engram rather than from a message format: it may say who wants to
know, or simply ask. A person relaying a question makes the same choice.
**A switched-off session answers.** The queue is still read and the state is the reply, with no
model involved. This is the same rule the host follows about a service that does not exist, and
it is the rule this repository has now paid for three times: **absence must never be
indistinguishable from a failure to answer**
([005](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md),
[008](../../04-ISSUES/008-the-documented-node-rescue-does-not-exist/00-report.md)).
## Model access is bound to the session, not to the machine
**A session is a consumer in its own right.** This is the gap
[`14-model-access.md`](14-model-access.md) records as *a consumer that is not a machine*, and it
is the one part of this design that cannot be built from what exists: the provisions model has no
consumer identity other than a node, so today a binding can only say *this module on this
machine*.
That is not sufficient here, and the shortfall is concrete rather than theoretical:
- the control-plane node hosts **two** sessions, which must be able to hold **different**
licences — a per-machine binding cannot express it at all;
- *this node's session uses the personal licence, the mesh's uses the company one* is the ordinary
case, not an exotic one.
**What is delivered still lands on a machine. What is chosen is chosen per session.** Delivery
follows the session's context root, which is where its credentials belong — the same rule ADR 0001
states for an agent's config directory, with the root standing in for it.
**Switching a licence remains a reaction, not a declaration** (ADR 0024). A session that exhausts
a licence is observed and its binding changed; the binding is then declared as usual. Nothing here
grows a conditional in the declaration language.
## What it remembers, and where
**A session's memory lives in its context root**, beside its engram and its tools. That is the
same rule as everything else here rather than a new one: the root is the whole of what makes one
session a different agent from another, and memory is part of what makes it *that* agent.
**The mesh session's memory is its own.** It is not assembled from the node sessions on demand,
and it is not a view over theirs. What the mesh has been asked, and what it worked out, is held
in the mesh's root — not in the root of the node that happens to host it.
**That distinction is the point of putting it there.** The control-plane node runs two sessions
on one machine. If memory belonged to the machine rather than to the root, they would share it,
and the mesh's recollection of a fortnight of questions would be indistinguishable from that
node's own — which is the collision ADR 0026 exists to avoid, arriving through the back door.
**The root holds two kinds of thing, and confusing them destroys the memory.** The engram and the
tools are **declared**: the mesh says what they are and the host writes them, so editing one on
the machine survives until the next heartbeat and no longer
([ADR 0011](../../02-DECISIONS/0011-managed-files-are-generated-never-edited.md)). The memory is
**written by the session itself** and is declared by nobody — the mesh does not get to say what a
session remembers, and a mechanism that regenerates the root wholesale would erase a fortnight of
it on the next pass, silently, while reporting success.
So the root is not uniformly managed, and **which parts are must be explicit rather than
inferred**. A session's memory is its own output, kept across restarts, backed up as data rather
than reproduced from a declaration — because there is nothing to reproduce it from.
## What the mesh session knows
**It holds the design record by reading it**
([ADR 0025](../../02-DECISIONS/0025-the-design-record-is-read-not-copied.md)), not by holding a
copy. It is that reader; there is not a second agent for it.
**It answers into a symptom search**, so what it knows appears beside ordinary results rather than
only when it is asked. **And when it cannot be reached, the search says so.** A result set that
silently omits this material looks identical to one where nothing matched — the same rule as the
switched-off session above, at a different layer.
**Reading is one-way.** It reads the repository and answers from it; nothing flows back. The
repository is public and the mesh is not, and a return path is how installation-specific detail
arrives into documents that must not carry it.
## What it does with work
**It dispatches; it does not hold a queue.** Asked for something that belongs elsewhere, it asks a
node session or hands the work to a worker. It is not an employee
([ADR 0003](../../02-DECISIONS/0003-agents-are-persistent-employees.md)): nobody hires it, it is
never drained, and it is not reassigned.
The distinction is the one ADR 0001 was written to keep: a session comes with the thing it belongs
to and goes when that thing goes; a worker is hired, holds tasks, and moves. Built from the same
parts, run on entirely different terms.
## The surface is separate
That a person usually reaches the mesh session through a board is likely and is not settled here.
The session is reachable over the broker like everything else; what puts a text box in front of it
is a different design, and the session does not know which surface asked.
**Several sessions open at once is a property of the surface, not of the sessions.** A board
showing the mesh's session beside one per node, switchable, leaves *one per node, permanent*
untouched: each is still one conversation remembering all its callers together. Only *concurrent
conversations with the same session* would touch ADR 0004, and that is not what is wanted.
Such a surface is a **client** and holds nothing — it sends prompts and shows what the sessions
themselves remember, which is what keeps it inside `11-a-board.md`'s rule that the board is not a
second implementation of anything. Prior art to draw the interaction from: impire's *soulstream*
(already cited in [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md)) and
`herdrdev/herdr`. **Not yet designed, and not on the path to replacing provisioning** — recorded
here so the shape is not rediscovered.
## Consequences
**Two sessions on one node, and no ambiguity.** The control-plane node hosts its own node session
and the mesh session. ADR 0004's *one per node* forbids ambiguity about who answers when a **node**
is addressed; these answer to different addresses.
**A new node gets a session by being declared, not by being set up.** The context root is declared
state, so a joining node's session arrives the way its packages and services do.
**The mesh session is a single point of convenience, and must never become one of reach.** Every
node stays directly addressable. If that ever stops being true, the failure is quiet in the worst
way: every machine keeps working and nobody can ask about it.
## How it is checked
**A test names the decision it defends** ([ADR 0017](../../02-DECISIONS/0017-a-test-defends-a-decision.md)),
and these run in the lab on real machines:
| Check | Defends |
|---|---|
| a node is asked something and its session answers | ADR 0004 |
| the mesh is asked something and the mesh session answers, on the control-plane node | ADR 0026 |
| both sessions on that node answer, to their own addresses, without ambiguity | ADR 0026 |
| a session switched off replies saying so, rather than timing out | ADR 0004 |
| a session whose engram was changed reports having applied it, like any declared file | ADR 0005 |
| the two sessions on one node hold different licences, and each uses its own | ADR 0024 |
| a node is messaged directly while the mesh session is stopped, and answers | ADR 0001 |
| a search consulting an unreachable mesh session says it was not consulted | ADR 0025 |
The last two are the ones worth writing first. Both defend properties that are invisible while
everything works, and both describe a mesh that looks entirely healthy at the moment it has
stopped telling the truth.
## Deliberately not decided
**How a person's identity reaches a session.** Callers are distinguished, but who a caller *is*,
and whether a session should act differently for different people, is the human-agent question
ADR 0001 leaves open and this does not close.
03-DESIGN/01-to-be/16-module-coverage.md
+176
View File
@@ -0,0 +1,176 @@
---
layer: to-be
status: designed
code: []
updated: 2026-09-01
decisions:
- 02-DECISIONS/0009-modules-and-the-graph.md
- 02-DECISIONS/0005-the-node-host.md
- 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
---
# What a module must be able to say
**Measured, not guessed.** 127 manifests in the system being replaced were read and every key
counted, then set against what the new manifest can express. This document is the coverage
checklist: what is already sayable, what is deliberately not, and what is missing.
*Surveyed 2026-09-01. Counts are modules, not occurrences, unless stated.*
## Already sayable
| what it says | used by | how it is said here |
|---|---|---|
| **depends on another module** | 65 | `requires` naming a module. A requirement naming a module means *that module*, not anything providing the name |
| **system packages** | 28 | the `package` shape |
| **a container** | 48 | the `container` shape, pinned by digest |
| **systemd units** | 17 | a `file` for the unit, a `service` for the state it should be in |
| **how to reach it** | 17 | `serves`, with the mesh adding which machine and where |
| **a public name** | 11 | requiring `route` and contributing the name |
| **ports it opens** | 11 | `listens`, from which filtering is computed |
| **data directories, and who owns them** | 38 | `directory` with `owner`; never removed while holding anything ([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)) |
| **what it provides and requires** | 15 + 2 | `provides` / `requires`, named for what the consumer is coupled to |
| **restart when something changes** | 11 | `restart-on` |
| **a generated credential** | 20 | `own-secrets`, sealed to the machine |
| **a value that differs per node** | 43 | settings, and `computed` for what only the mesh knows |
| **images built from source** | 2 | `build.artifacts` |
## Deliberately not sayable
**Stage hooks — 36 modules.** Arbitrary code at install, configure and start. **The link may not
carry an action** ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)): what may be pushed is
bounded by form, and a command to run is not a form. A module needing setup logic ships a program
that reads what the mesh delivered and reconciles — which is what the provisioners are, and they
are ~350 lines each including the reasoning.
**Flavours — 6 modules.** Variants of one module. Retired in favour of claims
([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)): two display servers are two
modules that both claim the seat, and adding a third changes nothing anywhere else. What is lost
is `extends` chains, which were doing inheritance and are better as separate modules.
## Missing, and what each would take
Ordered by how many modules need it. The largest entry turned out not to be a gap at all, which
is left in place rather than deleted: the first framing of it was wrong in an instructive way, and
a checklist that quietly loses its biggest item reads as though nobody looked.
### Tool servers — 56 modules — *not a gap*
Over half the modules ship a `tools/` directory that becomes tools an agent can call on that node.
This was written up as the largest single gap. **It is expressible with what exists**, and the
first framing of it was wrong in a way worth keeping: *a module provides `tools`, the session
requires them* does not work, because a requirement has exactly one answer and 56 modules offering
tools would be 56 answers to one question.
Turn it around and it fits exactly. The session **provides** `tool-host`; every module offering
tools **requires** it and **contributes** where its tools are. Many-to-one is what `contributes`
has always been, and the session receives all of them in one file:
```
given: { from: gitea, values: { at: … } }
{ from: minio, values: { at: … } }
{ from: umami, values: { at: … } }
```
Verified by resolving it, not by reading the code. It also only became possible today: until
[`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md) was
fixed, several modules on one node requiring the same thing was refused outright.
What remains is not vocabulary but a decision about **what a tool server is** — a container the
module already runs, and what the session does with the list. That is work, not a missing shape.
### Schema migrations — 14 modules
A module with a database needs its schema brought up to date before it runs. The mesh does this
for its own contexts and has no way for a *module* to declare it. The provisioner pattern covers
it — a program that runs migrations and exits — but nothing expresses *this must happen before
that starts*, which is the actual requirement.
### Configuration merging — 18 modules, 134 files
Files assembled from a module's default plus per-node overrides, with a strategy (`replace`,
`merge`) and a format (`toml`, `yaml`, `json`). Settings already merge into a file's content; what
is missing is format-aware merging.
**And it should stay missing.** A mechanism that understands TOML will be asked for YAML, then
INI — which is how the arrangement being replaced became something nobody could hold in their
head. The module knows its own format because it wrote the rest of the file.
### Health checks — 7 modules
`{type: port|url, expect: …}`. The mesh knows whether a container is running, which is not the
same as whether it answers — a distinction this project has paid for twice already.
An `action` carries a `verify` and is exactly this shape. **It is not available to a module**: the
link may not carry a command to run ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and a
module's resources reach a machine over the link. So a health check needs a way to say *ask this
and expect that* without saying *run this* — closer to a `listens` entry than to an action.
**This is the gap most worth closing**, because *running* and *answering* being conflated is a
class of fault, not an inconvenience.
### Theme knobs — 3 modules, 101 values
`{theme: {kind: color|font|string, label}}` — declared so a ricing tool can offer them. Settings
already carry the value; what is missing is the **metadata** saying a value is presentable and
what kind it is. Small, self-contained, and only interesting once something presents them.
### Event routing — 2 modules
`{routing-key: tool}`, generating a consumer. Two modules; wait for a third before deciding.
### Publishing a package — 6 modules
Modules published to a registry and consumed as libraries. This is a *build* output the mesh does
not deliver to a node, so it may not belong here at all.
## What checking the coverage found
Two faults, both surfaced by asking what a *real* node looks like rather than what a test does.
Neither is about the vocabulary; both are about the machinery under it.
**A credential belonged to a machine, not to a module**
([`022`](../../04-ISSUES/022-one-credential-per-node-per-provision-not-per-module/00-report.md),
fixed). A node running three services against one database could not be planned at all — and on
the consumer's side did not refuse, it just gave two of the three no credential. Every scenario
written to date had one consumer per node, which is the natural shape of a small test and not the
shape of a machine.
**A consumer could not build a connection string**
([`023`](../../04-ISSUES/023-a-consumer-cannot-build-a-connection-string/00-report.md), fixed). It
had its password in the right shape and the host, port and user name were out of reach: the user
name was invented by the provisioner and recorded nowhere, and the bound values sat in a JSON
document that an application reading `KEY=value` cannot use.
The asymmetry was backwards, which is what made it worth stating. **The secret is the hard case** —
the mesh must not be able to read it — and the secret was the part that already arrived. The host
and port are ordinary facts held in the clear, and they were the ones stuck. Both halves came from
the same thing: the mesh knew something and did not say it.
## What the survey found that is not about coverage
**Declaration and reality had drifted in the system being replaced.** Several live provisions are
brokered by modules whose manifests declare nothing — a speech-to-text engine served to a consumer
on another node, an object-store bucket held by a module whose manifest mentions only its
database. **A manifest that does not have to be true stops being true**, which is the argument for
resolution refusing rather than warning.
**Two derivations of the same fact.** A module's kind was computed in two places from different
evidence — one from the manifest, one from what is on disk — producing different labels for the
same module. There is one derivation here, and there should stay one.
**A live listing returned credentials in plaintext.** Not a coverage question, but the reason
sealing is worth its inconvenience.
**An image store is a module, and was written up here as something the mesh does.** It was
considered for the substrate and removed, because the test is not *can it grant itself one* —
nearly anything passes that — but whether the control plane needs it before it can give its first
instruction. It does not. So a registry somebody runs for their own images is the same module as
the one the mesh runs for its own: it offers a place to push, and claims that role once per
machine.
**A rule was enforced only at the far end.** A module may not declare an action, and the host
refused one correctly — but the control plane accepted it into the catalogue, resolved it and
pushed it, so the refusal arrived on a machine with nothing tying it back to the manifest. The
rule held; it was just unusable, which is the same shape as the network shape that cost five
failing tests before anyone read the host's log. It is now refused where it is written.
03-DESIGN/01-to-be/README.md
+27 -12
View File
@@ -9,18 +9,33 @@ document is written and this one's status becomes `implemented`.
| Document | Covers | Rests on |
|---|---|---|
| [`00-work-breakdown.md`](00-work-breakdown.md) | How the decomposition gets built, in what order, and where a human must look | [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md) |
| [`01-end-to-end-testing.md`](01-end-to-end-testing.md) | The lab: a real mesh a change can be run against before it reaches nodes | [ADR 0016](../../02-DECISIONS/0016-a-lab-node-is-a-virtual-machine.md), [0029](../../02-DECISIONS/0029-the-labs-first-scenario-has-no-pipeline.md) |
| [`02-scenario-declaration.md`](02-scenario-declaration.md) | What a scenario declares — the underlay, and what to place on it | [ADR 0031](../../02-DECISIONS/0031-the-lab-provides-the-underlay.md) |
| [`03-scenario-lifecycle.md`](03-scenario-lifecycle.md) | What happens to a scenario — raise, snapshot, restore, move, destroy | [ADR 0032](../../02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md) |
| [`04-lab-installation.md`](04-lab-installation.md) | Getting the lab onto a clean machine, and why it verifies capability rather than installation | [ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) |
| [`05-the-node-host.md`](05-the-node-host.md) | Tier 0 — the one thing installed by hand, and the only thing that changes a machine | [ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md) |
| [`00-work-breakdown.md`](00-work-breakdown.md) | How modules move across one at a time, until the old registry can be switched off | [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md), [ADR 0016](../../02-DECISIONS/0016-the-lab.md) |
| [`01-end-to-end-testing.md`](01-end-to-end-testing.md) | The lab: a real mesh a change can be run against before it reaches nodes | [ADR 0016](../../02-DECISIONS/0016-the-lab.md), [0029](../../02-DECISIONS/0016-the-lab.md) |
| [`02-scenario-declaration.md`](02-scenario-declaration.md) | What a scenario declares — the underlay, and what to place on it | [ADR 0016](../../02-DECISIONS/0016-the-lab.md) |
| [`03-scenario-lifecycle.md`](03-scenario-lifecycle.md) | What happens to a scenario — raise, snapshot, restore, move, destroy | [ADR 0016](../../02-DECISIONS/0016-the-lab.md) |
| [`04-lab-installation.md`](04-lab-installation.md) | Getting the lab onto a clean machine, and why it verifies capability rather than installation | [ADR 0010](../../02-DECISIONS/0010-delivery.md) |
| [`05-the-node-host.md`](05-the-node-host.md) | Tier 0 — the one thing installed by hand, and the only thing that changes a machine | [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
| [`06-the-control-plane.md`](06-the-control-plane.md) | Tier 2 — what the term means, and the test for what belongs in it | [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0048](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) |
| [`08-connectivity.md`](08-connectivity.md) | One context in full — overlay, resolution, exposure, filtering, certificates | [ADR 0007](../../02-DECISIONS/0007-connectivity.md), [0050](../../02-DECISIONS/0007-connectivity.md), [0051](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0055](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md) |
| [`09-the-node-lifecycle.md`](09-the-node-lifecycle.md) | How a machine becomes a node, stays one, and stops being one | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [0051](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) |
| [`10-delivery.md`](10-delivery.md) | Modules, the three edges, and how a change becomes a running thing | [ADR 0010](../../02-DECISIONS/0010-delivery.md), [0064](../../02-DECISIONS/0009-modules-and-the-graph.md), [0065](../../02-DECISIONS/0009-modules-and-the-graph.md) |
| [`11-a-board.md`](11-a-board.md) | What a person sees of the mesh, and why it is read from what runs | [ADR 0008](../../02-DECISIONS/0008-a-context-owns-its-store.md), [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md) |
| [`12-a-module-repository.md`](12-a-module-repository.md) | A module repository, and what builds it | [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md), [ADR 0010](../../02-DECISIONS/0010-delivery.md), [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
| [`13-credentials-and-their-rotation.md`](13-credentials-and-their-rotation.md) | Credentials, and moving them without a consumer holding one the provider does not know about | [ADR 0001](../../02-DECISIONS/0001-mesh-brokers-nodes-host-agents-think.md), [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) |
| [`14-model-access.md`](14-model-access.md) | Model access as a provision, and what a licence is bound to | [ADR 0024](../../02-DECISIONS/0024-model-access-is-a-provision.md), [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) |
| [`15-the-agent-session.md`](15-the-agent-session.md) | One mechanism started twice — a node's session and the mesh's | [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md), [ADR 0026](../../02-DECISIONS/0026-the-mesh-has-a-session-of-its-own.md) |
| [`16-module-coverage.md`](16-module-coverage.md) | What a module must be able to say, measured against 127 that exist | [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md), [ADR 0005](../../02-DECISIONS/0005-the-node-host.md) |
## Not yet written
- **The eight bounded contexts.** [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md)
decides the decomposition; the per-context specifications do not exist yet. The work
breakdown says in what order they are needed.
- **Domain grouping outside the core.** [ADR 0017](../../02-DECISIONS/0017-modules-outside-the-core-are-grouped-by-domain.md)
settles the principle and explicitly does not settle the domain list. That is a research
effort, not a design document, until it concludes.
- **The remaining six contexts.**
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md)
settles the list at seven; `connectivity` is the first written in full
([`08`](08-connectivity.md)) and the other six do not exist yet. The work breakdown says in
what order they are needed.
- ~~**Domain grouping outside the core.**~~ **Not needed.**
[ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) is
superseded by [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md):
there is no domain module to group into, so there is no domain list to settle. Relationships
are edges, and grouping is a tag and a query.