Remove the decision records and research efforts, which the knowledge base now holds whole (issue 426)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the change's judges the pull requests after it merges; it…
mesh/delivery superseded: a newer head of the same pull request
ADR 0290 rule 9: once the import is checked record by record, hq keeps no copy. The records and efforts are removed; every link or path to them reads "ADR NNNN" or "research NNN"; a design's decisions: lists "ADR NNNN". 02-DECISIONS keeps a pointer README with the reading order's six topics, which the knowledge base reads through the records module; 01-RESEARCH keeps a pointer README. The checks drop the rules that judged records (the knowledge base holds each at propose; checks/README.md says where) and refuse a record written here again. Playbooks 01 and 02 and the skills write and read records with the knowledge base's verbs.
This commit is contained in:
@@ -1,235 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-08-22
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 1. The mesh brokers capabilities; nodes host; agents think
|
||||
|
||||
## Context
|
||||
|
||||
HAL has 124 modules. The count is not the problem — it is the symptom. Modules are split
|
||||
because splitting is the only granularity the platform offers, and domains are merged
|
||||
because a shared database is the only integration it offers. Both pressures push in the
|
||||
same direction: boundaries end up drawn by deployment accident rather than by domain.
|
||||
|
||||
Three observations establish the state.
|
||||
|
||||
**The word "agent" means two different things.** The original design treated a node as an
|
||||
agent with thinking abilities. Later, noxflow implemented agents as employees with
|
||||
skills, workspaces and tasks. Both survive. The collision is visible in the data — there
|
||||
are **two agent rows per node**:
|
||||
|
||||
| agent | skills |
|
||||
|---|---|
|
||||
| one named after the node | `{deploy,verify,operate}` |
|
||||
| one named `hal-<node>` | `{}` |
|
||||
|
||||
One carries the work; the other carries only identity, existing to hold a licence for
|
||||
HAL's own sessions. The same split appears in the schema: `nodes.hal_claude_account` and
|
||||
`agents.claude_account` are one fact in two tables in two databases, and what was
|
||||
documented as "three licence touchpoints" is one concept modelled three times.
|
||||
|
||||
**Domains integrate by sharing a schema.** `noxflow` is 45 tables spanning five domains —
|
||||
tasks (7), agents (13), knowledge (8), meetings (7), scheduling (2). Its original purpose
|
||||
is 7 of 45. Because agent identity lives in noxflow's database, work that belongs
|
||||
elsewhere must be implemented there: per-agent Claude credentials had to be written by
|
||||
the noxflow runtime, even though node identity — the same kind of fact — lives in the
|
||||
mesh registry.
|
||||
|
||||
**The core domain has no context to live in.** The mesh's distinguishing feature is that
|
||||
a module declares `requires: postgres/database` and never learns where the database
|
||||
lives, who owns the credential, or how it rotates. That is capability brokering, and it
|
||||
is what makes this a mesh rather than four machines with a configuration manager. Yet the
|
||||
logic implementing it sits in `modules/postgres/tools/index.ts` — a provider module,
|
||||
where "the mesh brokers credentials" cannot be expressed. On 2026-08-22 three of its
|
||||
invariants were found violated simultaneously (see Consequences).
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep the current layout; fix bugs as they surface.** Rejected. The faults are not
|
||||
independent. Every incident on 2026-08-22 — credentials written by the wrong module, a
|
||||
dependency question answered wrongly twice, an invariant enforced nowhere — traced to
|
||||
a boundary that was never stated. Fixing them individually leaves the generator intact.
|
||||
|
||||
2. **Merge aggressively into few large modules.** Fewer names, same problem: a shared
|
||||
schema across domains is what produced noxflow, and doing it deliberately would
|
||||
produce it again at larger scale.
|
||||
|
||||
3. **Decompose by bounded context, with the mesh as a broker.** Name contexts after their
|
||||
aggregates, integrate through a published record rather than a shared schema, and let
|
||||
deployment granularity be a feature-level concern rather than a reason to create a
|
||||
module. **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
### The domain, in one sentence
|
||||
|
||||
**The mesh brokers capabilities. Nodes are places where work runs. Agents are personas
|
||||
that think and act.** Everything else supports one of those three.
|
||||
|
||||
### What this is, plainly — and what "mesh" does not mean
|
||||
|
||||
*Written 2026-08-29, from working through connectivity and asking whether the word still fits.*
|
||||
|
||||
Four layers. Naming them honestly is worth more than the word on the tin:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **machines are linked by a private network** | and every machine reaches every other over it |
|
||||
| **one node holds knowledge of all of them** | the control plane, and only it |
|
||||
| **modules are how anything is built and delivered** | this *is* the CI/CD, not something beside it ([ADR 0010](0010-delivery.md)) |
|
||||
| **every node runs a session you can message** | a feature of the node, remembering across callers; any node can message any other ([ADR 0004](0004-a-node-and-how-it-joins.md)) |
|
||||
| **workers are hired onto nodes to do tasks** | employees, with a lifecycle — a different thing from the row above ([ADR 0003](0003-agents-are-persistent-employees.md)) |
|
||||
|
||||
**The last two rows are not the same thing and the vocabulary of one does not describe the other.**
|
||||
A node's session comes with the machine: nobody hires it, it holds no tasks, it is never
|
||||
reassigned, and it goes when the node leaves. A worker is an employee — named, hired, drained,
|
||||
retired, movable. They are built from the same parts and run on entirely different terms, and
|
||||
collapsing them is how the employee vocabulary ends up stretched over something it does not fit.
|
||||
|
||||
**Neither makes the node itself a thinking thing.** A node is a machine; both of these run *on*
|
||||
one, which is why *a node does not authenticate to a model provider, agents do* is unaffected by
|
||||
either.
|
||||
|
||||
**Where the value is** is that both can reach across the whole set: a shell, a service, a file, or
|
||||
simply a question to another node. Not machines that can be configured centrally, which is
|
||||
ordinary, but a set of machines that can be worked across as though they were one.
|
||||
|
||||
**This is not a mesh in the peer-to-peer sense and will not become one.** The word describes what
|
||||
machines can reach, not how they are governed:
|
||||
|
||||
| | a mesh? |
|
||||
|---|---|
|
||||
| what a machine can reach | **yes** — genuinely any to any |
|
||||
| how the traffic travels | no — anything crossing sites transits the hub |
|
||||
| who decides | no. One node, declared |
|
||||
|
||||
**And *master* overstates it in the other direction.** A master implies the others need it in
|
||||
order to function. They do not: every node holds what it was last told and runs from that copy
|
||||
*always* — not as a fallback, as the only mode it has. So the control plane being gone is every
|
||||
node in the ordinary disconnected situation at once, and **what is lost is change, not
|
||||
operation.**
|
||||
|
||||
The accurate phrase is **one authority, no failover**, and both halves are deliberate
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md)).
|
||||
|
||||
### Nodes and agents are decoupled
|
||||
|
||||
A node is a place where an agent can run — that is the entire relationship. There is no
|
||||
resident agent, no node-owned identity, no ownership in either direction. Agents named
|
||||
after a node remain, as **ordinary agents** that happen to hold infra skills.
|
||||
|
||||
Consequently `nodes.node_license` and `nodes.hal_claude_account` cease to exist: a node
|
||||
does not authenticate to a model provider, agents do. The two agent rows per node merge.
|
||||
|
||||
### There is one kind of participant, and some are human
|
||||
|
||||
Human and non-human participants are both **agents**. Both hold identity and credentials;
|
||||
both act, remember and coordinate. What differs is **modality** — how an agent acts:
|
||||
|
||||
| modality | credential is delivered to |
|
||||
|---|---|
|
||||
| spawned session | that agent's own config directory |
|
||||
| shell or desktop | that agent's home on the node it acts from |
|
||||
|
||||
A node holds no licence. **An agent holds credentials, and delivery follows that agent's
|
||||
node bindings and modality.** A human agent's grant arrives in the home directory of the
|
||||
user it acts as, on the nodes it is bound to — the same rule that puts a spawned agent's
|
||||
grant in its config directory, with a different target.
|
||||
|
||||
This requires one fact the mesh does not record today: which user, on which node, a given
|
||||
human agent acts as. Adding it is what removes the node licence — the node is currently
|
||||
standing in for an identity the mesh cannot name. It is also what makes a second human
|
||||
agent require no new mechanism.
|
||||
|
||||
### Contexts
|
||||
|
||||
| context | aggregate | subdomain |
|
||||
|---|---|---|
|
||||
| `hal/mesh` | **Provision**, Node, Module — brokering and its bookkeeping | core |
|
||||
| `hal/agents` | **Agent** — identity, licence, runs, memory, thoughts | core |
|
||||
| `hal/work` | **Task** — workflows, bindings | core |
|
||||
| `hal/stream` | **Thread** — mentions, messages, meetings, notifications | core |
|
||||
| `hal/delivery` | **Pipeline** — jobs, artifacts, features | supporting |
|
||||
| `hal/knowledge` | **Document** — spaces, revisions, review | supporting |
|
||||
| `hal/ai` | **Licence** — provider grants and rotation | supporting |
|
||||
| `hal/observability` | **Check** | supporting |
|
||||
| `hal/config` | **Setting** — env, secrets, PKI | generic |
|
||||
|
||||
A *brain* — memory, thoughts, cognition — is a concept the Agent aggregate owns. It is
|
||||
not a module. Anatomy makes attractive names and poor boundaries; today `hal/brain` names
|
||||
infrastructure and `hal/cortex` describes itself as messaging while running nowhere.
|
||||
|
||||
### Provisioning is the core domain, not plumbing
|
||||
|
||||
`hal/mesh` is a broker; the registry is its bookkeeping. Its invariants are explicit and
|
||||
owned:
|
||||
|
||||
- one rotation source per resource
|
||||
- a credential change fans out to every consumer
|
||||
- a consumer never holds a credential the provider does not know about
|
||||
|
||||
### Contexts integrate through the record, never a shared schema
|
||||
|
||||
`hal/stream` is the published language. A context publishes; it does not join across a
|
||||
boundary. This is what dissolves "meetings" as a domain — a meeting is a thread, and a
|
||||
notification is a mention not yet read.
|
||||
|
||||
### Third-party software leaves the repository
|
||||
|
||||
`plex`, `sonarr`, `postgres`, `verdaccio`, `docker-registry` and 88 others run **on** the
|
||||
mesh; they are not **of** it. The pipeline and provisioning are deliberately
|
||||
module-agnostic, so HAL's own modules dogfood exactly what external modules use — which
|
||||
is what makes the separation safe rather than merely tidy.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The invariants now have an owner, and were measurably unowned before.** On 2026-08-22,
|
||||
`provision_ensure` — documented as "NEVER rotates an existing secret" — was found to mint
|
||||
a new password on every adoption and update only the provider's row. `hal_notifications`
|
||||
consumers on three nodes held dead credentials for two days; `hal_transcripts` had two
|
||||
rows written 216 ms apart, so at most one could match the live role.
|
||||
|
||||
**noxflow dissolves.** `hal/work` inherits tasks and workflows — the concept it was built
|
||||
for. Agents, knowledge, meetings and scheduling return to their contexts. The name goes
|
||||
away.
|
||||
|
||||
**`hal/sdk` shrinks.** 155 files, 34,636 lines, containing code from every context —
|
||||
including `workflow-engine.ts` and `task-commands.ts`, work-domain logic in the kernel
|
||||
every module imports. Each landed there to avoid a cycle between modules that both needed
|
||||
it; a domain module can only own its shared code once the domain has a module. Extraction
|
||||
is therefore downstream of this decision, not independent of it.
|
||||
|
||||
**Two mechanisms are prerequisites, not follow-ups.**
|
||||
|
||||
- *Named features with per-node opt-in.* Without it, every independently deployable unit
|
||||
inside a context becomes a module again and the count returns. `hal/claude-licences`
|
||||
exists solely because one daemon must run on one node.
|
||||
- *A local mesh in containers.* `dev_up` starts providers "via systemctl (same as
|
||||
production)" — it borrows the host, and no mesh can be stood up locally. Everything that
|
||||
manifests **between** nodes is therefore discoverable only in production, which is where
|
||||
every fault of 2026-08-22 was found. A refactor of this size is otherwise unverifiable.
|
||||
|
||||
**Credential delivery becomes uniform.** Per-agent credential directories, built for
|
||||
spawned sessions, extend to human agents unchanged. A file previously scoped to a node
|
||||
becomes scoped to an agent — which would have prevented the class of failure where a
|
||||
rotation reached one node of four while the mesh reported success.
|
||||
|
||||
**Migration is incremental and long.** Contexts can be extracted one at a time behind the
|
||||
existing pipeline. Nothing here requires a flag day, and nothing here is cheap.
|
||||
|
||||
## References
|
||||
|
||||
- [`01-RESEARCH/001-module-domain-decomposition`](../01-RESEARCH/001-module-domain-decomposition/analysis.md)
|
||||
— current-state evidence, table counts, open questions
|
||||
- [`00-META/how-we-build.md`](../00-META/how-we-build.md) — naming and integration rules
|
||||
- `modules/hal/sdk/src/feature-handlers/index.ts` — `FEATURE_HANDLERS`, the fixed handler
|
||||
array that makes a feature a singleton per module
|
||||
- `modules/postgres/tools/index.ts` — the adoption path that rotates a shared credential
|
||||
- Mediahuis `papa-hq`, ADR 0020 *Composable, independently-shippable modules* — the
|
||||
constraints that make a unit independently shippable, applicable unchanged to features
|
||||
- impire.io / soulstream — *the record* as integration substrate, personas over services,
|
||||
and "cheap awareness and expensive thinking"
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-02-25
|
||||
deciders: jochen
|
||||
reconstructed: true
|
||||
---
|
||||
|
||||
# 2. Nodes communicate over a message broker, not over HTTP
|
||||
|
||||
> Reconstructed after the fact from the evidence cited below. The decision was taken in
|
||||
> implementation, not in a record; this document states what was decided and why, not a
|
||||
> deliberation that happened.
|
||||
|
||||
## Context
|
||||
|
||||
The mesh is a set of machines that must call each other's capabilities. On the day the
|
||||
repository was founded there was no inter-node transport at all — each node was configured
|
||||
independently and shared nothing at runtime.
|
||||
|
||||
Three properties were required and are visible in everything built since:
|
||||
|
||||
- A node behind a household NAT must participate fully. It can dial out; nothing can dial in.
|
||||
- A node that is asleep, rebooting or upgrading must not cause a caller to fail — the request
|
||||
should wait, not error.
|
||||
- Adding a node must not require editing anything on the nodes that already exist.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **HTTP APIs between nodes.** Rejected. Every node becomes a server that every other node
|
||||
must be able to reach, which the NAT case makes impossible without inbound tunnels to each
|
||||
participant. It also makes node liveness a caller's problem: a request to a sleeping node
|
||||
is an error rather than a wait.
|
||||
2. **Polling a shared database.** Rejected. Latency is the poll interval, load is constant and
|
||||
independent of demand, and request/reply has to be built on top of it by hand.
|
||||
3. **A central message broker with per-node exchanges.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
All inter-node communication goes through a message broker. Every node owns a topic exchange
|
||||
named for itself and a request queue; a shared mesh exchange carries commands and events that
|
||||
are not addressed to one node.
|
||||
|
||||
Three message shapes, and only three:
|
||||
|
||||
- **RPC** — request/reply, for calling a capability that lives on another node.
|
||||
- **Commands** — instructions to do a stage of work, addressed by what is to be done.
|
||||
- **Events** — statements that something happened, addressed to nobody.
|
||||
|
||||
Every node dials the broker outbound. Nothing dials a node.
|
||||
|
||||
## Consequences
|
||||
|
||||
- NAT stops being an architectural concern. A node's reachability is a property of the broker
|
||||
connection, not of its network position.
|
||||
- A call to a node that is down waits in that node's queue instead of failing. This is usually
|
||||
right and occasionally the wrong thing entirely — a queued command for a node that never
|
||||
returns is a stall with no error, which is the failure shape this mesh keeps rediscovering.
|
||||
- The broker is a single point of failure and a single point of trust. Its credential is
|
||||
mesh-wide, so rotating it is a mesh-wide operation.
|
||||
- Tools never leave the host: a remote call proxies over the broker and the credentials stay
|
||||
where the capability is.
|
||||
|
||||
## References
|
||||
|
||||
- The broker was stood up on 2026-02-25, the second day of the repository.
|
||||
- Knowledge base: `mesh` (transport, exchanges, queue naming), `troubleshooting/amqp-credential-rotation`.
|
||||
- The stall shape is recorded in `troubleshooting/empty-pipeline-blocks-the-queue` and
|
||||
`troubleshooting/daemon-and-tool-server-share-a-request-queue`.
|
||||
@@ -1,75 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-07-12
|
||||
deciders: jochen
|
||||
reconstructed: true
|
||||
---
|
||||
|
||||
# 3. An agent is a persistent employee, not an instance of a pool
|
||||
|
||||
> Reconstructed after the fact from the evidence cited below.
|
||||
|
||||
## Context
|
||||
|
||||
Agents were originally a **pool**: a named kind of worker, scaled to some number of
|
||||
interchangeable instances. Work went to whichever instance was free.
|
||||
|
||||
That model has no place to put the things that turn out to matter. An agent that accumulates
|
||||
knowledge of a domain cannot keep it, because the next task lands on a different instance. An
|
||||
agent cannot own a workspace, because there are several of it. It cannot be held to a policy —
|
||||
warned for a violation, then dismissed — because there is no continuing subject to warn.
|
||||
|
||||
Scaling was also solving a problem the mesh does not have. Instances were being multiplied to
|
||||
get concurrency, when concurrency is a property of how much work one agent may hold at once.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Keep the pool, attach memory to the pool.** Rejected: shared memory across
|
||||
interchangeable workers is a knowledge base, not an agent's experience, and the mesh
|
||||
already has one.
|
||||
2. **Keep the pool, make instances sticky.** Rejected as a pool pretending to be identities —
|
||||
identity by scheduling accident, lost on any restart.
|
||||
3. **One agent is one persistent identity, with concurrency as a property of it.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
An agent is a **singular, named, persistent identity**: a home node, a workspace on that node,
|
||||
accumulating memory, and a lifecycle — hired, active, draining, retired. Not a pool member.
|
||||
|
||||
Concurrency is a property of the agent, not a count of copies: an agent has a cap on how many
|
||||
sessions it may hold at once.
|
||||
|
||||
Lifecycle is explicit and has verbs. An agent is hired onto a node; it may be reassigned while
|
||||
idle; it is retired by draining first, and forced only deliberately. Retired agents are not
|
||||
deleted.
|
||||
|
||||
Surge capacity is expressed within the model rather than against it: a template agent is a
|
||||
blueprint, cloned into a real agent with a lifetime when a queue grows, drained and retired
|
||||
when it expires. A temporary employee is still an employee.
|
||||
|
||||
Some agents are **human**. What differs is modality — how the agent acts — not category. A node
|
||||
itself is an agent of a kind exempt from the hiring lifecycle.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Memory, workspace and reputation have a subject to belong to. Policy becomes possible: an
|
||||
agent that violates a rule can be warned, and warned agents can be dismissed.
|
||||
- The mesh gained a hiring model, and with it the question of who may hire.
|
||||
- Scaling by adding instances is gone. If one agent is saturated, either its session cap rises
|
||||
or another agent is hired — both deliberate acts.
|
||||
- The transition was not free. Lifecycle columns had to reach every query that selects an
|
||||
agent, and the ones that were missed failed at the moment of hiring rather than at startup.
|
||||
- This is the decision [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) generalises:
|
||||
one kind of participant, differing only in modality.
|
||||
|
||||
## References
|
||||
|
||||
- `docs(adr): agents as persistent employees + MINERVA librarian` (#495), 2026-07-12 — the
|
||||
original record, in the code repository.
|
||||
- `feat(B4): one persistent employee, N sessions — rename max_instances → max_sessions` (#547)
|
||||
and `feat(noxflow): B3 — workspace provisioner for agent employee model` (#549), 2026-07-20.
|
||||
- `feat(noxflow): warn-then-fire agents who merge to main without review` (#209), 2026-06-01 —
|
||||
policy that presumes a continuing subject, predating the model that provides one.
|
||||
- Knowledge base: `agents/employee-lifecycle`, `agents/temp-surge`, `agents/workspace-layout`.
|
||||
- The migration cost: `troubleshooting`/`noxflow-agent-enriched-select-missing-lifecycle-columns` (#546).
|
||||
@@ -1,263 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 4. A node, and how it joins
|
||||
|
||||
*Consolidated 2026-08-28 from four records.*
|
||||
|
||||
## What a node is
|
||||
|
||||
**A managed machine inside the mesh.** Not a device that is merely known about, not an
|
||||
unprivileged something. If the mesh does not manage it, it is not a node — it is a client, a
|
||||
peer, or a thing on the network, and those want their own names rather than a weakened version of
|
||||
this one.
|
||||
|
||||
**A disconnected node is still a node, in a different situation.** Reachability is **state, not
|
||||
class**. A machine switched off, roaming, or behind a connection that has dropped has not become
|
||||
a lesser kind of thing; it has a last-known state and a pending set of declarations.
|
||||
|
||||
The distinction people reach for is real, but it is **capability** — what this machine can be
|
||||
asked to do — and that belongs in the host's profile rather than in the definition of a node.
|
||||
|
||||
**This is the rule that does the most work elsewhere.** A single control plane is tolerable
|
||||
because its absence is every node in the ordinary disconnected situation at once. An episodic
|
||||
host on a phone is that situation more often. Neither needed a new mechanism.
|
||||
|
||||
### A node runs one agent session
|
||||
|
||||
*Written 2026-08-29. It runs on every node today and appeared in no record, which is how something
|
||||
deliberate comes to look accidental.*
|
||||
|
||||
**A node is a machine. The session is a feature of it** — one of the things running there, like the
|
||||
host, like any workload. The node does not think; something on the node does. Which is why
|
||||
[ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)'s *a node does not authenticate to a model
|
||||
provider, agents do* holds unchanged: the session authenticates, and it is not the machine.
|
||||
|
||||
**It is permanent, and it remembers.** Anything in the mesh can send it a message; it replies; and
|
||||
what it was asked ten minutes ago is still there next week, alongside what everything else asked in
|
||||
between — the same way both sides of any conversation remember it.
|
||||
|
||||
Its system prompt is the node's **engram** — what makes one node's replies recognisably its own
|
||||
rather than generic.
|
||||
|
||||
**It has its own tools**, and fewer than a session a person is driving directly. So a question can
|
||||
be answered by going and looking: *what is in our forge*, not only *what is your battery*.
|
||||
|
||||
**Messages travel the broker like everything else** ([ADR 0002](0002-nodes-communicate-over-a-broker.md)).
|
||||
There is no second transport and nothing is dialled.
|
||||
|
||||
**Any node can message any node, and this is the one part of the system that is genuinely a mesh**
|
||||
— symmetric, with no centre. A node that is asked something it does not have can ask another, and
|
||||
how it passes the question on is its own business: it may say who wants to know, or simply ask. A
|
||||
person relaying a question makes the same choice, and it follows from the engram rather than from a
|
||||
message format.
|
||||
|
||||
**There is no authorisation between nodes.** Every node is the operator's own, so a message from
|
||||
one is a message from them, and asking a node something is asking a colleague rather than
|
||||
presenting credentials. Stated once so it is not discovered later: **the mesh boundary is therefore
|
||||
the security boundary** — anything inside can reach what any node can reach, which is what puts the
|
||||
whole perimeter on the token and the overlay
|
||||
([ADR 0007](0007-connectivity.md)).
|
||||
|
||||
**It can be switched off, and switched off it still answers.** A node whose session is disabled
|
||||
replies saying so, at once, with no model involved — the queue is still read, and the state is the
|
||||
reply. That is deliberate and it is the same rule the host follows about a service that does not
|
||||
exist: **absence must never be indistinguishable from a failure to answer.** A node with nothing
|
||||
there is a silence somebody has to go and diagnose; a node that says *I am switched off* is not.
|
||||
|
||||
**One per node, always, and it cannot be moved to another machine.** Two and nothing decides which
|
||||
replies; none and the node is mute; moved, and one machine is answering as another.
|
||||
|
||||
**It is not an employee** ([ADR 0003](0003-agents-are-persistent-employees.md)). Nobody hires it,
|
||||
it holds no tasks, it drains nothing and it is never reassigned — that vocabulary was written for
|
||||
workers and does not describe this. It exists because the node does, and it is gone when the node
|
||||
leaves.
|
||||
|
||||
## How it joins
|
||||
|
||||
**The host has one behaviour and two sources of declaration.** What differs between the first
|
||||
node and the fiftieth is not what the host does but where the declaration comes from — and, as
|
||||
above, that is a situation rather than a class.
|
||||
|
||||
| | declaration comes from |
|
||||
|---|---|
|
||||
| no mesh reachable | the pinned bundle the host carries |
|
||||
| mesh reachable | the control plane, over the link |
|
||||
|
||||
**The first node is not a different kind of node.** It is a node whose mesh is not up yet. It
|
||||
applies the bundle it carries, the control plane comes up on top of it, and from that moment it
|
||||
takes declarations like everything else. **Its specialness is temporary and self-erasing**, which
|
||||
is what the hand-run bootstrap scripts never were.
|
||||
|
||||
**A joining node does the minimum to be reachable and nothing else** — an identity, an address,
|
||||
and one peer to reach. It does **not** compute the overlay: the whole peer set is derived
|
||||
centrally and pushed down.
|
||||
|
||||
That is also why the migration is smaller than it looked. The hard part of the overlay — every
|
||||
node's key, address, site and reachability — is only needed to compute the *whole* mesh, and a
|
||||
joining node needs one peer.
|
||||
|
||||
## The link is the security boundary
|
||||
|
||||
**Everything reaching a node arrives one way**, and four properties make that a boundary rather
|
||||
than a pipe.
|
||||
|
||||
**It is outbound and node-initiated.** The node dials the control plane; nothing dials a node. Not
|
||||
only defensive — most nodes sit behind a household connection with no forwarded port, so an
|
||||
inbound control channel would work for one node and not the rest, and the difference would be
|
||||
invisible until it mattered. **A node has no listening control surface at all.**
|
||||
|
||||
**A node holds its own identity and nothing else.** No shared secret, no credential to anything it
|
||||
does not own. **Compromise of a node is compromise of that node** — which the current arrangement
|
||||
does not have, because every node permanently holds the same database and object-store
|
||||
credentials, and there is no mechanism that rotates one and informs everything holding it.
|
||||
|
||||
### What that identity is: a keypair the node generates
|
||||
|
||||
*Written 2026-08-29. This is the same rule as the sentence above, and it had been treated as an
|
||||
open question for weeks because of a word.*
|
||||
|
||||
**The node generates a keypair. The private half never leaves the machine. The mesh records the
|
||||
public half.** Ed25519, the same as the control plane's signing key, in the other direction:
|
||||
the mesh proves itself to a node by signing, and a node proves itself to the mesh by signing.
|
||||
|
||||
**This was never open.** [`08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md) already
|
||||
says it of the overlay keys, in these words: *each node generates its own keypair, the private key
|
||||
never leaves the machine, the public key is published to the mesh* — and adds that this **is**
|
||||
ADR 0004's *a node holds its own identity*, applied. What was missing was applying it to the thing
|
||||
this record is about.
|
||||
|
||||
**The word that caused it:** the lifecycle says a joining node *receives* its own durable identity,
|
||||
which reads as the mesh issuing something, and then the question becomes *issuing what*. It does
|
||||
not issue anything. The node arrives holding its identity; what it receives is **being known**.
|
||||
Enrolment is the moment the mesh writes down a public key it will believe, and the one-time secret
|
||||
is what buys the right to have it written down.
|
||||
|
||||
**Everything above then holds literally.** Nothing is stored that could be stolen and replayed: the
|
||||
mesh's copy is a public key, so a copy of the mesh's database grants nothing. *Compromise of a node
|
||||
is compromise of that node* becomes true rather than aspirational, because the only secret on a
|
||||
machine is the one that identifies it.
|
||||
|
||||
### What "connecting to the mesh" is, concretely
|
||||
|
||||
*Written 2026-08-29, because it was asked and this record had never said it.*
|
||||
|
||||
**One outbound AMQP connection from the node to the broker, held open.** That is all of it. There
|
||||
is no second connection and nothing is ever dialled *at* a node. Being in the mesh, operationally,
|
||||
means that connection is up; being disconnected means it is not
|
||||
([`09-the-node-lifecycle.md`](../03-DESIGN/01-to-be/09-the-node-lifecycle.md)).
|
||||
|
||||
**Two different things ride on it, and conflating them is what made this confusing:**
|
||||
|
||||
| | what it answers | who issues it |
|
||||
|---|---|---|
|
||||
| **an AMQP account** | may this connection be accepted at all | **the mesh, at enrolment** |
|
||||
| **the node's keypair** | which node is speaking, on every message | **the node**, above |
|
||||
|
||||
**The account is the mesh's to issue**, and per node. The broker has to authenticate somebody
|
||||
before a connection exists, and a shared account would let any node consume another's queue —
|
||||
which is the shared-credential fault this record exists to remove, reappearing at the transport.
|
||||
So enrolment creates that node's account and hands it over, and it is rotatable without touching
|
||||
the node's identity.
|
||||
|
||||
**The keypair is not made redundant by it.** With only an account, the control plane would know
|
||||
which node is speaking *because the broker says so* — and that is the same transitive authority
|
||||
this record refuses in the other direction. A compromised broker could then attribute reports to
|
||||
whichever node it liked, and the control plane would act on them. Signing is what removes the
|
||||
broker from the question in both directions.
|
||||
|
||||
**So a node holds two things after enrolment**: a credential the mesh issued for reaching the
|
||||
broker, and a key it generated itself that the mesh only ever sees the public half of. Both are
|
||||
its own, neither reaches anything else, and *compromise of a node is compromise of that node*
|
||||
still holds.
|
||||
|
||||
### Its own key, not the machine's SSH host key
|
||||
|
||||
Reusing the host key is the obvious economy and it is refused, for reasons that are operational
|
||||
rather than fastidious:
|
||||
|
||||
- **It is regenerated by ordinary events.** A reinstall, an image cloned, `ssh-keygen -A` on a
|
||||
rebuild — each silently un-enrols the node, and the failure appears as an authentication problem
|
||||
with no cause anybody changed.
|
||||
- **It is managed by something else.** Its lifecycle belongs to the machine's SSH daemon, and an
|
||||
identity the mesh depends on should not rotate on a schedule the mesh does not know about.
|
||||
- **Not every node has one.** A partial host has no SSH daemon
|
||||
([ADR 0005](0005-the-node-host.md)), and an identity scheme that excludes a supported kind of
|
||||
node is not one.
|
||||
|
||||
**The mesh should still know the host key** — it knows every node, so it can distribute host keys
|
||||
the same way it distributes authorised keys
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md)), and node-to-node SSH stops depending on
|
||||
trust-on-first-use. That is the good half of the idea, kept.
|
||||
|
||||
**Authority is mutual.** The node proves it may join, and the control plane proves it is the
|
||||
mesh. One-way is not enough: the host applies whatever the link delivers, so a node that cannot
|
||||
tell the mesh from something impersonating it will apply that something's declarations.
|
||||
|
||||
**What may be pushed is bounded by form, not by trust.** Declarations of known shape, never a
|
||||
command to run. Stated honestly, **this bounds form and not impact**: a compromised control plane
|
||||
can declare harmful state and the host will apply it faithfully, because that is what it is for.
|
||||
What the property buys is that the blast radius is *describable* — exactly what the declaration
|
||||
language can express, which can be reviewed. An arbitrary command channel has no such bound.
|
||||
|
||||
## The enrolment token carries the mesh
|
||||
|
||||
Mutual authority needs the node to verify something before it trusts anything, and that is a
|
||||
circle: verifying the mesh needs the mesh's certificate authority, and obtaining one means
|
||||
trusting whoever hands it over. There is a second circle beside it — a node must reach the mesh
|
||||
before the mesh has configured it, so it can resolve no mesh name.
|
||||
|
||||
**Both are the same shape: a node needs a fact about the mesh before it has any trustworthy way
|
||||
to obtain one.** So that fact arrives by a path other than the mesh.
|
||||
|
||||
**The token carries five things**, and it is the only thing a joining node needs:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **who it is** | the name the mesh calls this machine |
|
||||
| **where** | the broker's **address**, not a name — there is no resolution yet, and this is why none is needed |
|
||||
| **what it is connecting to** | the fingerprint of the broker's certificate |
|
||||
| **who it will believe** | the control plane's signing identity |
|
||||
| **the right to join** | a one-time secret, useless once used and useless after it expires |
|
||||
|
||||
*The first row was added 2026-08-30, from raising a mesh end to end for the first time.* It reads
|
||||
like an oversight and is not: **the node cannot work its own name out.** The name is the mesh's,
|
||||
chosen when the record was created, and the broker account the node authenticates as is named
|
||||
after it — so it must be known *before* the mesh can tell the node anything. It is not a secret,
|
||||
and whoever issues the token already has it.
|
||||
|
||||
Without it, enrolment fails at the broker with an empty username and a message about credentials,
|
||||
which points at everything except the cause. **A missing fact that surfaces as an authentication
|
||||
error is worse than one that surfaces as a missing fact.**
|
||||
|
||||
**Carried out of band**, by the person adopting the machine. That is what breaks both circles:
|
||||
its authenticity comes from the channel it travelled, not from anything the node can check
|
||||
afterwards. **Trust on first use, with the first use moved out of band** — the difference between
|
||||
a pin and a guess.
|
||||
|
||||
**The endpoint and the authority are two identities.** A node connects to the broker and takes
|
||||
instruction from the control plane behind it. Pinning only the broker would make the control
|
||||
plane's authority *transitive*, and a compromised broker could then forge declarations — which,
|
||||
since the host applies whatever the link delivers, is the whole machine. So the transport is
|
||||
verified once at connect, and **each declaration is verified by its signature, every time**.
|
||||
|
||||
**What this settles:** the mesh's certificate authority is not a bootstrap concern — it certifies
|
||||
internal names once a node is a member. Nothing needs name resolution before the link. And
|
||||
nothing is placed on disk beforehand except the token, which is the first moment *a node holds
|
||||
only its own identity* becomes true rather than aspirational.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Declarations must be signed**, and the host must tell *this is not from the mesh I joined*
|
||||
apart from *this is malformed*. Rotating the signing identity is a fleet-wide operation with an
|
||||
overlapping rollover, and that is the cost of not trusting the broker.
|
||||
- **The token becomes security-critical**, because it carries the pin. Tampering substitutes the
|
||||
mesh — which is strictly better than the alternative, where there is nothing to tamper with and
|
||||
the node trusts the first answer unconditionally.
|
||||
- **A rejoining node is ordinary.** There is no long-lived secret to recover, so a node that lost
|
||||
its identity gets a new token.
|
||||
@@ -1,155 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 5. The node host
|
||||
|
||||
*Consolidated 2026-08-28 from eight records. Tier 0 is one component and was decided over a
|
||||
week; the reasoning is kept, the fragmentation is not.*
|
||||
|
||||
## It applies; it does not decide
|
||||
|
||||
**The host makes a machine match what it was told, and never works out what that should be.**
|
||||
|
||||
This is the line the whole tier rests on, and it is not about privilege — it is about what a
|
||||
single machine can *know*. Deciding needs knowledge the machine does not have: which nodes should
|
||||
run a store, which peers belong in an overlay, whether a node has been unreachable for a week.
|
||||
Anything needing a second node is the control plane's.
|
||||
|
||||
The practical form: **the host never queries the mesh's database and holds no credential to it.**
|
||||
Two modules in the current mesh do, and they are the reason every node permanently carries a
|
||||
database credential.
|
||||
|
||||
## It depends on nothing that must be installed first
|
||||
|
||||
**A statically linked binary. Copy it onto a machine and run it — that is the whole
|
||||
installation.** Written in Go, because the job is system-level and because a runtime that must be
|
||||
installed first would make the host depend on the thing it exists to install.
|
||||
|
||||
**What it needs from the machine is not a dependency in this sense.** An init is not installed;
|
||||
it is what the machine already is. A package manager is the distribution. Those are what a
|
||||
machine *is*, not what must be put on it before the host works.
|
||||
|
||||
## It is built per operating system
|
||||
|
||||
**`systemd` and `pacman` are the Arch host's implementation, not abstractions the mesh grows.**
|
||||
They are not independent choices: a machine has pacman *because* it is Arch, and the package
|
||||
manager, service manager and packaging format arrive together as one decision somebody made at
|
||||
install time.
|
||||
|
||||
```
|
||||
mesh-host-arch pacman · systemctl · a container runtime
|
||||
mesh-host-alpine apk · rc-service
|
||||
mesh-host-android neither — a partial host
|
||||
```
|
||||
|
||||
**Abstracting them was rejected on correctness, not effort.** The service applier reads systemd's
|
||||
`LoadState` to tell *not installed* apart from *stopped* — which is what stops it reporting
|
||||
absence as success — and OpenRC has no equivalent. An interface spanning both must drop it, and
|
||||
the lowest common denominator is exactly where that fault lives.
|
||||
|
||||
**Almost all of it is shared.** The declaration vocabulary, the store, the apply loop, the
|
||||
read-back discipline, the refusal model and the link are portable. Two appliers differ.
|
||||
|
||||
**A host that cannot implement a shape refuses it.** Android has no package manager it may drive
|
||||
and no init it may register with, so it implements `file`, `directory` and `action` and refuses
|
||||
the rest — the same refusal an unknown type gets, with a different reason. Those three are the
|
||||
portable floor, and they are what makes a partial host a real thing rather than a broken one.
|
||||
|
||||
## It is a root service, and it never manages its own unit
|
||||
|
||||
**Root**, because no useful part of the job is unprivileged: it writes under `/etc`, installs
|
||||
packages, manages units and runs containers.
|
||||
|
||||
**It cannot run in a container**, and the reason is decisive rather than stylistic: installing the
|
||||
container runtime is a step of the bootstrap, so a host inside a container would need the thing
|
||||
it exists to install. Everything above tier 0 is a container; the host is not. That split is the
|
||||
tier boundary made concrete.
|
||||
|
||||
**The installation owns the host; the host owns everything else.** It manages `service` resources
|
||||
and its own unit is one — the temptation is obvious and it ends with a host stopping itself half
|
||||
way through an apply, leaving a machine with nothing running to fix it.
|
||||
|
||||
## An init is asked for one thing
|
||||
|
||||
**Start this at boot.** That is all, and every init can express it — systemd, OpenRC, runit, s6.
|
||||
|
||||
**Everything else is a launcher the host ships**, which supervises it: restart it when it exits,
|
||||
count consecutive failures, roll back after too many, halt after that. Policy in a unit file can
|
||||
only be read and hoped for; a script with a counter can be tested, and this is the one piece that
|
||||
must work on a machine where the host does not.
|
||||
|
||||
**The launcher does not exec the host, it supervises it** — so restarting is ours rather than the
|
||||
init's. The cost is signals: a supervisor that exits while its child runs leaves the host to be
|
||||
*killed* rather than to *stop*, and an apply interrupted that way is the half-configured machine
|
||||
this design is about. So it traps the shutdown signal, passes it down, and waits.
|
||||
|
||||
**A clean exit is the upgrade path**, and it is the easiest thing to get wrong — twice now. The
|
||||
host stands aside for a new binary by exiting zero, so anything supervising must restart on a
|
||||
zero exit and must not count it as a failure.
|
||||
|
||||
**Recovery is local, and detection is the mesh's.** Nothing dials a node and a host that cannot
|
||||
start cannot report, so the node must recover itself. But a local supervisor sees one process
|
||||
failing and cannot tell a broken machine from a broken release — only something watching every
|
||||
node can, which is why a host rollout is staged and stops when nodes go quiet.
|
||||
|
||||
## A host may be episodic
|
||||
|
||||
**Resident or episodic, and both are hosts.** A phone has no init to register with and nothing
|
||||
worth supervising, because a supervisor would be killed alongside what it supervises. So it runs
|
||||
when the platform allows and is killed when the platform wants the memory — **and that is
|
||||
disconnection**, which is already an ordinary situation.
|
||||
|
||||
It needs no keep-alive and no new mechanism: the store is already authoritative while
|
||||
disconnected, reconcile already happens on start, and *last heard from* is already reported
|
||||
rather than alarmed on. An episodic host cannot be the first node, because every bootstrap step
|
||||
is a shape it refuses.
|
||||
|
||||
## What a declaration is
|
||||
|
||||
**An ordered list of resources the host owns.** JSON, because Go's standard library carries a
|
||||
JSON parser and no YAML, and the one binary whose argument is that it needs nothing must not
|
||||
gain a parser to buy authoring comfort in a machine-written document.
|
||||
|
||||
**Ordered, because ordering is a decision.** The host does not sort and does not resolve
|
||||
dependencies — that would be deciding, and deciding the thing most likely to differ between what
|
||||
the control plane intended and what the machine does.
|
||||
|
||||
**Every resource has a stable identity** — a name the control plane keeps across declarations, not
|
||||
a position and not a hash of content. It is what lets the store say *this is the same resource I
|
||||
applied last time*, which is what makes removal possible at all.
|
||||
|
||||
**Unknown is refused, whole.** A field the host does not know is something the control plane
|
||||
believes it asked for. A declaration naming one is rejected entirely, naming every problem at
|
||||
once — a host that applied the parts it understood would leave a machine that looks configured
|
||||
and is not.
|
||||
|
||||
**Six shapes:** `file`, `directory`, `service`, `package`, `container`, `action`. Every addition
|
||||
widens what a compromised control plane can express, so the list is a security artefact and grows
|
||||
deliberately.
|
||||
|
||||
### The bundle may carry actions; the link may not
|
||||
|
||||
An `action` runs a command, and the host never learns what it means. It is needed because the
|
||||
bootstrap creates a database before there is any mesh to ask for one, and the host must not learn
|
||||
what a database is.
|
||||
|
||||
**Permitted from the bundle, refused from the link**, and the asymmetry is the whole point: a
|
||||
bundle arrives *with* the binary, so anyone able to put a hostile action there could have put it
|
||||
in the host itself — refusing it buys nothing and costs the bootstrap. The link is a separate
|
||||
party, reachable separately, and an action there is an unbounded blast radius.
|
||||
|
||||
**An action must carry its own verification**, which is also its idempotency check. The host does
|
||||
not know what a database is, so *is it already there* is a question only the declaration can ask.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The migration is smaller than it looks.** A joining node never needs mesh-wide state — it
|
||||
needs an identity, an address and one peer, and the rest arrives as declarations.
|
||||
- **What is applied is recorded after it works, never before.** A failed apply leaves the machine
|
||||
in whatever state it reached, and nothing must claim otherwise.
|
||||
- **A second operating system is additive**: two appliers and a four-line init file.
|
||||
@@ -1,284 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 6. The substrate and the control plane
|
||||
|
||||
> **Superseded in part — 2026-10-07, by [ADR 0244](0244-the-mesh-is-described-in-domains-and-one-word-names-one-thing.md).**
|
||||
> The seven contexts below no longer describe the mesh: it is described in ten domains, and "domain" replaces
|
||||
> "context" in this sense; *observability* becomes Health and repair. This record keeps its word, and the
|
||||
> glossary says how to read it. Everything else decided here stands.
|
||||
|
||||
*Consolidated 2026-08-28 from six records. Extended 2026-08-29, by building it: the language, and
|
||||
what must be running before the control plane starts — which this record had left not established
|
||||
and could not have settled the way it was asking.*
|
||||
|
||||
## The control plane is what needs to know about more than one node
|
||||
|
||||
That is the whole test, and it follows from the host applying rather than deciding: **deciding
|
||||
needs knowledge a single machine does not have.**
|
||||
|
||||
| question | whose |
|
||||
|---|---|
|
||||
| write this file, with this content, with this mode | the **host** |
|
||||
| which nodes should run the store | the **control plane** |
|
||||
| is this unit running | the **host** |
|
||||
| which peers belong in this node's overlay | the **control plane** |
|
||||
| has this node been unreachable for a week | the **control plane** — nobody else is watching |
|
||||
|
||||
**Anything a single machine could answer alone is not the control plane's.**
|
||||
|
||||
### Seven contexts and one interface
|
||||
|
||||
**inventory, config, connectivity, provisioning, delivery, observability, identity** — plus
|
||||
`api`, the one interface every surface speaks to. Each earns its place by the test above rather
|
||||
than by being ours.
|
||||
|
||||
**`work`, `knowledge` and `stream` are mesh-hosted applications, not control plane.** A task does
|
||||
not need to know a node exists. *Being ours does not make something infrastructure.*
|
||||
|
||||
**`identity` owns SSH access.** *Written 2026-08-29, on noticing it was assumed everywhere and
|
||||
stated nowhere.* SSH appears three times across this design and every time as something that
|
||||
*uses* the overlay — "the way back in", "every node reaches every other: SSH, services, ordinary
|
||||
traffic" — while nothing said who hands out the keys. Nobody else could: the mesh is the only
|
||||
thing that knows which humans and agents exist and which nodes they may reach, which is
|
||||
`identity`'s definition. The node end already works, since an `authorized_keys` file is a file.
|
||||
|
||||
It is three questions wearing one name, and only two of them are the mesh's:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **humans** | their key, on the nodes they are allowed on |
|
||||
| **agents** | the same, with a lifetime — and revocation that has to actually work |
|
||||
| **an agent reaching another node** | **this is the point of the mesh, not an exception to it** — see below |
|
||||
|
||||
**There is no such thing as node-to-node SSH here, and that is a clarification rather than a
|
||||
restriction.** The actor is always an **agent**; a node is only where it happens to be running —
|
||||
[ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md): *a node is a place where an agent can
|
||||
run, that is the entire relationship.* An agent hired onto one node reaching another to do work is
|
||||
the capability the whole arrangement exists to provide.
|
||||
|
||||
**The credential is the agent's, never the node's.** It lives in the agent's own credential
|
||||
directory ([ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)), so a node's
|
||||
`authorized_keys` lists **agents** and never nodes. Three things follow, and they are why this
|
||||
shape is better rather than merely allowed:
|
||||
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md)'s *a node holds its own identity and nothing else*
|
||||
stays true — no node holds a key that reaches another node;
|
||||
- a compromised node costs the credentials of the agents that were on it, not a way into
|
||||
everything;
|
||||
- **who may reach what is a mesh-wide fact**, which is exactly why it is `identity`'s and not
|
||||
something arranged locally.
|
||||
|
||||
**And it does not conflict with the host having no inbound control surface.** That rule is about
|
||||
how a node's *declared state* changes: over the broker, never by being dialled. An agent with a
|
||||
shell is not the mesh reconfiguring a machine — it is what a person with a terminal has always
|
||||
been, and this design already depends on it working
|
||||
([ADR 0007](0007-connectivity.md): the overlay is *the way back in*). What such a session can leave
|
||||
behind is drift, and drift is what reconciliation is for.
|
||||
|
||||
**Where the record lives is deliberately open.** Contexts integrate through it, which makes it
|
||||
load-bearing, and putting it in the substrate risks recreating the circularity the tiers just
|
||||
removed. Listing it as an eighth context would settle by naming what has not been settled by
|
||||
arguing.
|
||||
|
||||
### One node runs it, and nothing takes over
|
||||
|
||||
**Declared, never elected.** No promotion, no quorum, no fencing, no split brain — none of it
|
||||
built, so none of it can be subtly wrong.
|
||||
|
||||
#### The option that would make it a real mesh, and why not
|
||||
|
||||
*Written 2026-08-29. It had been rejected by never being written down, which is the weakest way
|
||||
to reject anything.*
|
||||
|
||||
A genuine peer-to-peer mesh means **no node is special**, and that has a concrete price:
|
||||
|
||||
- every node holds the **whole inventory**, so there is a replication process between them;
|
||||
- replication needs a writer, so one node is elected **master**, and something promotes a new one
|
||||
when it drops — Redis Sentinel and its whole family of problems;
|
||||
- and it still would not deliver what the name promises, because **application databases are not
|
||||
replicated.** A workload's store lives where the workload lives.
|
||||
|
||||
That last point is the one that settles it. To make the mesh genuinely peer-to-peer we would have
|
||||
to become **a replicated database system for everything running on it** — not for our own
|
||||
inventory, for every consumer's data too. That is a product, and a much larger one than the thing
|
||||
it would be supporting.
|
||||
|
||||
**So there are three central roles, not one**, and it is worth seeing them separately because
|
||||
only the third costs operation:
|
||||
|
||||
| | its loss costs |
|
||||
|---|---|
|
||||
| **the control plane** | nothing can be *changed*. Nothing stops running |
|
||||
| **the broker** | nothing can be told anything, or report anything |
|
||||
| **the hub** | nodes in different places **cannot reach each other** ([ADR 0007](0007-connectivity.md)) |
|
||||
|
||||
**Whether these are one node is not decided here.** All three must be dialable by every node, which
|
||||
pushes toward one; nothing says they must be.
|
||||
|
||||
**That is sound rather than merely cheap**, because the design already tolerates its absence by
|
||||
construction: a node reconciles from its own store and never needed to ask anybody to hold the
|
||||
state it was last given. **The control plane being down is not a new failure mode — it is every
|
||||
node in the ordinary disconnected situation at once.** What is lost is *change*, not *operation*.
|
||||
|
||||
The honest half: this node is a single point of failure, recovery is **restore rather than
|
||||
failover** — which makes backup the availability mechanism rather than hygiene — and
|
||||
**certificate renewal is the clock.** An outage outlasting a renewal window expires every public
|
||||
name, which turns an inconvenience into an outage on a timer. Nothing measures that today.
|
||||
|
||||
## The authority is the control plane, not a database
|
||||
|
||||
**There is no single mesh database.** Each context owns its store exclusively, and *the mesh
|
||||
database* names a thing that will not exist.
|
||||
|
||||
**No node reads any of them** — not for writes, not for reads. A node is *told* what to own, over
|
||||
the link, in a bounded vocabulary; it **states** what it applied, and the owning context writes.
|
||||
The difference is the security boundary: something that can write cannot be prevented from
|
||||
writing anything.
|
||||
|
||||
**A node runs from its own store always, not as a fallback.** The current arrangement's nastiest
|
||||
property is that *a node running from cache looks identical to a node running from the database*,
|
||||
with no age on the cache and nothing reporting divergence. Under this there is no second mode to
|
||||
be mistaken for the first.
|
||||
|
||||
**What survives from the original decision:** the repository defines what exists, the mesh defines
|
||||
what runs where, and no node-to-module mapping is ever committed. That is what makes the
|
||||
repositories node-agnostic and why anything about the mesh can be published at all.
|
||||
|
||||
**The error underneath was a category error**: *source of truth* named a storage location when it
|
||||
meant an **authority**. Once the store is the answer, *which database* becomes the question, and
|
||||
shared schemas follow.
|
||||
|
||||
## The substrate is what the control plane consumes and cannot grant itself
|
||||
|
||||
Every module needing a database asks provisioning for one. The control plane needs a database too
|
||||
and cannot ask itself, because it is not running yet. **That circularity is the definition**, and
|
||||
anything on the wrong side of it is raised from the bundle the host carries.
|
||||
|
||||
| role | product | |
|
||||
|---|---|---|
|
||||
| relational store | **PostgreSQL** | its own state lives there |
|
||||
| message bus | **LavinMQ** | it cannot grant itself a virtual host — and **precedes it**, below |
|
||||
| object store | **MinIO** | it cannot grant itself a bucket |
|
||||
| image registry | **an OCI registry** | it cannot grant itself a repository |
|
||||
| identity provider | — | **conditional**: substrate only if the control plane delegates authentication, which is undecided |
|
||||
|
||||
**The role and the product are both written.** The role is what the argument turns on; the product
|
||||
is what gets installed and pinned, and a design that names only the role does not record that the
|
||||
choice was made. **The dependency is on the protocol** — AMQP, S3, OCI — which is what keeps
|
||||
naming them safe. The store is the exception: the provisioning model uses databases, roles and
|
||||
schemas as PostgreSQL means them.
|
||||
|
||||
**A container runtime is detected, not chosen** — docker or podman, because a machine that
|
||||
already has one keeps it. Only the version probe differs between them; the behavioural difference
|
||||
(podman has no daemon, so containers do not return after a reboot unless a unit is enabled)
|
||||
belongs in the declaration rather than the host.
|
||||
|
||||
**Being substrate and being in the bundle are different questions.** PostgreSQL and LavinMQ must
|
||||
precede the control plane; the object store and the registry are substrate by role and ordinary by
|
||||
delivery, provisioned once there is a control plane to do it.
|
||||
|
||||
### Why the broker precedes it too
|
||||
|
||||
Written after the fact, because this record first left it *not established* and framed it as
|
||||
turning on whether the control plane's own contexts talk to each other over the bus.
|
||||
|
||||
**They do not** — they are one process and dispatch internally. Under that framing the broker is
|
||||
provisioned like anything else and the bundle stays at one image.
|
||||
|
||||
**The framing cannot answer the question.** What decides it is not how the contexts reach each
|
||||
other. It is how the control plane reaches a *node* — and that is settled above: never except over
|
||||
the link, and the link is AMQP ([ADR 0002](0002-nodes-communicate-over-a-broker.md),
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md)). So:
|
||||
|
||||
```
|
||||
the bundle raises the control plane
|
||||
the control plane provisions the broker ← by telling a host to run it
|
||||
telling a host happens over the link
|
||||
the link is the broker
|
||||
```
|
||||
|
||||
**And it is not avoided by the first node being local.** `enrol` dials the broker at the address in
|
||||
its token, which is the first node's own third step. A machine that raised the mesh still joins it
|
||||
the ordinary way, and that was deliberate — its specialness lasts two commands. Making it join by
|
||||
some other route would buy a smaller bundle by giving up the property the design was built to have.
|
||||
|
||||
The broker precedes the control plane for the same reason PostgreSQL does: **the control plane
|
||||
cannot grant itself the thing it would need in order to grant it.**
|
||||
|
||||
**What it costs.** Two images rather than one, against the wish above to keep the bundle at roughly
|
||||
one so a person can read it — two is still readable, four would not be. And three things that are
|
||||
not images, each an action the bundle declares and the host runs, the way the database already is:
|
||||
a virtual host, a credential on it, and **a certificate**. That last is the awkward one: a token
|
||||
pins the fingerprint a host must expect *before it sends anything*, so the broker needs a
|
||||
certificate at a moment when there is no mesh to issue one and no public name to obtain one for.
|
||||
Self-signed and pinned is the shape that fits; how it is later replaced by the certificates in
|
||||
[ADR 0007](0007-connectivity.md) is not decided here.
|
||||
|
||||
## The control plane is written in Go
|
||||
|
||||
The same language as the host, so tiers 0 and 2 are one language and not two.
|
||||
|
||||
The reason that decides it is not familiarity. **Its image is pinned by digest in the bundle**,
|
||||
which means it is fetched and run on a machine where no mesh exists yet — nothing to check it
|
||||
against, nothing watching, and a person expected to have read the bundle and believed it. A
|
||||
statically linked binary makes that image the program and nothing else: no interpreter, no package
|
||||
tree, no transitive dependency that arrived because something needed a date library. Everything
|
||||
under that line is something somebody would have to audit, on the one image the whole mesh is
|
||||
raised from.
|
||||
|
||||
A second reason, smaller and still real: the control plane runs a reconcile loop of its own
|
||||
([ADR 0010](0010-delivery.md) — artifacts against source, as the host reconciles machine state
|
||||
against declarations). Two loops of the same shape are cheaper to hold in one head when they are
|
||||
also the same language.
|
||||
|
||||
**The option rejected** is TypeScript, matching the lab and the surfaces that will speak to this.
|
||||
The argument for it is that tier 3 is web and CLI, so a TypeScript control plane would share types
|
||||
with its callers rather than generating a contract. True, and it does not reach far enough:
|
||||
`mesh-sdk` is *contracts shared across tiers* and **tier 0 is Go**, so the contracts cross a
|
||||
language boundary whatever tier 2 is written in. The choice is between generating them for one
|
||||
consumer or for two.
|
||||
|
||||
**What it costs, plainly:** the control plane can import nothing that exists today, and a person
|
||||
moving between tier 2 and tier 3 changes language. Neither is recovered later — the language is the
|
||||
most expensive thing in this record to reverse.
|
||||
|
||||
## The installer fetches what it pins
|
||||
|
||||
`substrate.lock` carries **references, not payload** — an image name and a **digest**, fetched at
|
||||
apply time. A tag moves; a digest does not, and reproducibility comes from pinning the identity of
|
||||
a thing rather than carrying its bytes.
|
||||
|
||||
The assumption that a machine might have no network came from the lab and was wrong: a machine
|
||||
being adopted has one, and the sealed case is the lab.
|
||||
|
||||
**The lab places images by raising a registry inside the scenario**, which is what a real node
|
||||
pulls from anyway — so it tests the real path rather than a stand-in for it. The digests that
|
||||
registry serves are its own, and that satisfies this rule: what is required is a reference that
|
||||
is **exact and cannot move**, and one it assigned is both. Assuming an upstream digest had to be
|
||||
preserved is what made this look impossible for a while
|
||||
([04-ISSUES/009](../04-ISSUES/009-a-digest-pinned-image-cannot-be-placed-in-the-lab/00-report.md)).
|
||||
|
||||
**Its contents are per operating system** even though its mechanism is not — package names, unit
|
||||
names and service names all differ, so an Arch host embeds an Arch bundle.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The bundle stays small and reviewable.** A list of pinned references is something a person can
|
||||
read; a bundle containing images is not.
|
||||
- **An apply can fail because something is unreachable**, which a self-contained artifact could
|
||||
not. That must fail *legibly*, naming what could not be fetched and from where.
|
||||
- **Cross-context reporting is harder, and that is the point.** Anything wanting to see across
|
||||
contexts consumes their events or calls their interfaces.
|
||||
- **A queue with no limit grows until the broker's disk is full**, and the broker is what every
|
||||
node depends on. The bound is per queue and is not decided.
|
||||
- **The bundle carries two images and four actions**, and the substrate bootstrap grows a step.
|
||||
- **Nothing in the first node's path is special-cased.** Enrolment is walked on node one.
|
||||
- **The broker's certificate at bootstrap has no answer yet**, and is named as unfinished rather
|
||||
than assumed. It is the first thing that will be wanted when the link is built.
|
||||
- **The language cannot be revisited cheaply.** It is the one line here close to irreversible.
|
||||
@@ -1,191 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 7. Connectivity
|
||||
|
||||
*Consolidated 2026-08-28 from three records. Overlay, resolution, exposure, filtering and
|
||||
certificates are one design.*
|
||||
|
||||
## Why it is control-plane work
|
||||
|
||||
Apply the test — *everything that needs to know about more than one node* — and not one of the
|
||||
five can be answered by a machine on its own:
|
||||
|
||||
| | needs to know |
|
||||
|---|---|
|
||||
| **overlay** — who peers with whom | every node, and which can be dialled |
|
||||
| **resolution** — which name is which node | every node |
|
||||
| **exposure** — which public name reaches which container | which node is publicly reachable |
|
||||
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape |
|
||||
| **certificates** — who may present which name | which name belongs to which node |
|
||||
|
||||
That is exactly what the current arrangement gets wrong, by computing all five on the node from a
|
||||
direct database connection. Two modules do this, and they are the only two left holding a
|
||||
credential to the control plane's database.
|
||||
|
||||
**The shape of the fix, once for all five:** the connectivity context computes the configuration;
|
||||
it arrives over the link as `file` resources; the service reads files and knows nothing about the
|
||||
mesh. **This costs no new host vocabulary.**
|
||||
|
||||
## A route is a grant
|
||||
|
||||
**Ingress is not substrate.** The control plane does not need a route to start — it listens
|
||||
locally — and no node needs one to reach it, because the node dials out and has no listening
|
||||
control surface. It grants itself a route afterwards, the way it grants itself a bucket.
|
||||
|
||||
The strongest objection deserves stating: the `api` is the one interface every surface speaks to,
|
||||
so eventually it *does* want a public name. But **wanting one later is not needing one to
|
||||
start**, and that distinction is the entire substrate test.
|
||||
|
||||
**A module that must be reachable declares it needs a route; the proxy provides one.** Ordinary
|
||||
instantiation, with the direction mirrored — the consumer supplies a target and receives a name.
|
||||
|
||||
**Exposure is three facts at two scopes**, which is why it cannot live on the node:
|
||||
|
||||
| the fact | scope |
|
||||
|---|---|
|
||||
| the public name resolves to an address | **mesh** — which node is publicly reachable |
|
||||
| a certificate valid for that name exists | **mesh** — issued once, used on one node |
|
||||
| the proxy maps that name to that container | **node** |
|
||||
|
||||
**A node without a public address is proxied by one that has**, across the overlay. Most nodes sit
|
||||
behind a connection with no forwarded port, so exposure cannot assume the workload's node is
|
||||
reachable.
|
||||
|
||||
## Reachability is declared, not inferred
|
||||
|
||||
The overlay's peer graph is computed from whether a node can be dialled, and that was inferred
|
||||
from a regular expression over the address. **The address is evidence of reachability; it is not
|
||||
the fact**, and the gap has already cost:
|
||||
|
||||
| address | the regex says | actually |
|
||||
|---|---|---|
|
||||
| `100.64.0.0/10` — carrier-grade NAT | **public** | **not reachable.** An endpoint is written to an address nothing can reach |
|
||||
| any IPv6 address | public | the test is v4 shapes only |
|
||||
| a routable address behind a closed firewall | public | not reachable |
|
||||
| a documentation range standing in for a public segment | private | reachable — this is the lab bug |
|
||||
|
||||
**A test environment having to choose its addresses to satisfy a regex is the regex telling us it
|
||||
is not a fact.**
|
||||
|
||||
So: **an endpoint, or none** — declared. And **the hub is declared, never derived from an address
|
||||
prefix**, because an election decided by the first four characters of an address fails silently,
|
||||
cannot be queried, and makes a renumbering an outage.
|
||||
|
||||
**The address remains evidence and stops being the fact.** Where an observed endpoint disagrees
|
||||
with a declared one, the disagreement is a **reportable condition**, not a silent correction.
|
||||
|
||||
**What does not change** is the lesson underneath: role does not imply reachability — a
|
||||
home-hosted node is a server that cannot be dialled. This keeps that and stops encoding it as a
|
||||
pattern match.
|
||||
|
||||
### Some nodes must be reachable, and this had not been said
|
||||
|
||||
*Written 2026-08-29, on being asked and finding no answer.*
|
||||
|
||||
Everything above treats reachability as a **fact to record** — which node can be dialled, so that
|
||||
exposure and certificate issuance can be placed. It never said the converse, and the converse is a
|
||||
hard requirement:
|
||||
|
||||
| role | dialled by | so it needs |
|
||||
|---|---|---|
|
||||
| **the node running the broker** | every node, outbound ([ADR 0002](0002-nodes-communicate-over-a-broker.md)) | to be reachable from wherever nodes are, at a **stable address** |
|
||||
| **the hub** | every node not co-located with its peer | the same |
|
||||
|
||||
**Across the internet, "reachable from wherever nodes are" means publicly reachable.** For a mesh
|
||||
confined to one network it does not — the requirement is about the nodes that exist, not about the
|
||||
public internet.
|
||||
|
||||
**Stable is the sharper half.** A token carries the broker's *address, not a name*, because there
|
||||
is no resolution before joining ([ADR 0004](0004-a-node-and-how-it-joins.md)). A broker node whose
|
||||
address moves invalidates every token issued for it, and a node that was disconnected across the
|
||||
change cannot get back.
|
||||
|
||||
**A mesh whose nodes are all behind NAT cannot be raised.** That is a real precondition and it
|
||||
belongs with the others rather than being discovered.
|
||||
|
||||
### The link stays on the underlay, and that is a repair channel
|
||||
|
||||
The obvious objection is that all traffic should run over the overlay. Nearly all of it does — SSH,
|
||||
services, node to node — and the exception is each node's own outbound link to the broker.
|
||||
|
||||
**At join time it is forced**: a node has no overlay yet, so it cannot use one to ask for one.
|
||||
**Afterwards it is a choice**, and the reason is that the link is how a broken node is fixed. A
|
||||
repair channel carried over the thing being repaired is not a repair channel: a node whose only
|
||||
path home was the overlay is gone the moment an overlay declaration is wrong.
|
||||
|
||||
**What it does not cost is confidentiality.** The link is already authenticated and encrypted
|
||||
against a pinned fingerprint ([ADR 0004](0004-a-node-and-how-it-joins.md)), so moving it onto the
|
||||
overlay would not protect traffic that is unprotected today.
|
||||
|
||||
## A filter rule names its source
|
||||
|
||||
`scope: public` is declared in five manifests, is part of no rule type, and is **referenced by no
|
||||
code**. So five manifests appear to restrict a port and restrict nothing — on the modules most
|
||||
worth restricting.
|
||||
|
||||
**A rule names its source. `from:` is the only way to scope one, and a rule without one is open**
|
||||
— which it must say plainly rather than appear to deny.
|
||||
|
||||
**`scope:` is removed rather than implemented**, because giving it meaning would leave two ways to
|
||||
express one thing. And the general fix is that **an unknown key is refused**: the host's
|
||||
declaration parser already works this way, and manifests are the layer where that discipline is
|
||||
missing. `scope:` survived because nothing rejected it, and it spread by copying to five
|
||||
manifests.
|
||||
|
||||
## Order, and what it costs
|
||||
|
||||
**The link runs on the underlay and never on the overlay.** The overlay is configured by the mesh,
|
||||
so a link requiring it could never be established on a new node.
|
||||
|
||||
**The first declaration is the overlay and nothing else** — because a node's address and peers are
|
||||
*assigned* so it cannot come earlier, and because it is the way back in. A node reachable over the
|
||||
overlay can be fixed by hand if a later declaration breaks the machine; **a large first
|
||||
declaration risks a node that is broken and unreachable at once.**
|
||||
|
||||
**Reachable is not the same as having a control surface.** Every node reaches every other over the
|
||||
overlay — SSH, services, ordinary traffic — and every node consumes from the broker. What is
|
||||
forbidden is a listening thing that accepts instructions and changes the machine.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The last two direct database connections leave the nodes**, and with them the database
|
||||
credential every node carries.
|
||||
- **The `/etc/hosts` floor goes**, along with the bootstrap circularity it patched.
|
||||
- **Two certificate authorities stay separate on purpose**: a public one for public names, the
|
||||
mesh's own for internal ones. A single-CA lab would hide any bug living in the split.
|
||||
- **What happens when the hub is down**: nothing takes over. Non-co-located paths stop; co-located
|
||||
peers and every assigned workload keep running.
|
||||
|
||||
## Open — the link over the overlay, with a fallback
|
||||
|
||||
*Raised 2026-08-29 and deliberately left open, because the honest gain is smaller than it looks
|
||||
and it is a decision rather than a derivation.*
|
||||
|
||||
The proposal: a node prefers the overlay for its link and drops to the underlay when the overlay
|
||||
is not working — so ordinary operation is private and the underlay stays as the way back.
|
||||
|
||||
**Two things it would have to get right:**
|
||||
|
||||
- **The trigger cannot be "is the overlay up".** A WireGuard interface has no link state; once
|
||||
configured it is up whether or not the far end exists. So there is no flag to read, and failing
|
||||
over means *try, fail, time out, retry elsewhere*.
|
||||
- **Running on the fallback has to be visible.** A node that quietly drops to the underlay is a
|
||||
node whose overlay is broken with nothing to say so, and it will stay broken because everything
|
||||
still works. That is this repository's recurring fault — a failure that reads as success — and a
|
||||
fallback is the easiest place in the design to reintroduce it.
|
||||
|
||||
**What stops it being an obvious win:** if the underlay path must stay available for the fallback,
|
||||
the broker stays exposed on it. So the exposure is unchanged and the traffic was already encrypted
|
||||
— the gain is which network carries bytes, not what an attacker can reach.
|
||||
|
||||
**The version that would buy something is overlay-only**, with the broker firewalled to the overlay
|
||||
in steady state, accepting that a node whose overlay breaks needs hands-on recovery. That is a real
|
||||
trade: it exchanges the automatic way back in for a closed port.
|
||||
|
||||
Not decided either way here.
|
||||
@@ -1,114 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-26
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 8. A context owns its store, exclusively
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0009](0009-modules-and-the-graph.md) settles what a module
|
||||
declares. This settles what a grant may be, and it is the half that **removes** things.
|
||||
|
||||
`how-we-build` §4 already says *contexts integrate through the record, never through a shared
|
||||
schema*, and states the cost: several domains share one forty-five-table schema, which is why
|
||||
work belonging to one context keeps having to be implemented in another.
|
||||
|
||||
That was written as a principle. Counted, it is thirteen foreign tables belonging to three
|
||||
separate contexts, living in the mesh's own registry database.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **A schema per consumer inside a shared database.** Namespaced, revocable by dropping the
|
||||
schema, with a cross-context join possible but deliberate. Rejected: it keeps the letter of
|
||||
§4 and leaves the temptation in place, and a boundary that is merely inconvenient to cross
|
||||
gets crossed.
|
||||
2. **Read-only roles on another context's store.** Rejected for the same reason and one worse:
|
||||
reading another context's tables couples you to its layout exactly as firmly as writing them,
|
||||
and the coupling is invisible until the owner changes a column.
|
||||
3. **Exclusive ownership.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
> **A context is granted only what it exclusively owns.**
|
||||
|
||||
No shared writes. No read-only role on another context's store. If you need what another context
|
||||
holds, you ask it or you subscribe to it.
|
||||
|
||||
**The unit is the context, not the process.** Everything inside a context — its service, its
|
||||
surface, its tools — reads its own store freely. A board showing the mesh's own nodes and
|
||||
modules is the mesh showing its own data, not a boundary crossing. What is forbidden is a
|
||||
*different* context reading it.
|
||||
|
||||
### Asking or subscribing is derived, not chosen
|
||||
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md) makes disconnection an ordinary situation. So:
|
||||
|
||||
- **Anything that must keep working while disconnected cannot ask** — there is nobody to ask. It
|
||||
keeps a local copy, which means subscribing.
|
||||
- **Anything where a stale answer is worse than none cannot subscribe.** A display may lag; a
|
||||
decision about whether a grant is still valid may not.
|
||||
|
||||
Neither is a query against another store, whatever transport it travels over.
|
||||
|
||||
### What that is, concretely
|
||||
|
||||
*Written 2026-08-29, on building the first one — the rule above was clear and what to type was not.*
|
||||
|
||||
**One PostgreSQL database per context, named for the context.** A separate database rather than a
|
||||
separate schema is the whole point: a cross-schema join is a qualified name away, and a
|
||||
cross-database join needs a foreign data wrapper somebody has to install and explain.
|
||||
|
||||
**And one credential per context, held only by it.** There is no mesh-wide connection setting and
|
||||
no way to ask for one, so reaching another context's store is not a matter of restraint — a process
|
||||
has no address for it and nothing to present. That is also how this rule is *checked*: what a
|
||||
context can reach is the list of variables the declaration running it grants, and it is read there
|
||||
rather than audited in code.
|
||||
|
||||
**Contexts that do not exist yet do not get a database.** The bootstrap creates the ones there are.
|
||||
|
||||
**How a context added later gets its database is open**, and it is a real question: by then there is
|
||||
a control plane, but a control plane holding a credential that can create databases is holding
|
||||
rather more than the thing it exclusively owns.
|
||||
|
||||
## What this removes
|
||||
|
||||
The first clear list of what the design deletes rather than adds:
|
||||
|
||||
- **Grant kinds.** There is one: an exclusive resource. No schema grants, no read roles, no
|
||||
rules about who may see what inside a shared thing.
|
||||
- **The question of who owns which table**, and the guessing at revocation time. Removing a
|
||||
consumer drops what it was granted, whole.
|
||||
- **Cross-context migration ordering.** Two contexts migrating one database must be ordered
|
||||
against each other. Exclusive ownership means a context's migrations are ordered only against
|
||||
itself.
|
||||
- **A class of permission modelling** a shared store would otherwise need.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Cross-context reporting is harder, and that is the point.** Anything wanting to see across
|
||||
contexts consumes their events or calls their interfaces. That is §4's argument, and the cost
|
||||
it names is the one already paid.
|
||||
- **A single surface over several contexts still works** — that is what a surface is. It reads
|
||||
interfaces, not stores. This holds while the contexts sit behind **one** interface; splitting
|
||||
a context into its own deployable costs that, and the composition would have nowhere to live
|
||||
that tier 3 permits. **A real constraint on how far the control plane may be split.**
|
||||
- **Three contexts must move out of the registry database**, taking thirteen tables with them.
|
||||
Their dependency on the registry then shrinks to almost nothing — one of them needs a single
|
||||
table.
|
||||
- **The node appliers were already handled.** [ADR 0005](0005-the-node-host.md)
|
||||
stopped the host querying the mesh database for tier reasons unrelated to this, and it removes
|
||||
most of the remaining direct readers as a side effect.
|
||||
- **What a consumer does about events missed while disconnected is not decided** — replay from a
|
||||
point, ask once and resume, or rebuild. The question every projection has.
|
||||
|
||||
## References
|
||||
|
||||
- [`how-we-build.md`](../00-META/how-we-build.md) §4 — the rule this makes enforceable.
|
||||
- [Research 011](../01-RESEARCH/011-the-module-graph/worked-provider.md) — the count, the worked
|
||||
provider, and the dashboard case.
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — why asking or subscribing is derived.
|
||||
@@ -1,451 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 9. Modules and the graph
|
||||
|
||||
> **Superseded in part — 2026-10-06, by [ADR 0226](0226-the-private-network-is-assigned-by-its-own-name-and-the-proxy-names-its-public-issuer.md).**
|
||||
> The section *what a domain module turns out to be* no longer describes what runs: `networking` is retired,
|
||||
> every machine is assigned the private network's own module, and the catalogue holds no domain module. The
|
||||
> mechanism, a module with requirements and no files, and the rest of this record stand as decided here.
|
||||
|
||||
*Consolidated 2026-08-28 from six records.*
|
||||
|
||||
## Everything is a module
|
||||
|
||||
One kind of thing, one manifest describing all of them. A database, a web application, a window
|
||||
manager and a firewall rule set are all modules — not because they are alike, but because
|
||||
**anything else means a second kind of thing with its own rules, and then a third.**
|
||||
|
||||
**A module is the unit of delivery**: assignable to a node, versionable, replaceable on its own.
|
||||
|
||||
## There are no domain modules
|
||||
|
||||
An earlier decision grouped modules by domain — four things constituting *how a node is
|
||||
reachable* becoming one `networking` module. **That was wrong, and the correction is worth
|
||||
keeping** because the observation behind it was right.
|
||||
|
||||
The measurement holds: reachability is the **only** place in the catalogue where modules
|
||||
genuinely change together under one intent. What did not hold is the conclusion. Tight coupling
|
||||
means they share an **authority** — one place that decides for all of them — and not that they
|
||||
should be one artifact. `wireguard` and the proxy are deployed to different sets of nodes, so a
|
||||
module containing both would be assigned where half of it is unwanted.
|
||||
|
||||
> **Coherence is a context. Delivery is a module.**
|
||||
|
||||
**Folders assert relationships; edges record them.** What grouping was for — finding things,
|
||||
seeing what belongs together — is a tag and a query, neither of which anybody has to keep true by
|
||||
hand.
|
||||
|
||||
### What a domain module turns out to be, and why it is not the one refused above
|
||||
|
||||
> **Superseded in part — 2026-10-06, by [ADR 0226](0226-the-private-network-is-assigned-by-its-own-name-and-the-proxy-names-its-public-issuer.md).** The catalogue no longer
|
||||
> holds a domain module: `networking` was retired, and every machine is assigned the private network's
|
||||
> own module. A module with requirements and no files stays a thing a module may be; "why it is worth
|
||||
> having" below describes what this record decided for `networking`, not what runs.
|
||||
|
||||
*Written 2026-08-29, from building it. The heading above reads as a contradiction of what now
|
||||
exists and is not one — but only if the difference is stated, so it is stated here.*
|
||||
|
||||
**What was refused contains things. What exists contains nothing.**
|
||||
|
||||
| | `networking` as refused | `networking` as built |
|
||||
|---|---|---|
|
||||
| what is in it | WireGuard, a proxy, a firewall — artifacts | nothing at all |
|
||||
| what it says | *these ship together* | *I want a private network and names* |
|
||||
| what is assigned | one module, half of it unwanted | whatever answers each requirement, each on its own |
|
||||
|
||||
The objection above is untouched by this and still correct: a module holding WireGuard and a
|
||||
proxy is assigned where half of it is unwanted. **A module holding nothing cannot be, because
|
||||
there is no half.** It is requirements and a name, and every artifact it leads to is still an
|
||||
ordinary module assigned on its own terms.
|
||||
|
||||
**Why it is worth having.** Most people want the network working and do not want to choose a VPN.
|
||||
`assign networking` finds one answer to each requirement and takes it without asking, because
|
||||
with one candidate there was never a question — the rule below about refusing does the work.
|
||||
Somebody who does care assigns the VPN they want, and *that is the whole of choosing*: there is no
|
||||
flavor field, no variant syntax, and no second verb. **Picking an implementation is assigning a
|
||||
module.**
|
||||
|
||||
**What it costs, stated because it is real.** Adding a second implementation to the catalogue
|
||||
turns a settled question into an open one for **everyone using the bundle**, not only for whoever
|
||||
wanted the alternative. Every node assigned `networking` refuses until somebody says which. That
|
||||
is [the refusing rule](#a-requirement-with-several-answers-is-refused-never-guessed) applied
|
||||
consistently, and the alternative is a default — which is the flavor field returning under a
|
||||
better name. The cost is one assignment per node, and the message names the candidates.
|
||||
|
||||
**A consequence that had to be found by running it.** A bundle can drag an implementation in
|
||||
through a requirement nobody looked at. Choosing a different VPN still installed WireGuard,
|
||||
because the names module needed addresses only WireGuard hands out, and nobody was told. Two VPNs
|
||||
on one machine is not always wrong — a machine may run one for another purpose — but being **the**
|
||||
network the mesh runs over is singular, so that is a claim, and the collision is refused by name.
|
||||
**The general rule: what a bundle pulls in is only as safe as the claims on what it pulls in
|
||||
from.**
|
||||
|
||||
## Three edges
|
||||
|
||||
| edge | means | declared? | satisfied |
|
||||
|---|---|---|---|
|
||||
| **presence** | that thing must exist and be reachable here | yes | at provisioning |
|
||||
| **instantiation** | that thing makes something for me and hands back credentials — a database, a bucket, a route | yes | at provisioning, and again whenever it must be |
|
||||
| **build** | I was compiled against that artifact | **no — read from imports** | **at build, once** |
|
||||
|
||||
**Instantiation implies presence; presence does not imply instantiation.**
|
||||
|
||||
**A route is an instantiation edge**, and it is worth noticing because the direction is the mirror
|
||||
of a database: the consumer supplies a target and receives a *name*, rather than supplying nothing
|
||||
and receiving credentials. Same edge.
|
||||
|
||||
**Provider stops being a category.** Any hosted thing can be a factory — an identity provider
|
||||
grants clients, a mail server grants mailboxes. It is a facet, not a kind.
|
||||
|
||||
**A module may also declare what it claims**, because some things cannot coexist and that is a
|
||||
fact about the module rather than about a particular node. What that means precisely is below.
|
||||
|
||||
### Where the answer to a requirement is allowed to live
|
||||
|
||||
*Written 2026-08-29, from building it. The table above distinguishes **presence** from
|
||||
**instantiation** and this is the half of that distinction nobody had noticed was missing: not
|
||||
what the edge hands over, but **where the thing on the other end is.***
|
||||
|
||||
Two different things were both being written as a requirement:
|
||||
|
||||
| | *a shell*, *a display server*, *a private network* | *a database*, *an object store*, *an identity provider* |
|
||||
|---|---|---|
|
||||
| where the answer lives | **this machine** | **somewhere in the mesh** |
|
||||
| how it is answered | install another module here | find the node already running it |
|
||||
| what is missing if absent | a module to assign here | **a decision about where**, which is nobody's to make silently |
|
||||
|
||||
Answering the second like the first installs a database on every machine that uses one, which is
|
||||
what it did.
|
||||
|
||||
**So a provided name carries a scope**, the same idea a claim already has, and written short in
|
||||
the ordinary case so the few that are not node-scoped stand out rather than drowning. Scope is a
|
||||
property of **the name, not of each provider**: two modules disagreeing about whether a database
|
||||
is local would make one requirement mean two things depending on which happened to answer it, so
|
||||
that is refused.
|
||||
|
||||
**A requirement answered from the mesh is never satisfied by installing it here.** Nothing, and
|
||||
the mesh refuses and says which module to assign somewhere. Two, and it refuses and says how to
|
||||
choose — the same rule as everywhere else, for the same reason: picking is guessing, and the wrong
|
||||
guess puts somebody's data on a machine they did not choose.
|
||||
|
||||
**Choosing is recorded per node**, because that is the granularity the choice actually has — two
|
||||
machines may reasonably use two different databases and a mesh-wide answer could not say so. A
|
||||
choice pointing at a machine that does not provide the thing is refused rather than quietly
|
||||
replaced by one that does, and a single available provider does not override a choice either.
|
||||
**Both are the same rule: the mesh does not overrule a person, and it does not move data without
|
||||
being told to.**
|
||||
|
||||
**What this is a prerequisite for.** Knowing *which node* answers is the first half of handing a
|
||||
credential back — you cannot be given a database's password before it is settled whose database it
|
||||
is. So a node's resolution now records what it takes from elsewhere, which is both the only part
|
||||
of its set that stops working when a *different* machine goes away, and the place a credential
|
||||
will hang.
|
||||
|
||||
### An edge has two directions, and only one of them is built
|
||||
|
||||
*Written 2026-08-29, from building it. The row above already says a consumer **supplies a target
|
||||
and receives a name**; what it did not say is that those are two separate mechanisms, and that
|
||||
having one without the other is what forced two modules outside the system entirely.*
|
||||
|
||||
| direction | the consumer says | who needs it |
|
||||
|---|---|---|
|
||||
| **contribution** | *publish me at this name, on this port* | the proxy, the DNS server, a firewall |
|
||||
| **binding** | *and give me back a credential to it* | the database, the object store, the identity provider |
|
||||
|
||||
**Contribution is built.** A module declares what it contributes to a requirement; the control
|
||||
plane collects every contribution on a node and writes them to a path the provider named, as a
|
||||
file, in the mesh's own shape. **Contributing to something is requiring it** — asking to be
|
||||
published means a publisher must exist, and a module that had to say both would eventually say
|
||||
one, with the failure appearing as a machine where nothing serves the route.
|
||||
|
||||
**The control plane does not know what a reverse proxy is**, and does not write one's
|
||||
configuration. It delivers the facts; the module turns them into whatever it runs. That boundary
|
||||
is what makes swapping the proxy cost nothing in any module that publishes through it, and it is
|
||||
[the same separation](0001-mesh-brokers-nodes-host-agents-think.md) that keeps third-party
|
||||
software *on* the mesh rather than *of* it. It also costs the host nothing: a received file is a
|
||||
file, which was checked by putting the control plane's output through the host's own parser rather
|
||||
than by asserting it.
|
||||
|
||||
**Binding is built except for the secret**, and that turned out to be the useful way to cut it.
|
||||
|
||||
A provider says what a consumer needs in order to use it — a port, a driver, a realm — and a
|
||||
consumer says where it wants to be told. The mesh adds the half only it has: **which machine, and
|
||||
what that machine is called on the private network.** So an application on one node is handed the
|
||||
address of its database on another, as a file, and reaches it by a name the mesh also created.
|
||||
|
||||
**The file states that it carries no credential, and why.** A missing field looks like a bug; a
|
||||
stated absence looks like a boundary, and somebody wiring this up should not spend an afternoon
|
||||
looking for a password that was never going to be there.
|
||||
|
||||
### And the secret, which is delivered without ever being held
|
||||
|
||||
*Written 2026-08-30, after looking at how the existing mesh does it. The design here is a reaction
|
||||
to a measurement, not a preference.*
|
||||
|
||||
**The obvious arrangement is a credentials column, encrypted at rest.** It exists, and its own
|
||||
tooling records what it bought:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| the tool for finding a secret matches **by value**, not by name | because one password is in the provisions table, the environment table, each node's environment file in plain text, and **inside every connection string composed from it** — copies its documentation calls *"often the only copies actually in use"* |
|
||||
| a query against the encrypted column **returns zero rows and proves nothing** | so auditing moved to the decrypted copies on the machines |
|
||||
|
||||
**Two faults, and encryption at rest addresses neither.** The control plane can read what it
|
||||
stores, so a copy of its database is a copy of every credential in the mesh. And one secret has
|
||||
many homes with nothing tracking them — **composition is what mints the untracked ones**, because
|
||||
building a connection string centrally creates a new secret-bearing value no rotation path knows
|
||||
about.
|
||||
|
||||
**So the value is sealed to the node that will use it before it is stored.** With a key that node
|
||||
generated and whose private half the mesh has never seen — a third key beside the identity and the
|
||||
overlay, for the same reason those are two rather than one. What is stored is unusable by whoever
|
||||
holds it, the mesh included, and the broker relays a blob it cannot read. This is what makes
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md)'s *compromise of a node is compromise of that node*
|
||||
true of secrets rather than true of identity and quietly false of everything that matters.
|
||||
|
||||
**And nothing is composed centrally.** A connection string is assembled on the machine that needs
|
||||
one, if at all. The mesh delivers parts.
|
||||
|
||||
**What it costs, stated because it is real:** the mesh cannot audit by value. That is the right
|
||||
trade rather than an oversight — a query over an encrypted column could not either, so the audit
|
||||
was never real. What *is* answerable is which node holds what, which is the question rotation
|
||||
actually asks.
|
||||
|
||||
**A consequence that shapes the mechanism.** The mesh discarded the plaintext, so it cannot
|
||||
compose a file containing it. The credential is therefore **its own file**, holding the value and
|
||||
nothing else, beside the readable one. That is better than the alternative it was forced into:
|
||||
the readable half stays readable in the declaration, and the secret half changes only when the
|
||||
secret does, so a service reloading on it reloads for a real reason.
|
||||
|
||||
**Rotation is generating a new one**, because reading the old one back is not possible. Both ends
|
||||
are re-sealed and reach their machines in the same push — which removes the window where half the
|
||||
mesh holds a dead credential, the failure
|
||||
[recorded in ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) as consumers on three nodes
|
||||
holding one for two days.
|
||||
|
||||
### The provisioner, which is where the mesh stops
|
||||
|
||||
*Written 2026-08-30, from building one and running it against a real database.*
|
||||
|
||||
**A password nothing was told to create authenticates nowhere.** The mesh generates one, seals it
|
||||
to both ends and cannot read it — so it cannot tell the software to start accepting it either.
|
||||
Something on the providing machine reads what arrived and makes it true. That is a provisioner.
|
||||
|
||||
**It belongs to the module, not to the mesh**, and the boundary is the same one that keeps
|
||||
third-party software running *on* the mesh rather than being *of* it
|
||||
([ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)). The control plane decides and never
|
||||
touches a machine. **What the mesh owns is the contract**, which is two files the host writes from
|
||||
an ordinary declaration:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| the manifest | every consumer, what it asked for, and **where** its credential is |
|
||||
| one file per consumer | that credential, alone in it |
|
||||
|
||||
Two files because the mesh discarded the value and cannot compose a document containing it. As
|
||||
before, the constraint produces the better shape: the readable half stays readable and auditable
|
||||
in the declaration, and the secret half changes only when the secret does.
|
||||
|
||||
**It reconciles; it is never told what changed.** It runs after every declaration and must reach
|
||||
the same state from wherever it starts. Three consequences, and each of them is a fault that has
|
||||
been shipped somewhere:
|
||||
|
||||
- **the password is set every time, not only on creation** — otherwise the role already exists,
|
||||
nothing happens, and a rotation reports success while changing nothing
|
||||
- **what it made and nobody asks for any more is removed** — otherwise a consumer that left keeps
|
||||
a working login for ever and nothing ever says so. This is the same rule the host follows about
|
||||
[removing what it declared and no longer declares](../04-ISSUES/010-the-first-declaration-destroys-the-substrate/00-report.md)
|
||||
- **what it did not make is left alone** — otherwise it cannot be run on a system that predates
|
||||
it, which is every system anybody would want to adopt
|
||||
|
||||
**A missing credential is refused rather than worked around.** A role created without one is a
|
||||
login nothing can use, and nothing would report it until something tried to connect.
|
||||
|
||||
**This is where the mesh stops**, and saying so is the point of the section. It decides, delivers
|
||||
and can prove what it delivered; the last inch belongs to whoever knows what `create role` means.
|
||||
|
||||
**One check that only became possible now.** Two machines wired together across no private network
|
||||
is a mesh that reports itself configured and does not work, and the failure surfaces as a
|
||||
connection timing out — the slowest place to find anything. It is refused, and it is only
|
||||
*checkable* because the network became [something a machine is
|
||||
given](#what-a-domain-module-turns-out-to-be-and-why-it-is-not-the-one-refused-above) rather than
|
||||
something it has by virtue of holding an address.
|
||||
|
||||
**What the absence cost, measured.** Exactly two modules opened a direct connection to the control
|
||||
plane's database — the proxy and the VPN — and they are the reason every node permanently holds a
|
||||
credential to it. Both were doing by hand what this edge is for. The VPN's half is closed by being
|
||||
[a module whose files are computed](../03-DESIGN/01-to-be/08-connectivity.md); the proxy's is
|
||||
closed by contribution. **Neither needed a new kind of thing, and both had been outside the model
|
||||
for as long as there was one.**
|
||||
|
||||
### Why the build edge is a different kind
|
||||
|
||||
It is fixed inside an artifact rather than negotiated when something runs, and **its only remedy
|
||||
is a rebuild** — nothing can re-provision it.
|
||||
|
||||
It is also **derived rather than declared**, and the asymmetry is deliberate: a runtime edge is an
|
||||
*intention* somebody has about how the mesh should be wired, and only a person can state it. A
|
||||
build edge is a *fact about code that already exists*, and a declared list of dependencies drifts
|
||||
from the imports it describes.
|
||||
|
||||
**An artifact is out of date when its source moved, or when anything it was built against moved.**
|
||||
So what is recorded is a commit *and the identity of every artifact it was built against*, which
|
||||
is what makes the rebuild set computable and *is this current?* answerable without building.
|
||||
|
||||
**The graph measures design quality, not just build order.** A module with many inbound build
|
||||
edges is one whose every change is expensive — and that is readable before anything is built. The
|
||||
current shared library is exactly that, and nobody could see it because nothing drew the edges.
|
||||
|
||||
## Provisioning is declared, never configured by hand
|
||||
|
||||
A module declares what it **provides** and what it **requires**. The mesh satisfies it: a
|
||||
provisioner belonging to the provider creates the resource and its credential, records the grant,
|
||||
and the values are derived onto the consumer. **Neither the credential nor the topology is ever
|
||||
written by hand.** A requirement may name a provider on another node, so cross-node wiring is the
|
||||
same declaration.
|
||||
|
||||
## What a module claims, and why it is not a list of rivals
|
||||
|
||||
*Written 2026-08-29, replacing pairwise exclusion.*
|
||||
|
||||
**Exclusivity is not a property of a module. It is a property of a singular resource the module
|
||||
takes over.** Two shells do not compete for anything and any number may be installed. Two display
|
||||
servers both want the seat, and only one may have it.
|
||||
|
||||
> **A module declares what it *claims*. Two modules claiming the same thing cannot both be
|
||||
> assigned within that claim's scope.**
|
||||
|
||||
**Not "xorg conflicts with wayland".** Pairwise exclusion has a property that only shows up later:
|
||||
adding a third display server means **editing xorg and wayland to know about it**. Every new
|
||||
module requires changing modules nobody who wrote it owns, and the edits grow as the square of
|
||||
the count. With a claim, the third one says `claims: the seat` and nothing else changes anywhere.
|
||||
**The new module is the only thing that has to know anything** — which is the difference between
|
||||
a catalogue that grows and one that calcifies.
|
||||
|
||||
The pattern is common enough to be worth listing, because seeing it is most of understanding it:
|
||||
|
||||
| these coexist | these claim one thing |
|
||||
|---|---|
|
||||
| shells — bash, zsh, fish | display servers — xorg, wayland (*the seat*) |
|
||||
| editors — vim, emacs, helix | init — systemd, openrc (*pid 1*) |
|
||||
| language runtimes | container runtime — docker, podman |
|
||||
| terminal emulators | reverse proxies — nginx, caddy, traefik (*ports 80/443*) |
|
||||
| browsers | time — chrony, timesyncd, ntpd (*the clock*) |
|
||||
| | resolvers — resolved, dnsmasq, unbound (*`/etc/resolv.conf`*) |
|
||||
| | network management — NetworkManager, networkd, netctl |
|
||||
| | mail — postfix, exim, msmtp (*port 25*) |
|
||||
| | audio — pipewire, pulseaudio (*the device*) |
|
||||
|
||||
**A claim has a scope**, because not everything singular is singular per machine:
|
||||
|
||||
| scope | example |
|
||||
|---|---|
|
||||
| **node** | the seat, pid 1, port 443 |
|
||||
| **site** | a DHCP server on a segment |
|
||||
| **mesh** | the hub, the control plane |
|
||||
|
||||
The last is not new — the mesh already enforces exactly one hub with a unique index
|
||||
([ADR 0007](0007-connectivity.md)). Scope is that idea, said once rather than hard-coded per case.
|
||||
|
||||
**Some conflicts need no claim at all.** Two modules declaring the same file, or binding the same
|
||||
port, are visible from *what they declare* — the mesh already holds every resource of every
|
||||
declaration. So a claim is only written for the abstract ones, where nothing in the declaration
|
||||
reveals the clash. That keeps the manifest small, which is worth protecting.
|
||||
|
||||
## A requirement with several answers is refused, never guessed
|
||||
|
||||
A module requiring *a shell* may be satisfied by three. The mesh does not pick.
|
||||
|
||||
| candidates | what happens |
|
||||
|---|---|
|
||||
| exactly one | assigned, silently — there was no choice to make |
|
||||
| none | refused, naming what is missing |
|
||||
| several | **refused, naming them**, and a person chooses |
|
||||
|
||||
**This is what makes a solver unnecessary.** Counting candidates is a few lines and has no
|
||||
surprising behaviour; a solver that picks has to be understood before its answer can be trusted,
|
||||
and it is understood by whoever is debugging it at the time. Nothing here is lost by waiting —
|
||||
a solver can be added later without changing a single manifest, and the reverse is not true.
|
||||
|
||||
**Requiring a module and requiring a capability are different fields**, because the remedies
|
||||
differ and the message should say which:
|
||||
|
||||
- *i3 needs xorg, which is not assigned here* — assign it.
|
||||
- *this machine has no seat* — wrong machine; nothing can be installed to fix it.
|
||||
|
||||
### A capability may carry a value, and that is not a new idea
|
||||
|
||||
A capability is a named fact about a machine, **detected and never assumed**. Its presence gates
|
||||
an assignment; its detail can also carry a value — `seat: card1-DP-1`, `panel: oled`, an
|
||||
architecture, an amount of memory. Nothing new is needed for that: a verdict has always had a
|
||||
detail beside its yes or no.
|
||||
|
||||
So *can this run here* and *what should it be configured as* are answered by the same fact, read
|
||||
two ways. A module that must not be assigned without an OLED panel and one that dims itself
|
||||
differently on one are reading the same line.
|
||||
|
||||
**What keeps the set from sprawling is the cost of adding one.** A capability must be detected,
|
||||
and the detector must say how it knows — so nobody can add one they cannot check, which is the
|
||||
whole of [04-ISSUES/007](../04-ISSUES/007-an-installed-package-is-not-a-capability/00-report.md):
|
||||
an installed package was treated as a capability and a node was assigned work it could not do.
|
||||
|
||||
**And detectors ship inside the host**, which is one statically linked binary. Adding a capability
|
||||
means shipping a new host to every node that needs it. That is a real cost and it argues for
|
||||
keeping the vocabulary small and general — `seat`, not `has-nvidia-with-two-outputs`.
|
||||
|
||||
## "Flavor" is retired
|
||||
|
||||
It was carrying three unrelated meanings — variants of a thing, a subset of one module a node
|
||||
installs, and whatever the current system does, which earned two knowledge-base entries about
|
||||
going wrong. **A word with three meanings cannot be reasoned about**, and every attempt to design
|
||||
around it produced a rule that was right for one meaning and wrong for the others.
|
||||
|
||||
What it was reaching for is two ordinary things:
|
||||
|
||||
- **Different modules that provide the same thing.** `zsh` and `fish` both provide *a shell*. They
|
||||
are two modules, not one module with a switch: they share a name and nothing else — different
|
||||
packages, different configuration, different everything.
|
||||
- **One module with a setting.** A monitoring module that is an agent here and a server there is
|
||||
one module, configured. Nothing varies but a value.
|
||||
|
||||
If something is neither, it is probably two modules.
|
||||
|
||||
**A third thing it was reaching for, added 2026-08-29:** *I want this working and I do not care
|
||||
which one.* That is a module with requirements and no files —
|
||||
[a domain module](#what-a-domain-module-turns-out-to-be-and-why-it-is-not-the-one-refused-above) —
|
||||
and it is what makes "different modules that provide the same thing" bearable for somebody who
|
||||
does not want to know there is a choice.
|
||||
|
||||
## The core library is the mesh's domain
|
||||
|
||||
One module everything may depend on. It holds **what is true of the mesh regardless of which
|
||||
context you are in**: a module, a node, an assignment.
|
||||
|
||||
The test: *would this still mean the same thing in a context that had never heard of the one it
|
||||
came from?* A node would. A pipeline stage would not — that is delivery's.
|
||||
|
||||
**Types ship with the module that owns them**, not here. A consumer needing `inventory`'s types
|
||||
depends on `inventory` — one narrow, visible edge — rather than everything depending on a hub
|
||||
where the relationship cannot be seen. **A library everything depends on is expensive to change
|
||||
whether it holds types or code; the fan-in is what makes it expensive**, which is why *types, not
|
||||
behaviour* was the wrong guard.
|
||||
|
||||
**It stays small on its own.** A domain model changes when what the mesh *is* changes, which is
|
||||
rare. A drawer labelled *shared* changes whenever anybody writes something reusable, which is
|
||||
constantly — and *who else might want this* always answers yes, which is how the current one grew.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Fewer things will be shared, and some code will be written twice.** That is the trade: the
|
||||
current library exists because sharing felt free. Two similar functions in two modules is often
|
||||
the better answer.
|
||||
- **The check is a measurement rather than a prohibition.** Inbound build edges say when something
|
||||
is becoming a hub, while it is happening rather than after.
|
||||
- **Reading build edges needs a language-aware tool per language**, which is the real cost and the
|
||||
reason declaring them looks tempting. It is still wrong.
|
||||
@@ -1,161 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 10. Delivery
|
||||
|
||||
*Consolidated 2026-08-28 from five records.*
|
||||
|
||||
## Delivery is a comparison, not a pipeline
|
||||
|
||||
**The control plane holds what source exists and what has been built from it, and builds the
|
||||
difference.** A change becomes a build because source is **ahead of artifacts** — answerable at
|
||||
any moment — rather than because a message arrived.
|
||||
|
||||
**An event makes it fast. Nothing makes it necessary.** A missed notification costs latency and
|
||||
cannot cost correctness.
|
||||
|
||||
That is the same shape the host uses on a machine, one layer up:
|
||||
|
||||
| | reconciles | against |
|
||||
|---|---|---|
|
||||
| the control plane | artifacts | source |
|
||||
| the host | machine state | declarations |
|
||||
|
||||
**This is not the current coordinator repaired.** That is a state machine over stages; the value
|
||||
of it here is as a catalogue of the ways this fails, and it has been used for exactly that.
|
||||
|
||||
### The module system is the CI/CD
|
||||
|
||||
*Written 2026-08-29, because this was the intention throughout and was never stated in one line.*
|
||||
|
||||
**There is no pipeline product beside the mesh, and there is not going to be one.** A module
|
||||
declares what it is ([ADR 0009](0009-modules-and-the-graph.md)); the control plane notices its
|
||||
source is ahead of its artifacts and builds it; the graph says what else that invalidates; the
|
||||
node that should run it is told. Build, test, publish and deploy are the same reconciliation seen
|
||||
at four points, not four stages wired together.
|
||||
|
||||
**Which is why the module system is the core of the setup rather than one component of it.** Every
|
||||
other layer is carried by it: the substrate is modules the bundle raises before there is a mesh,
|
||||
the control plane is a module, and an application is a module with a different manifest. A thing
|
||||
that cannot be expressed as a module cannot be delivered at all — that is a real constraint, and it
|
||||
is the one keeping a second delivery mechanism from growing beside this one.
|
||||
|
||||
**What disappears is the pipeline as a state machine** — no stage list something can be omitted
|
||||
from, which is how a verify stage was built and never scheduled, and no run to lose.
|
||||
|
||||
### Currency is the whole input closure
|
||||
|
||||
**An artifact is out of date when its source moved, or anything it was built against moved.** So a
|
||||
shared library changing invalidates everything with a transitive build edge to it, in dependency
|
||||
order, because a module cannot be built against a new library until it exists.
|
||||
|
||||
**The module graph is a prerequisite of this, not an enabler of it.** Without it there is no
|
||||
rebuild set and no ordering, and this cannot be implemented.
|
||||
|
||||
## An artifact is build output, never a source tree
|
||||
|
||||
Compiled and bundled with its dependency graph inlined. **A deploy is extract-and-run and touches
|
||||
no network.**
|
||||
|
||||
The consequence is the whole cost of the decision: **anything not in the build output does not
|
||||
ship.** Every file kind had to be brought into that rule separately, and each was discovered by
|
||||
something silently not happening after a deploy — migrations reading a source layout,
|
||||
provisioning scripts reading a source layout, selection files never packaged at all.
|
||||
|
||||
## Three silos, and the third is not a stage
|
||||
|
||||
The cardinality observation holds and is what the split is for:
|
||||
|
||||
| silo | runs | ends with |
|
||||
|---|---|---|
|
||||
| **build** | once per module | a self-contained artifact |
|
||||
| **publish** | once per module | that artifact addressable — an image by digest, a package in the mesh's repository |
|
||||
| **deploy** | **once, not once per node** | the affected nodes' **declarations updated** |
|
||||
|
||||
**Deploy stops sending commands to nodes.** It changes what the control plane says each node
|
||||
should be, which is one write. What happens on the machines is the host's ordinary reconcile.
|
||||
|
||||
**Why this fixes the failure class rather than patching it.** Every recorded fault shares one
|
||||
shape: *the thing that reported success was not the thing that did the work.* A coordinator
|
||||
dispatching a command can only report on dispatch. Under this the reporter **is** the applier —
|
||||
which already refuses to record a resource until it read it back, and already fails the whole
|
||||
apply on one failed step.
|
||||
|
||||
**The verify stage disappears as a stage**, which is the strongest evidence for the shape:
|
||||
verification stops being a step that can be omitted from a list and becomes a property of applying
|
||||
at all.
|
||||
|
||||
**There is no fan-out**, so the defect class that came from the build node having passed through
|
||||
two silos while others had not cannot arise.
|
||||
|
||||
## A step that fails must fail the job
|
||||
|
||||
A step that fails and lets the job continue **reports success for work that did not happen**.
|
||||
Absence of an error is not evidence of an effect.
|
||||
|
||||
This is the mesh's most consistent failure shape, and it is not incidental — it is what stage
|
||||
reporting measured. Documented instances: a service reported started when the container command
|
||||
merely returned; an image pull failure that did not fail the deploy; a package install that 404'd
|
||||
from every mirror while the job went green; a node left on old code after a failed download with
|
||||
a version marker that had already advanced.
|
||||
|
||||
## The verdict is tiered
|
||||
|
||||
An artifact may not be declared until something has judged it fit. **Two tiers, because one gate
|
||||
would be both slow and unreliable:**
|
||||
|
||||
| | judged by | when |
|
||||
|---|---|---|
|
||||
| the module's own tests | the build | **always** — this is most of it |
|
||||
| the lab | a raised scenario | when an assertion genuinely needs a mesh |
|
||||
|
||||
A lab scenario takes tens of seconds and can fail for reasons that have nothing to do with the
|
||||
artifact, and a shared-library change produces a cascade of dozens. One expensive
|
||||
non-deterministic gate fails in both directions: a flaky run marks a good artifact unfit, a lucky
|
||||
one marks a bad artifact fit, and **neither failure looks like itself.**
|
||||
|
||||
**A run that failed environmentally is not a verdict.** A machine that would not boot says nothing
|
||||
about the artifact, and recording it as *unfit* is the same untruth as recording a dispatch as a
|
||||
deploy.
|
||||
|
||||
## What a result means
|
||||
|
||||
> **The declaration is updated, and here is which nodes have applied it.**
|
||||
|
||||
A pipeline does not wait for every node — one may be legitimately switched off for a week, and a
|
||||
delivery mechanism that blocks on a sleeping laptop is one nobody will use.
|
||||
|
||||
```
|
||||
delivered declaration updated for 5 nodes
|
||||
applied 3 of 5
|
||||
outstanding 2 — last seen 4 days ago, 20 minutes ago
|
||||
```
|
||||
|
||||
**Outstanding is not failure**, and conflating them is how the old system produced a stall with no
|
||||
error anywhere.
|
||||
|
||||
## What must exist first
|
||||
|
||||
1. **The module graph, with build edges.** No graph, no rebuild set and no ordering.
|
||||
2. **A recorded input closure per artifact**, so currency is answerable without building.
|
||||
3. **Something that notices a reconciler is not converging.** Below.
|
||||
|
||||
## Open, and the first is the real risk
|
||||
|
||||
- **A loop that will not converge is harder to debug than a job that failed.** A failed job stops
|
||||
and names its step; a reconciler retries forever. Without something that notices *this has been
|
||||
trying for an hour*, the failure is **silence** — the fault this removes, reintroduced in a new
|
||||
place.
|
||||
- **The run identity people use is lost.** *Did my change go out?* is answerable today by opening
|
||||
a pipeline. Something must replace that or this is worse to live with, whatever its properties.
|
||||
- **Does a fit artifact declare itself?** If it does, merging to main deploys to production —
|
||||
which may be wanted and is far too large a property to acquire by omission.
|
||||
- **Rebuild storms are mostly behaviourally empty.** Reproducible builds would stop a cascade at
|
||||
the first module whose output did not move; without them one commit redeploys the fleet for no
|
||||
change in behaviour.
|
||||
- **Detection stays the fragile input for latency**, though no longer for correctness.
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-04-03
|
||||
deciders: jochen
|
||||
reconstructed: true
|
||||
---
|
||||
|
||||
# 11. Managed files are generated onto nodes and never edited there
|
||||
|
||||
> Reconstructed after the fact from the evidence cited below.
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md) put every binding in the mesh
|
||||
database. But the things that consume those bindings — environment files, service
|
||||
definitions, daemon configuration, firewall rules — are files on a node's disk, because that
|
||||
is what the software reading them requires.
|
||||
|
||||
So the same value exists twice: authoritatively in the database, and materialised in a file.
|
||||
Any edit to the file is a change to a copy. Before this decision, environment values could be
|
||||
pushed from a node back into the database, which made the direction ambiguous in both
|
||||
directions at once.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Bidirectional sync** — a node's edits flow back to the database. Rejected, and removed.
|
||||
Two writers and no arbiter: whichever synced last wins, and neither is authority.
|
||||
2. **Files are authoritative; the database is a cache of them.** Rejected — it inverts
|
||||
ADR 0013 and returns to state that cannot be reconciled across nodes.
|
||||
3. **Strictly one-directional: the database is written, files are generated.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
Every managed file is **derived**. A synchroniser regenerates it from the mesh database
|
||||
whenever the underlying values change. The write path is the mesh tool that owns the value;
|
||||
the file is an output.
|
||||
|
||||
This applies to generated environment files, service definitions, managed configuration, and
|
||||
anything else a synchroniser lists as its own.
|
||||
|
||||
An edit to a managed file survives until the next synchronisation and is then overwritten,
|
||||
without a warning, taking whatever it was fixing with it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A file edited on a node is a bug with a delay on it.** This is now one of the mesh's core
|
||||
values, and it is a consequence of this decision rather than a stance taken independently.
|
||||
- To change a value you must know which tool owns it. That is a real cost, paid every time,
|
||||
and the reason the mesh provides a way to ask whether a given file is managed.
|
||||
- Debugging by editing a file no longer works, and fails in the most confusing way available:
|
||||
it works, and then stops working later for no locally visible reason.
|
||||
- Recovery is cheap. A node's entire managed surface can be regenerated from the database.
|
||||
- Values resolve by precedence — database override, then existing file value, then generated,
|
||||
then manifest default — which means an unset override does not clobber a generated
|
||||
password. The subtlety is real and has caused its own confusion.
|
||||
|
||||
## References
|
||||
|
||||
- `Extract hal/env-sync module, remove syncEnvToDb`, 2026-04-03 — the commit that removed the
|
||||
node-to-database direction.
|
||||
- Knowledge base: `conventions/no-direct-file-mutation`, `provisioning` (resolution priority).
|
||||
- Regeneration gaps: `troubleshooting/changed-manifest-default-not-rerendered`,
|
||||
`troubleshooting/config-removed-from-manifest-not-pruned`.
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-08-23
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 12. The mesh creates no symlinks — a derived file is a copy
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0012](0012-the-mesh-creates-no-symlinks.md) responded to production data loss — a hand-made
|
||||
link, resolved through a container engine's volume handling, pointing a mount somewhere it
|
||||
should not have — by centralising linking in the installer and forbidding it everywhere else.
|
||||
|
||||
That narrowed the incident class. It did not close it. The hazard is not *who* made the link;
|
||||
it is that a path can resolve somewhere other than where it appears to. A link made by the
|
||||
installer resolves exactly the same way as a link made by hand. The rule made the mechanism
|
||||
rarer and better-governed while leaving the mechanism in place.
|
||||
|
||||
Two things have changed since, and together they remove the argument that kept it.
|
||||
|
||||
**The original case for linking was staleness.** A copy of a service definition goes stale
|
||||
silently while the catalogue moves on, so a link was the cheap way to guarantee the running
|
||||
node reads a current definition. That argument assumes the node's copy is unmanaged.
|
||||
|
||||
**It is not.** [ADR 0011](0011-managed-files-are-generated-never-edited.md) established that
|
||||
everything on a node's disk is derived from the mesh and regenerated when its inputs change,
|
||||
and the installer already **reconciles** links rather than assuming them — repointing stale
|
||||
ones, adopting real files it finds where a link belongs. Reconciling content is the same
|
||||
operation as reconciling a pointer, plus a comparison.
|
||||
|
||||
So the mesh already has the machinery that makes a copy safe, and is using a link to solve a
|
||||
problem that machinery solves better. Worse, a link is conceptually the wrong shape: it makes
|
||||
the node's runtime state a *pointer into source*, which is the one thing
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md) and ADR 0011 exist to prevent.
|
||||
State is derived onto nodes; it does not reach back.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Keep ADR 0019 as the final position** — centralised linking, forbidden elsewhere.
|
||||
Rejected as the status quo. It governs the mechanism rather than removing it, and the
|
||||
failure it was written for remains reachable by any code path the installer trusts.
|
||||
2. **Keep links but harden them** — canonicalise before mounting, refuse a link that escapes
|
||||
an expected root. Rejected: it is a check bolted onto a hazard, and it has to be correct in
|
||||
every consumer, including container engines the mesh does not control.
|
||||
3. **Copy, reconciled by the installer, with staleness detected rather than assumed away.**
|
||||
Proposed here.
|
||||
|
||||
## Decision
|
||||
|
||||
*Accepted 2026-08-25. The position is settled; the migration is not designed. See "Open" below.*
|
||||
|
||||
**The mesh creates no symlinks.** A file a node needs is placed on that node as a real file,
|
||||
derived from the mesh and reconciled by the installer like every other managed file
|
||||
([ADR 0011](0011-managed-files-are-generated-never-edited.md)).
|
||||
|
||||
The prohibition in ADR 0019 stands and widens: it ceases to be "only the installer may link"
|
||||
and becomes "nothing links, the installer included".
|
||||
|
||||
When this is accepted, ADR 0019 becomes superseded rather than edited — its reasoning is why
|
||||
the rule exists at all, and the incident behind it is the reason anyone believes either record.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The path-resolution hazard is removed rather than governed. There is no link for a container
|
||||
engine to resolve, so the class of failure that cost production data is closed by
|
||||
construction.
|
||||
- A node's runtime state stops pointing into source. What a node holds is derived output, which
|
||||
is what the mesh's model already says it is everywhere else.
|
||||
- **Staleness becomes a real problem that must be answered, not assumed away.** This is the
|
||||
cost, and it is the whole cost: today a link cannot be stale, and a copy can. The answer has
|
||||
to be detection — the installer comparing what is on disk against what the mesh says should
|
||||
be — and it must be loud, because a silently stale definition is exactly the failure shape
|
||||
this mesh keeps producing ([ADR 0010](0010-delivery.md)).
|
||||
- Reconciliation gets more expensive: comparing content rather than checking a pointer's
|
||||
target, on every module, on every node.
|
||||
- Disk usage rises, trivially, and is not a consideration.
|
||||
- Existing links must be converted. A node mid-migration holds both forms, so reconciliation
|
||||
has to handle finding a link where a file now belongs — the mirror image of the adoption it
|
||||
already does.
|
||||
|
||||
## Open
|
||||
|
||||
- **How staleness is detected.** Content hash, version marker, or regeneration on every
|
||||
reconcile. This is the decision that makes or breaks the change and it is not taken here.
|
||||
- **Whether anything must keep a link** for reasons outside the mesh's control. If something
|
||||
does, that is a finding worth recording rather than an exception worth granting quietly.
|
||||
- **Migration order.** Converting a node's links is a change to how its services resolve their
|
||||
own definitions, which is not a change to make everywhere at once.
|
||||
|
||||
Until those are answered this record stays `proposed`, and ADR 0019 remains the governing rule.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0012](0012-the-mesh-creates-no-symlinks.md) — the incident, and the rule this widens.
|
||||
- [ADR 0011](0011-managed-files-are-generated-never-edited.md) — the machinery that makes a
|
||||
copy safe.
|
||||
- [`03-DESIGN/00-as-is/05-runtime-and-installation.md`](../03-DESIGN/00-as-is/05-runtime-and-installation.md)
|
||||
— what the installer does today, including reconciliation and adoption.
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-05-14
|
||||
deciders: jochen
|
||||
reconstructed: true
|
||||
---
|
||||
|
||||
# 13. Schema and state changes are numbered migrations, in the same language as the code
|
||||
|
||||
> Reconstructed after the fact from the evidence cited below.
|
||||
|
||||
## Context
|
||||
|
||||
Modules own persistent state. That state has to change as they change, across nodes that are
|
||||
at different versions, some of which have data that predates the change.
|
||||
|
||||
Two things were being done that do not survive contact with a second node. Schema was created
|
||||
at startup, so what a table looked like depended on which version last started. And migrations
|
||||
were shell scripts, so they could not use the types, connection handling or helpers the module
|
||||
already had, and were not compiled or checked with it.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Startup SQL / create-if-missing.** Rejected. It converges only for a node that started
|
||||
with the newest version. A node that never restarts never migrates; a node that restarts on
|
||||
an old version can undo a change.
|
||||
2. **Shell migrations.** Rejected. Unchecked, untyped, and a separate dialect from the module
|
||||
they belong to. Also, as later discovered, packaged differently — and therefore
|
||||
occasionally not packaged at all.
|
||||
3. **Numbered migrations in the module's own language, compiled with it.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
Every schema or state change is a numbered migration file, written in the same language as the
|
||||
module and compiled with it. There is no startup schema creation and no ad-hoc statement.
|
||||
|
||||
Rules that come with it:
|
||||
|
||||
- The initial migration is **frozen** once it has run anywhere. It is never modified; a change
|
||||
is a new number.
|
||||
- Every statement is **idempotent** — guarded so that re-running is safe.
|
||||
- A change needs **both** a baseline for a fresh installation and an incremental migration for
|
||||
installations that already exist. Code referencing a column requires that the migration
|
||||
creating it exists.
|
||||
- Migration numbers are unique. A duplicate prefix is a defect, not a style issue.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A node at any version converges to the current schema by running the migrations it has not
|
||||
run.
|
||||
- Migrations are checked by the same compiler as the code, and a migration that does not
|
||||
compile fails the build rather than the deployment.
|
||||
- The rules are enforced unevenly. Duplicate prefixes have shipped repeatedly and been fixed by
|
||||
renumbering afterwards; the mesh now validates for them, which is the check this rule needed
|
||||
in order to be real.
|
||||
- A migration directory is a feature like any other, which means it is packaged like any other
|
||||
— and when packaging is wrong, migrations silently do not ship. This has happened.
|
||||
- Freezing the initial migration means a fresh installation replays the entire history. That
|
||||
cost grows and nothing currently bounds it.
|
||||
|
||||
## References
|
||||
|
||||
- `fix(agents): convert workflow migrations to TypeScript` (#50), 2026-05-14.
|
||||
- `feat(dev_validate): guard against duplicate migration numeric prefixes` (#379), 2026-06-26 —
|
||||
the rule acquiring a check.
|
||||
- Knowledge base: `migrations/schema-drift`, `noxflow/troubleshooting/migration-number-collision`.
|
||||
- Packaging failures: `troubleshooting/shell-migrations-never-packaged`,
|
||||
`troubleshooting/provision-migration-never-applied`.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-06-04
|
||||
deciders: jochen
|
||||
reconstructed: true
|
||||
---
|
||||
|
||||
# 14. No workspace — each module is a standalone package consuming published dependencies
|
||||
|
||||
> Reconstructed after the fact from the evidence cited below.
|
||||
|
||||
## Context
|
||||
|
||||
Modules depend on each other, above all on the shared library every module builds against.
|
||||
A workspace was the obvious way to express that: sibling packages, resolved locally, one
|
||||
install at the root.
|
||||
|
||||
It produced a divergence that is worth stating precisely, because it is not obvious. In
|
||||
development, a workspace member importing a sibling resolves to that sibling's **local source**.
|
||||
In the pipeline, each module is built alone, from a clone, without its siblings present — so
|
||||
the same import resolves to the **published version**. The two environments were therefore
|
||||
building different code from identical source, and the failure appeared only in the pipeline,
|
||||
in a module that had not been touched.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Keep the workspace and make the pipeline replicate it** — clone every module, build the
|
||||
graph. Rejected: it makes every build a whole-repository build, which is the cost the
|
||||
per-module pipeline exists to avoid, and it does not extend to modules in their own
|
||||
repositories.
|
||||
2. **Keep the workspace and pin siblings to published versions.** Rejected as the worst of
|
||||
both: the workspace's local resolution silently overrides the pin, so the divergence
|
||||
remains while looking solved.
|
||||
3. **No workspace. Every module is standalone and consumes published dependencies.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
There is no workspace. Each module is an independent package that declares its dependencies
|
||||
and consumes them from the private registry, including the mesh's own shared library.
|
||||
|
||||
A cross-package change is therefore two steps: publish the producer, then consume it. The
|
||||
pipeline does the first on push and resolves the levels so that a module always builds against
|
||||
its dependencies' freshly published versions.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Development and the pipeline resolve imports identically. The divergence is gone by
|
||||
construction rather than by discipline.
|
||||
- A module in its own repository is not a special case. It builds exactly as a module in the
|
||||
monorepo does — which is what makes [ADR 0015](0015-applications-live-in-their-own-repository.md)
|
||||
cheap.
|
||||
- A cross-package change costs a publish-and-consume round trip. This is the real price, paid
|
||||
on every shared-library change.
|
||||
- There is no repository-wide install and no repository-wide build. Anything that assumed one
|
||||
broke, and one thing that assumed one has stayed broken: the end-to-end pipeline harness has
|
||||
not built since this decision landed. See
|
||||
[`04-ISSUES/005`](../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md).
|
||||
|
||||
## References
|
||||
|
||||
- `fix(noxflow): kill npm workspace, restore encryption inside PgAdminRepo` (#240),
|
||||
2026-06-04. The reason is recorded in the root package manifest, which still carries the
|
||||
note explaining why no workspace exists.
|
||||
- The divergence it fixed is named there: workspace members importing each other resolved to
|
||||
local unbuilt source in the pipeline.
|
||||
@@ -1,65 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-07-10
|
||||
deciders: jochen
|
||||
reconstructed: true
|
||||
---
|
||||
|
||||
# 15. Applications live in their own repository; the monorepo is for the mesh
|
||||
|
||||
> Reconstructed after the fact from the evidence cited below.
|
||||
|
||||
## Context
|
||||
|
||||
The module system makes adding anything to the monorepo trivial — a directory and a manifest.
|
||||
That ease is the problem. Standalone applications, sites and side-projects accumulated beside
|
||||
the mesh's own components, and once there they inherited the monorepo's review cadence, its
|
||||
pipeline detection, and its history.
|
||||
|
||||
The mesh's own code and an application that merely runs on the mesh have nothing in common
|
||||
except the manifest format. They change for different reasons, are reviewed by different
|
||||
criteria, and have no reason to share a branch.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Everything in the monorepo.** Rejected — it is what existed. The monorepo becomes an
|
||||
inventory of one installation's applications, and every application change queues behind
|
||||
mesh review.
|
||||
2. **A second monorepo for applications.** Rejected: the same coupling with an extra name.
|
||||
Applications have no more in common with each other than with the mesh.
|
||||
3. **One repository per application, registered with the mesh as a build source.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
Every standalone application, site or side-project lives in its own repository, with a manifest
|
||||
at the root. It registers with the mesh as a build source and is then built, provisioned,
|
||||
deployed and verified by exactly the same pipeline as anything in the monorepo.
|
||||
|
||||
The monorepo holds the mesh: the runtime, the core modules, the delivery machinery, and the
|
||||
shared infrastructure the mesh itself provisions against.
|
||||
|
||||
Creating an application directory in the monorepo is a convention violation, and reviewers
|
||||
reject it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An application's cadence is its own. It is not reviewed as mesh code and does not queue
|
||||
behind mesh work.
|
||||
- The separation is safe **only because** the pipeline and provisioning are identical either
|
||||
side of it — which [ADR 0014](0014-no-npm-workspace.md) is what makes true. Without
|
||||
standalone packages this decision would fork the build.
|
||||
- The monorepo stops being an inventory of the installation, which is a precondition for
|
||||
publishing anything about it.
|
||||
- Discovery gets harder: there is no single listing of everything the mesh runs, and the
|
||||
registry of build sources becomes the closest thing to one.
|
||||
- A module source that is not registered is silently skipped by the pipeline. The cost of
|
||||
being outside the monorepo is that being forgotten is possible.
|
||||
|
||||
## References
|
||||
|
||||
- The rule is stated in the governed constitution page authored 2026-07-10, §3, as a
|
||||
convention violation reviewers must reject.
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) decision 4 extends this from *new*
|
||||
applications to the modules already in the monorepo.
|
||||
- Knowledge base: `troubleshooting/unregistered-module-source`.
|
||||
@@ -1,92 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 16. The lab
|
||||
|
||||
*Consolidated 2026-08-28 from five records. The lab is one design and was split across five
|
||||
decisions taken over three days; the reasoning is kept, the fragmentation is not.*
|
||||
|
||||
The environment a change is run against before it reaches real machines.
|
||||
|
||||
> **Still the lab, no longer the test bed — 2026-09-30, by [ADR 0149](0149-the-live-mesh-is-the-test-bed.md).**
|
||||
> Everything here stands. What changed is what the lab is *for*: a change is verified against the mesh
|
||||
> that is running, because the faults that cost the most are faults of a mesh that already exists —
|
||||
> bound consumers, containers made against an older roster, an adopted machine — and a bed is by
|
||||
> construction a mesh that does not. Raising a mesh from bare is now the lab's whole job, which is the
|
||||
> one thing the live mesh cannot be asked to do. 0149 also supersedes
|
||||
> [ADR 0068](0068-the-lab-takes-requests.md), which extended this one and was never built.
|
||||
|
||||
## A node in the lab is a virtual machine
|
||||
|
||||
It boots a stock Linux image, runs the real install, and becomes a node. **It is not a model of
|
||||
a node**, so no question arises about how good the model is — which is the whole reason for
|
||||
paying the cost of virtual machines rather than containers.
|
||||
|
||||
The lab is driven by **incus**, and a scenario is raised from a declaration.
|
||||
|
||||
## A router is scenery, and is therefore a container
|
||||
|
||||
**Nothing under test runs on a router.** It is not a participant, holds no identity, has nothing
|
||||
installed on it by the mesh, and no assertion is ever made about its internals. It exists so that
|
||||
packets between machines behave the way they behave in the world.
|
||||
|
||||
The fidelity argument that makes a node a virtual machine does not reach it: what a router *is*
|
||||
does not matter, only what it *does to traffic*. So a router is a system container, and the lab
|
||||
is cheaper for it.
|
||||
|
||||
## A scenario declares the underlay, and only the underlay
|
||||
|
||||
**What a hosting provider and a home router would have provided**, before any of our software
|
||||
touched the machine:
|
||||
|
||||
- which segments exist, and their address ranges
|
||||
- which machine sits on which segment, at which address
|
||||
- what NAT sits between them, and which ports are forwarded through it
|
||||
- which machines are detached, and may be attached or detached during a run
|
||||
|
||||
**A scenario declares nothing about the overlay** — no overlay addresses, no hub, no peering, no
|
||||
names, no certificates. Those are the mesh's job, and a scenario that supplied them would be
|
||||
testing itself.
|
||||
|
||||
> A scenario provides what a hosting provider and a home router would provide, and nothing our
|
||||
> software is responsible for.
|
||||
|
||||
## A scenario is a closed address space
|
||||
|
||||
Every segment materialises as its own isolated link belonging to one scenario instance. **Two
|
||||
scenarios raised from the same declaration hold the same addresses and never meet**, because
|
||||
nothing joins their links. The declaration therefore keeps its literal addresses and they mean
|
||||
exactly what they say.
|
||||
|
||||
**The consequence that constrains everything else: the lab never reaches into a scenario over
|
||||
IP.** It talks to a machine through the virtualisation layer's own channel — the way one would
|
||||
use a console rather than the network. That is what makes two identical scenarios able to run at
|
||||
once, and it is why placing anything inside a machine is a hypervisor operation rather than a
|
||||
network one.
|
||||
|
||||
## Two scenario classes, and the first has no pipeline
|
||||
|
||||
| | **bootstrap** | **full** |
|
||||
|---|---|---|
|
||||
| contains | machines, the host binary, a pinned substrate bundle | a complete mesh: forge, coordinator, delivery, modules |
|
||||
| verdict from | what the host reports about the state it reconciled | a delivery result ending in verification |
|
||||
| exercises | tiers 0 and 1 | tiers 2 and above, and modules |
|
||||
|
||||
**The bootstrap class comes first**, because it is what develops the node host, and because a
|
||||
full scenario needs tiers that do not exist yet. A lab that could only raise the larger class
|
||||
would be a lab nobody could use until everything else was built.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The lab tests the real code path**, not a reimplementation of it. The network a scenario
|
||||
produces is generated by the same code production runs.
|
||||
- **Isolation is what makes it usable in parallel**, and it costs the ability to reach in over
|
||||
IP. Everything the lab puts inside a machine — a binary, an image, a file — goes through the
|
||||
hypervisor.
|
||||
- **A sealed scenario cannot fetch anything**, which is a real limit rather than an inconvenience:
|
||||
it is why images have to be placed and why a container runtime has to be in the base image.
|
||||
@@ -1,92 +0,0 @@
|
||||
---
|
||||
topic: checking it
|
||||
status: accepted
|
||||
date: 2026-08-24
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 17. A test defends a decision
|
||||
|
||||
## Context
|
||||
|
||||
[`how-we-build.md`](../00-META/how-we-build.md) §5 already says that **if a document states a
|
||||
rule about the mesh, it says how the rule is verified**, on the grounds that an unenforced rule
|
||||
is indistinguishable from a wrong one and costs more, because people believe it.
|
||||
|
||||
That rule is applied to prose and to acceptance criteria. It has never been applied to
|
||||
**decisions**, and it should be — a decision record states something that must be true, which
|
||||
is the same kind of claim.
|
||||
|
||||
The gap was found by review. The lab reached 2,128 lines with 1,072 of them untested, and
|
||||
**no stated rule was broken.** There is no testing posture in `how-we-build.md` at all: no
|
||||
expectation, no gate, no definition of done. Every decision the lab embodies — the underlay
|
||||
boundary, the closed address space, routers as scenery, waiting for usable rather than for a
|
||||
call to return — was verified by hand, by running scenarios and reading output, and none of
|
||||
that survives the terminal it was run in.
|
||||
|
||||
Which is the fault the mesh already has catalogued at scale: an end-to-end harness that has
|
||||
not built since 2026-06-04, and nothing said so
|
||||
([`04-ISSUES/005`](../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). Coverage
|
||||
assumed rather than checked.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **A coverage percentage.** Rejected. It measures how much code a test touched, not whether
|
||||
anything important is defended, and it is satisfied by tests that assert nothing. A number
|
||||
would have been met by testing the parser harder while the hypervisor integration stayed
|
||||
unasserted.
|
||||
2. **Test-driven development as a hard rule.** Rejected, and not because it is wrong in general.
|
||||
Half the lab's implementation was discovery: that the hypervisor CLI reads a definition from
|
||||
stdin and hangs, that it assigns a MAC without recording it, that a stock image's boot-time
|
||||
networking flushes a static address. A test written first against undiscovered behaviour
|
||||
asserts a guess.
|
||||
3. **A test defends a decision.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**Every decision record states something that must be true. A test asserts it.**
|
||||
|
||||
A decision with no test is a decision that will quietly stop being true, and nobody will find
|
||||
out from a document. Concretely:
|
||||
|
||||
- Where a decision is about **structure or logic** — what a declaration may say, what is
|
||||
refused, how a name is derived — the test is a unit test, and it is **written first**, because
|
||||
the behaviour is knowable before the code.
|
||||
- Where a decision is about **behaviour against a real system** — a hypervisor, a broker, a
|
||||
daemon — the test runs against the real thing, and is written **alongside**, because the
|
||||
behaviour is discovered rather than known.
|
||||
- **Mocking the boundary is forbidden.** A test that fakes a hypervisor asserts that the fake
|
||||
behaves as expected, which is the shape of test this whole effort exists to stop shipping.
|
||||
- **The gate is blocking, and green is the definition of done.** A change that has not run its
|
||||
tests is not finished, whatever its diff looks like.
|
||||
|
||||
A test names the decision it defends. Not as ceremony: it is what makes the pairing checkable,
|
||||
so a decision without one can be *found* rather than noticed.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The question *"which tests matter"* has an answer that is not a number. The decisions are the
|
||||
list, and they are already written down.
|
||||
- **A new decision costs a test.** That is the intended friction — a decision nobody will assert
|
||||
is one worth reconsidering.
|
||||
- Integration tests need real infrastructure and are slow. That cost is accepted: a fast test
|
||||
suite that mocks the boundary would tell us nothing about the boundary, which is where every
|
||||
interesting fault in this session actually was.
|
||||
- Some decisions are not mechanically assertable — *the mesh brokers capabilities; nodes host;
|
||||
agents think* is a shape, not a predicate. Those should say so in the record rather than being
|
||||
quietly exempt, so the exemption is visible.
|
||||
- Records 0001–0033 were made before this rule. They are not retroactively invalid, but each
|
||||
should acquire a test or an explicit note that it cannot have one, and until then this rule
|
||||
is aspirational for them — which is exactly the state §5 warns about, recorded rather than
|
||||
hidden.
|
||||
|
||||
## References
|
||||
|
||||
- [`how-we-build.md`](../00-META/how-we-build.md) §5 — the rule this extends from prose to
|
||||
decisions.
|
||||
- [`04-ISSUES/005`](../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md) — coverage
|
||||
assumed rather than checked, for two and a half months.
|
||||
- The sibling HQ repository for the PAPA platform states the same boundary rule — the contract
|
||||
is tested against the real system, mocking the client is forbidden, and a blocking gate is the
|
||||
definition of done. This record adopts that posture and adds the decision pairing.
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
topic: checking it
|
||||
status: accepted
|
||||
date: 2026-08-24
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0017-a-test-defends-a-decision.md
|
||||
---
|
||||
|
||||
# 18. A picture of a system is read from the system, never from what asked for it
|
||||
|
||||
## Context
|
||||
|
||||
A scenario declaration is a file. A raised scenario is a set of machines, links and rulesets.
|
||||
The two are supposed to correspond, and the entire value of the lab rests on noticing when
|
||||
they do not — [ADR 0017](0017-a-test-defends-a-decision.md) says a claim nothing checks is a
|
||||
claim that will quietly stop being true.
|
||||
|
||||
Drawing a scenario makes that concrete, and forces a choice that looks cosmetic and is not.
|
||||
A diagram of a running system can be produced two ways: parse the declaration and lay it out,
|
||||
or interrogate the system and lay *that* out. The first is far easier — the declaration is
|
||||
already parsed, already validated, already in memory.
|
||||
|
||||
It is also worthless for the only question worth asking of such a picture: *is what is running
|
||||
what I asked for?* A drawing built from the request and captioned **as raised** answers that
|
||||
question with the request, which always agrees with itself.
|
||||
|
||||
This is the same fault as
|
||||
[04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) — a firewall key declared in
|
||||
five manifests and read by no code, so a manifest appears to restrict a port and restricts
|
||||
nothing. The declaration was never wrong. Nothing ever asked the system.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Draw the declaration, and label it honestly.** Cheap, and useful for review before
|
||||
anything is raised. Insufficient alone: it can never disagree with itself.
|
||||
2. **Draw the system, inferring the rest from the declaration where the system is silent.**
|
||||
The tempting middle. Rejected — a picture where some facts are observed and some are
|
||||
assumed has no honest caption, and the assumed ones are exactly the interesting ones.
|
||||
3. **Two pictures, one layout, neither borrowing from the other.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**A picture captioned *as raised* reads only the running system.** It never opens the
|
||||
declaration, not even for a fact the system happens not to record.
|
||||
|
||||
Where the hypervisor does not natively hold a fact the picture needs — whether a segment is
|
||||
public, what a gateway translates, whether a machine refuses inbound — **the raise records it
|
||||
on the resource** as metadata, and the picture reads it back from there.
|
||||
|
||||
That recording carries its own rule, which is the substance of this decision rather than an
|
||||
implementation note:
|
||||
|
||||
> **A behavioural tag is written after the behaviour works, never when the resource is
|
||||
> created.**
|
||||
|
||||
Written at creation, a tag restates the request in a new location and inherits none of the
|
||||
authority of having happened. A failed raise deliberately leaves its wreckage standing, so a
|
||||
tag written up front would let a picture of that wreckage badge translation the gateway was
|
||||
never configured to do — reproducing, inside the tool built to catch the fault, exactly the
|
||||
fault.
|
||||
|
||||
So: the gateway is tagged after its ruleset applies; the machine after the read-back proves
|
||||
its firewall loaded.
|
||||
|
||||
Both pictures render through **one layout**, so they can be put side by side and the
|
||||
difference read off directly.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **It earned itself on the first comparison.** Drawn side by side, every virtual machine in
|
||||
the live picture held no addresses at all. A container's interface carries the name of the
|
||||
device it was configured as; a virtual machine names its own — so joining addresses to
|
||||
devices by name attached every address to a container and none to a VM. Nothing failed;
|
||||
a whole class of machine silently lost its addresses. The two pictures disagreed, so it was
|
||||
visible in seconds. It is now joined on MAC.
|
||||
- Raise does more work, and writes metadata it does not itself consume. Accepted: the cost is
|
||||
a few config keys, and it is what makes a raised instance self-describing.
|
||||
- A resource raised before a tag existed is missing it. The reader says so rather than filling
|
||||
the gap from the declaration — an untagged link draws as unknown, not as what the file said
|
||||
it should be.
|
||||
- **The rule generalises past diagrams.** Anything reporting on the mesh — a status view, an
|
||||
inventory, a health check — is subject to it. A report assembled from the intended state is
|
||||
not a report.
|
||||
- The declared picture stays, and stays useful: it is review before raising, and it is one half
|
||||
of the comparison. It carries no runtime status, because it cannot know any.
|
||||
|
||||
- **The constitution sync is now owed.** Accepted 2026-08-25, so §5 carries the rule
|
||||
unqualified and playbook [05](../00-META/process/05-constitution-sync.md) is due. Until it
|
||||
runs, the mesh does not enforce this — an unsynced rule is a rule the mesh does not enforce,
|
||||
whatever this document says.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0017](0017-a-test-defends-a-decision.md) — a claim nothing checks stops being true.
|
||||
- [ADR 0016](0016-the-lab.md) — why the lab must not supply what the
|
||||
mesh is responsible for; the same instinct, applied to facts rather than to configuration.
|
||||
- [04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) — the fault in production
|
||||
form.
|
||||
@@ -1,158 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-28
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 19. How this repository works
|
||||
|
||||
> **Superseded in part — 2026-10-11, by [ADR 0291](0291-an-issue-lives-whole-in-the-issue-tracker.md).**
|
||||
> `04-ISSUES` is no longer a part of this repository: an issue lives whole in the issue tracker, its report,
|
||||
> diagnosis and resolution included, and the folders are imported there and removed (issue 427). The other three
|
||||
> folders, and everything else here, stand.
|
||||
|
||||
> **Superseded in part — 2026-10-07, by [ADR 0248](0248-the-decision-index-is-generated-when-it-is-read-and-never-stored.md).**
|
||||
> The paragraph "the index is written, not only generated on demand" no longer holds: a stored list was
|
||||
> a line every decision pull request added to one file, so two open at once always conflicted (issue
|
||||
> 297). The list of records is generated when it is read, and `02-DECISIONS/README.md` writes only the
|
||||
> topics in their reading order. Everything else here stands, including the reading order and the check
|
||||
> that every record has a topic one of the README's.
|
||||
|
||||
*Consolidated 2026-08-28 from ten records that were one decision seen from ten angles. The
|
||||
reasoning is kept; the fragmentation is not.*
|
||||
|
||||
## The repository
|
||||
|
||||
**`novox/hq` is Novox's headquarters, and it is public.**
|
||||
|
||||
Company-scoped, not the mesh's. Today almost everything in it is about the mesh, because the
|
||||
mesh is what Novox is building — a fact about the present rather than a definition. A second
|
||||
product would live here too.
|
||||
|
||||
**Public** means written for a reader who is not its author and has no access to the mesh it
|
||||
describes. Nothing here may contain routable addresses, real domain names, node names, absolute
|
||||
paths, usernames or credentials. The test: *would this paragraph still teach a stranger running
|
||||
an entirely different mesh?*
|
||||
|
||||
**Separate from the code** because the cadence differs — a decision changes when thinking
|
||||
changes, not when code changes — and because a public repository cannot be a private one's
|
||||
subdirectory.
|
||||
|
||||
**The naming rule:** a repository belonging to a product carries that product's prefix. A
|
||||
company-scoped one does not. So this is `hq` and the mesh's are `mesh-*`.
|
||||
|
||||
| Repository | Tier | Holds |
|
||||
|---|---|---|
|
||||
| `novox/mesh-host` | 0 | the node host — the one binary installed by hand |
|
||||
| `novox/mesh-substrate` | 1 | the pinned tier-1 services, as declarations |
|
||||
| `novox/mesh-control` | 2 | the control plane and its contexts |
|
||||
| `novox/mesh-surfaces` | 3 | tools, web, cli — thin, no logic |
|
||||
| `novox/mesh-sdk` | — | the mesh's own domain ([ADR 0009](0009-modules-and-the-graph.md)) |
|
||||
| `novox/mesh-lab` | — | the lab: scenario lifecycle, networking, placement |
|
||||
| `novox/hq` | — | this one |
|
||||
|
||||
**The product is `Novox Mesh`**, shortened to `mesh` in internal use — repository names, the
|
||||
module namespace, environment variables, paths. **`Nox` is an identity of Novox**, an agent
|
||||
participant within the mesh's own model, not a second system.
|
||||
|
||||
## The folders, and why they are numbered
|
||||
|
||||
**The numbering is the flow.** Research produces a decision; the decision authorises a design.
|
||||
Following the numbers walks the process in the order it happens.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `01-RESEARCH` | an open question, while it is open |
|
||||
| `02-DECISIONS` | what was decided, and why |
|
||||
| `03-DESIGN` | what is being built |
|
||||
| `04-ISSUES` | something wrong at the level of design or governance |
|
||||
|
||||
**`03-DESIGN` has two layers and they are never mixed.** `00-as-is/` describes the mesh that
|
||||
exists, written from the implementation and the operational record. `01-to-be/` describes the one
|
||||
being built toward. Every document says which it is. A statement about the future does not belong
|
||||
in an as-is document, and an as-is document is never edited to describe an intention.
|
||||
|
||||
**`04-ISSUES` is for design-level faults** — a rule enforced by nothing, a stated invariant that
|
||||
is false, a failure the design permits to be silent. Not an operational ticket queue.
|
||||
|
||||
## What a decision record is, and is not
|
||||
|
||||
**If a decision is worth recording, it is worth a record. If it is not worth a record, it is not
|
||||
recorded.**
|
||||
|
||||
That bar has been read too generously. A *finding* is not a decision. A bug is not a decision.
|
||||
**A record is warranted when there is a genuine fork**: a direction reversed, an alternative
|
||||
seriously considered and likely to be proposed again, or something contested that needs to stay
|
||||
settled. Everything else belongs in the design document, where the reasoning is read.
|
||||
|
||||
**There is no ledger** — no separate document summarising, ranking or tracking decisions. A
|
||||
chronological view is generated from frontmatter, which is what a ledger was actually for.
|
||||
|
||||
**A number identifies a record and never changes.** It is not a position, and it cannot be
|
||||
both — a position moves when the set changes, and an identity that moves is not one.
|
||||
|
||||
That is not a preference. Records are referenced from **outside** this repository: code
|
||||
comments, commit messages, the knowledge base. Renumbering once cost 96 references across two
|
||||
code repositories, and nothing in either would have failed to compile — the comments would
|
||||
simply have pointed at the wrong reasoning, which is worse than a broken link because nothing
|
||||
reports it.
|
||||
|
||||
**So the reading order lives in a generated index**, from each record's `topic:` — what the mesh
|
||||
is, then its tiers from the bottom up, then what runs on them and how it gets there, then how it
|
||||
is built, how it is checked, and how we work.
|
||||
|
||||
**And the index is written, not only generated on demand.** A reader looking at the folder on a
|
||||
forge sees the folder, not a command. The objection to a written index is that it drifts, and
|
||||
that is answered by **checking** it rather than by refusing to write one — which is §5's own
|
||||
rule: a rule states how it is checked. A record with no topic, or a topic nobody defined, fails
|
||||
the same check, because the quiet failure is a record that vanishes from the order rather than
|
||||
appearing in the wrong place.
|
||||
|
||||
**The design layer is what you read.** These records explain *why* a thing is as it is. They are
|
||||
not a description of the system, and needing to read them to understand it would mean the design
|
||||
documents had failed.
|
||||
|
||||
## Status, and views over it
|
||||
|
||||
**Every document carries its state in YAML frontmatter** — research overviews, design documents,
|
||||
decision records, issue reports.
|
||||
|
||||
> **Progressive insight — 2026-10-10.** For issue reports this stopped holding the day slice 3 of
|
||||
> [ADR 0275](0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md)
|
||||
> landed: an issue's status, owner and fix live in the issue tracker, a module of the mesh, which refuses
|
||||
> a wrong status at the write; a report here carries the day it was opened and nothing of its status, and
|
||||
> `cycle.py` refuses a status key in a report opened after the import. The reports written before keep
|
||||
> their keys as a dated snapshot. The reason stands exactly as written — one place, never two — and the
|
||||
> one place for an issue is now the tracker, because a file every branch may edit was the place where
|
||||
> numbers collided and statuses were hard to see. Research overviews, design documents and decision
|
||||
> records carry their state here as before.
|
||||
|
||||
> **Progressive insight — 2026-10-10, later.** The reports written before no longer keep their keys: once
|
||||
> the import was checked against every folder, one pull request removed them (ADR 0275's last note), and
|
||||
> `cycle.py` refuses a status key in any report.
|
||||
|
||||
**There are no central status files.** Every cross-cutting view — a status matrix, a decision
|
||||
index, an open-issue list — is generated from frontmatter when asked for, never written to disk.
|
||||
Two places holding one fact drift, and the written one wins by being closer to hand.
|
||||
|
||||
**Prose does not restate status.** One place, and two is one too many.
|
||||
|
||||
## Workflows are playbooks
|
||||
|
||||
Every workflow is a playbook in [`00-META/process/`](../00-META/process/): trigger, who runs it,
|
||||
steps, outputs. People and agents follow the same ones, and **agents do not act outside them**.
|
||||
|
||||
Each is wrapped by a thin skill that defers to the playbook as authoritative and adds only the
|
||||
mechanical scaffolding — so the process has one definition rather than a document and an
|
||||
implementation that disagree.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A reader has one place per thing.** The design layer describes the system; these records say
|
||||
why; the playbooks say how work is done.
|
||||
- **Records will accumulate more slowly**, because the bar is a fork rather than a finding. This
|
||||
record is itself the correction: ten records became one because they were one decision.
|
||||
- **The public rule constrains everything written here**, permanently and at every commit. It is
|
||||
the reason research describes real observations without identifying the mesh it observed.
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-07-10
|
||||
deciders: jochen
|
||||
reconstructed: true
|
||||
---
|
||||
|
||||
# 20. The mesh is governed by a constitution, injected where work is decided
|
||||
|
||||
> Reconstructed after the fact from the evidence cited below.
|
||||
|
||||
## Context
|
||||
|
||||
By mid-2026 the mesh was doing a large share of its own design and implementation work through
|
||||
agents. The rules those agents were expected to follow existed — in operating instructions, in
|
||||
convention documents, in the knowledge base — but they were **retrieved**: an agent had to know
|
||||
a rule existed in order to look it up.
|
||||
|
||||
Rules that must be looked up are followed by whoever already knows them, which is precisely the
|
||||
population that does not need them. The rules being violated were the ones nobody thought to
|
||||
search for.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Documentation plus review.** Rejected — it is what existed. Review catches a violation
|
||||
after the work is done, and only if the reviewer knows the rule.
|
||||
2. **Lint and automated checks only.** Rejected as insufficient, not wrong. A check catches
|
||||
what can be expressed mechanically; most of these rules are about judgement — what belongs
|
||||
in a repository, when a criterion counts as verified.
|
||||
3. **A canonical rule set, injected into context wherever work is decided, with a check phase
|
||||
before output is accepted.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
A single canonical document states the mesh's non-negotiable rules. It is **injected
|
||||
proactively** into every eligible design and analysis session — agents do not fetch it, it
|
||||
arrives — and a check phase verifies the session's output against it before the work proceeds.
|
||||
|
||||
It is a governed document, not a page. Changing it requires a proposal, sign-off by reviewers
|
||||
who are not the proposer, and a recorded decision. Drive-by edits are reverted.
|
||||
|
||||
Scoped override pages may **tighten** it for a team or product. They may never relax it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A rule reaches the work whether or not anyone remembered it existed.
|
||||
- The check phase makes a violation a blocking outcome rather than a review comment.
|
||||
- Two copies of the same rules now exist: this document, and the reasoning in HQ that earned
|
||||
them. The enforced copy wins by default, so the reasoned copy quietly stops being true —
|
||||
which is why [`00-META/how-we-build.md`](../00-META/how-we-build.md) is now the source
|
||||
and the governed page is derived from it, via playbook
|
||||
[`05-constitution-sync.md`](../00-META/process/05-constitution-sync.md).
|
||||
- Injection costs context on every eligible turn, and grows with the document. Nothing
|
||||
currently bounds that.
|
||||
- The amendment process requires two reviewers, which a mesh with one human operator satisfies
|
||||
only by counting agents. That tension is real and unresolved.
|
||||
|
||||
## References
|
||||
|
||||
- The governed page was authored 2026-07-10 and carries its own amendment process.
|
||||
- `feat(noxflow): HAL architectural conformance gate for reviewer + architect` (#296),
|
||||
2026-06-10 — the check phase, predating the document it checks against.
|
||||
- Knowledge base: `platform/constitution`.
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-23
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 21. HQ is the source of the mesh constitution
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0020](0020-the-mesh-is-governed-by-a-constitution.md) established a canonical rule set,
|
||||
injected into every eligible design session and checked before output is accepted. It lives in
|
||||
the knowledge base, where the orchestrator reads it.
|
||||
|
||||
HQ separately carried a document stating overlapping rules with the reasoning that earned each
|
||||
one. Two texts, one enforced and one not.
|
||||
|
||||
That arrangement has a predictable outcome and it is not a tie. The enforced copy wins by
|
||||
default, because it is the one that blocks work. The reasoned copy quietly stops being true,
|
||||
and the rules survive without the incidents that justify them — at which point a rule reads as
|
||||
arbitrary, and an arbitrary rule is the kind people route around.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **The knowledge-base page is the source; HQ points at it.** Rejected, though it is the
|
||||
honest description of what was already happening. It leaves the reasoning downstream of the
|
||||
rule, and the reasoning is the part that makes a rule survive a challenge.
|
||||
2. **Accept the overlap and let both stand.** Rejected: two authorities is no authority, and
|
||||
the drift is silent.
|
||||
3. **HQ is the source; the governed page is derived and published from it.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
[`00-META/how-we-build.md`](../00-META/how-we-build.md) is the source. The governed page the
|
||||
mesh injects is **derived** from it — the rules without the reasoning — and is never edited
|
||||
directly.
|
||||
|
||||
Publishing is a playbook step, not a manual act, and it ends with **reading the page back and
|
||||
verifying the change is present**. A publish that reported success and did nothing is exactly
|
||||
the failure class this mesh keeps producing
|
||||
([ADR 0010](0010-delivery.md)).
|
||||
|
||||
Section numbering is stable, because the orchestrator and the review fragments cite sections by
|
||||
number.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One source, many surfaces — the same argument HQ's separation already rests on
|
||||
([ADR 0019](0019-how-this-repository-works.md)), applied to the rules themselves.
|
||||
- Each rule keeps the incident that earned it, in a place that is reviewed as a diff.
|
||||
- An edit to the derived page survives until the next sync and then vanishes. The playbook says
|
||||
so, and nothing mechanically prevents it.
|
||||
- **The sync is manual and is the weak point.** An unsynced rule is a rule the mesh does not
|
||||
enforce, whatever the source says — so the playbook requires the failure to be stated rather
|
||||
than passed over. This is the same class of gap as
|
||||
[`04-ISSUES/006`](../04-ISSUES/006-hq-is-not-indexed-into-the-knowledge-base/00-report.md),
|
||||
and it is worth watching for the same reason.
|
||||
- The document grew from four rules to seven sections, because it now has to carry everything
|
||||
the mesh enforces rather than only what someone thought to write down.
|
||||
|
||||
## References
|
||||
|
||||
- [`00-META/process/05-constitution-sync.md`](../00-META/process/05-constitution-sync.md) —
|
||||
the sync, including the read-back.
|
||||
- [ADR 0020](0020-the-mesh-is-governed-by-a-constitution.md) — the governed page and why it
|
||||
exists.
|
||||
@@ -1,105 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-26
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 22. The constitution absorbs what is already enforced
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0021](0021-hq-is-the-source-of-the-constitution.md) makes this repository the source
|
||||
and the knowledge base a derived copy, and playbook
|
||||
[05](../00-META/process/05-constitution-sync.md) publishes the copy whenever a rule changes.
|
||||
|
||||
Three rules were accepted on 2026-08-25 and the sync came due. Reading the target before
|
||||
overwriting it — as [§2](../00-META/how-we-build.md) requires — found the two documents had
|
||||
diverged in a way nobody had recorded, and that a literal republish would have **deleted rules
|
||||
the mesh currently enforces**.
|
||||
|
||||
| | source (`how-we-build.md`) | enforced page |
|
||||
|---|---|---|
|
||||
| §4 | naming and boundaries | **code quality** — structure, layering, types, restraint |
|
||||
| §5 | four evidence rules | one runtime-evidence rule |
|
||||
| §6 | review by a non-proposer | a design meeting with **two node operators** |
|
||||
|
||||
The enforced page was last written 2026-07-10, is agent-authored, and its code-quality section
|
||||
appears in no decision record anywhere. It has been checked against for six weeks regardless.
|
||||
|
||||
Two facts made this urgent rather than tidy:
|
||||
|
||||
**Removal is not the safe option.** The orchestrator reads *"when absent, no constitution is
|
||||
injected (backward-compatible)"* — so deleting the page would not fail, it would silently
|
||||
inject nothing, and every design meeting would run unchecked with no error anywhere. Three
|
||||
unenforced rules would become all of them.
|
||||
|
||||
**The stricter review bar has never been met.** The enforced page requires two node operators,
|
||||
neither the proposer. There is one operator. Every amendment ever made violated it.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Republish wholesale.** What the playbook literally says. Rejected — it deletes the
|
||||
code-quality rules, which exist nowhere else.
|
||||
2. **Merge surgically** — update §2, §3, §5 in the copy and leave §4 alone. Nothing is lost and
|
||||
it is quick, but the source becomes authoritative for some sections and the copy for others,
|
||||
which is the drift being fixed.
|
||||
3. **Absorb, then republish.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**What the mesh enforces and cannot cite is written down here first, then republished.**
|
||||
|
||||
The code-quality rules become §8 rather than §4, because appending renumbers nothing and every
|
||||
existing citation stays valid.
|
||||
|
||||
**They are recorded as inherited, and marked as such.** Every other rule in the document states
|
||||
the incident or measurement that earned it. These state nothing, because nothing was ever
|
||||
written down. Importing them silently would have made the document claim a provenance it does
|
||||
not have, and this repository's whole argument is that the reasoning is the expensive half.
|
||||
|
||||
**The review bar is resolved in favour of what is achievable.** A rule requiring two operators
|
||||
where there is one is not a high standard, it is a rule everything silently violates — the same
|
||||
shape as a firewall key declared in five manifests and read by no code
|
||||
([04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md)). Review means
|
||||
a person who is not the proposer, which is both achievable and what has been practised.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The copy can now be a faithful projection**, which is what makes "the source is the source"
|
||||
true rather than aspirational.
|
||||
- **§8 has no reasoning behind it and says so.** Anyone may propose replacing an inherited rule
|
||||
with an earned one; until then the marking is the honest state. It is also a standing
|
||||
invitation to delete any of them that turn out to serve nothing.
|
||||
- **Six weeks of drift went unnoticed** because nothing compares the two. The sync is manual and
|
||||
runs only when someone remembers a rule changed. **Nothing detects divergence**, and this
|
||||
record does not fix that — it is worth an issue.
|
||||
- **The operator's instinct to remove the copy was right about the direction and wrong about the
|
||||
method.** The end state where the mesh reads this repository directly — rendered and operable
|
||||
through the board — removes the second copy rather than blanking it. That is not designed and
|
||||
is not decided here.
|
||||
- **Section numbering is now append-only in practice.** §8 sits after the overrides section
|
||||
purely because renumbering would break citations, which is a cost of the copy existing at all.
|
||||
|
||||
## Sync
|
||||
|
||||
Run 2026-08-26, and the read-back earned its place in the playbook.
|
||||
|
||||
**The first publish reported success and changed nothing.** It created a new revision and
|
||||
updated the title, and the body did not apply — a malformed argument was dropped silently. Had
|
||||
the sync been marked done on the strength of the tool saying *"updated; new revision created"*,
|
||||
three accepted rules would have gone unenforced while every record said otherwise, and nothing
|
||||
would ever have contradicted it.
|
||||
|
||||
That is §5 demonstrating itself during its own publication: a green result proves transport,
|
||||
not effect. The republish was verified by reading the rule back out of the live page, not by
|
||||
trusting the second success message either.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0021](0021-hq-is-the-source-of-the-constitution.md) — source and copy.
|
||||
- [ADR 0020](0020-the-mesh-is-governed-by-a-constitution.md) — why the copy is injected at all.
|
||||
- [Playbook 05](../00-META/process/05-constitution-sync.md) — the sync this record interrupts.
|
||||
- [ADR 0017](0017-a-test-defends-a-decision.md), [ADR 0018](0018-a-picture-is-read-from-what-runs.md),
|
||||
[ADR 0012](0012-the-mesh-creates-no-symlinks.md) — the three rules whose sync surfaced this.
|
||||
@@ -1,78 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-26
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 23. The approval is the checkpoint, not the second pair of hands
|
||||
|
||||
## Context
|
||||
|
||||
[§2](../00-META/how-we-build.md) reads *"one change per pull request, and never merge your
|
||||
own"*, and gives the reason: *"self-merging removes the checkpoint that is the entire point."*
|
||||
|
||||
The rule was written for people. Applied literally to an agent it produces a contradiction
|
||||
that surfaced immediately: an agent asked to merge cannot merge, because it authored what it
|
||||
is being asked to merge. Every merge this repository has seen was therefore either performed
|
||||
by the thing that wrote it, or not performed at all.
|
||||
|
||||
Stated by the operator: *"you are allowed to self-merge, after I've given you explicit
|
||||
approval."*
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Read it literally.** An agent never merges; the operator merges by hand every time.
|
||||
Rejected: it makes the operator a mechanical step rather than a reviewer, and a step
|
||||
performed dozens of times without reading is not a checkpoint — it is the *shape* of one,
|
||||
which is worse, because the record then claims a review that did not happen.
|
||||
2. **Drop the rule for agents.** Rejected: the rule is right, and the failure it prevents —
|
||||
work merged with nobody having looked — is not one an agent is less prone to.
|
||||
3. **Require notification and approval, and stop there.** Chosen. Who performs the merge is
|
||||
not the thing worth constraining.
|
||||
|
||||
## Decision
|
||||
|
||||
**Every merge into the main branch is notified and approved.** Stated by the operator in
|
||||
exactly those terms, and the whole of the rule.
|
||||
|
||||
Notified: the merge is proposed and said out loud, not performed and mentioned. Approved: a
|
||||
person says yes to *that merge*. Who then performs it does not matter, which is what makes an
|
||||
agent merging its own work unremarkable — the checkpoint already happened.
|
||||
|
||||
**What approval is not**, because this is the half that can rot:
|
||||
|
||||
- A standing permission granted once and cited forever.
|
||||
- An instruction to do the work, read as approval to merge it.
|
||||
- Silence.
|
||||
- The author's own judgement that the work is ready.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The record must show which it was.** A merge performed on approval and a merge performed
|
||||
unilaterally look identical afterwards, and the difference is the whole rule. Until the forge
|
||||
can record an approval, the merge commit says so — which is weaker than a recorded review and
|
||||
is stated here rather than left to be discovered.
|
||||
- **Branch protection would make this structural instead of textual.** Required approvals turn
|
||||
the rule into something the forge enforces. It cannot be set through the API on the forge
|
||||
version in use, so this remains a rule held by intention — the condition
|
||||
[§5](../00-META/how-we-build.md) exists to name.
|
||||
- **The one-change-per-pull-request half is untouched**, and was broken twice on 2026-08-26.
|
||||
Both breaches are recorded in the pull requests themselves rather than in a conversation.
|
||||
- **This narrows a rule rather than relaxing one.** The obligation moves from *who performs the
|
||||
merge* to *whether a person decided*, which is a higher bar in the case the original wording
|
||||
actually permits: a reviewer who merges someone else's work without reading it satisfies the
|
||||
old rule and fails this one.
|
||||
|
||||
## Sync
|
||||
|
||||
Run and verified 2026-08-26. §2 in the enforced page now reads *never merge unapproved work*,
|
||||
confirmed by reading the rule back out of the live page rather than by trusting the publish —
|
||||
the previous sync reported success and changed nothing.
|
||||
|
||||
## References
|
||||
|
||||
- [`how-we-build.md`](../00-META/how-we-build.md) §2 — the rule, now carrying this.
|
||||
- [ADR 0010](0010-delivery.md) — the standard a checkpoint is held to: a
|
||||
step that reports success without doing anything is the fault, not the shortcut.
|
||||
@@ -1,95 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-30
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 24. Model access is a provision, and a licence is a thing with a name
|
||||
|
||||
## Context
|
||||
|
||||
Everything in this mesh that thinks needs a model, and there is more than one way to reach one:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **hosted services** | several vendors, each with its own account, quota and key |
|
||||
| **models the mesh runs itself** | open-weight models on a node with the hardware for them |
|
||||
|
||||
And the choice is **per consumer, deliberately**: a workstation's own session on one account, a
|
||||
laptop on the organisation's, two hired workers on the mesh's local model because their work does
|
||||
not justify paid tokens. Those are three different answers to one requirement, held at once, in
|
||||
one mesh.
|
||||
|
||||
**The existing system has the hard half of this already** — automatic licence refresh and
|
||||
switching between accounts when one is exhausted — and it works. It is not being replaced because
|
||||
it was wrong; it is being rebuilt because it lives in a place that cannot express the rest.
|
||||
|
||||
## Decision
|
||||
|
||||
**Model access is a provision.** A module that needs to think declares `requires: model-access`;
|
||||
anthropic, openai, grok and a locally-run model are four modules that provide it. That is
|
||||
[ADR 0009](0009-modules-and-the-graph.md)'s mechanism unchanged, and it buys the things that
|
||||
mechanism already buys: several implementations of one job, a refusal when more than one could
|
||||
answer, and choosing by assigning the one you want.
|
||||
|
||||
**A locally-run model needs nothing new at all.** It is a mesh-scoped provision on the node with
|
||||
the hardware — the same shape as a database, including the credential.
|
||||
|
||||
### A licence is a named thing, and the name is the operator's
|
||||
|
||||
Not an anonymous credential hanging off a provider. *The personal account*, *the organisation's
|
||||
account* — those are names a person uses, and the mesh has to use them too, because the whole
|
||||
point is saying **which one** a given consumer uses.
|
||||
|
||||
**Many to many.** One provider has several licences; one licence serves several consumers. So it
|
||||
is **not a claim** — claims are for things only one holder may have, and two machines sharing an
|
||||
account is the ordinary case rather than a collision.
|
||||
|
||||
### Four things this needs that the mesh does not have
|
||||
|
||||
Written as gaps rather than as design, because each is a real piece of work and pretending
|
||||
otherwise is how a plan becomes a surprise.
|
||||
|
||||
**1. A provider that is not on a node.** A mesh-scoped provision today is answered by *the machine
|
||||
running it*, and the reachability rule refuses two ends that share no private network. A hosted
|
||||
service is on nobody's machine and is reached over the public internet. That is a third scope —
|
||||
answered by a record rather than by a node — and the reachability rule must not apply to it.
|
||||
|
||||
**2. A secret the mesh is given rather than one it mints.** Every credential the mesh handles
|
||||
today it generated itself, sealed to both ends, and discarded. An API key arrives from a person.
|
||||
The missing verb is *accept*: take a value, seal it to each holder, and **discard the plaintext**
|
||||
— because a mesh that keeps operator-supplied keys readably is the arrangement this project
|
||||
[measured and rejected](0009-modules-and-the-graph.md).
|
||||
|
||||
**3. A consumer that is not a machine.** *This worker uses that licence* is a binding to an agent,
|
||||
not to a node. [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) already says an agent
|
||||
holds credentials and that delivery follows its node bindings and modality — so what is delivered
|
||||
still lands on a machine, and what is **chosen** is chosen per agent. The provisions model has no
|
||||
consumer identity other than a node.
|
||||
|
||||
**4. Switching is a reaction, not a declaration.** Everything here is desired state, reconciled by
|
||||
comparison. A licence that hits its limit and must be swapped is a response to something observed,
|
||||
and it cannot be expressed as a declaration without the declaration meaning *whatever is working
|
||||
right now* — which is not a thing anybody declared. **It belongs with observability, changing a
|
||||
binding**, and the binding is then declared as usual. Saying this plainly is what stops the
|
||||
declaration language growing a conditional.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The refusing rule applies here and will be felt.** A mesh holding three ways to reach a model
|
||||
refuses every consumer that has not said which — which is correct and is a great deal of
|
||||
saying-which the first time. The remedy is one assignment per consumer, and the message names
|
||||
the candidates.
|
||||
- **A licence outliving its holder is a live credential nobody is watching.** The same rule the
|
||||
provisioner follows applies: what the mesh granted and no longer grants is withdrawn.
|
||||
- **Nothing here makes a node authenticate to a model provider.** ADR 0001 holds: an agent does.
|
||||
What changes is that the mesh can now say *which agent, which licence, which node it lands on*,
|
||||
which is the fact ADR 0001 records as missing.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — provisions, scope, choosing, and sealed credentials
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) — agents hold credentials, not nodes;
|
||||
`hal/ai` as the context owning provider grants and rotation
|
||||
@@ -1,111 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0019-how-this-repository-works.md
|
||||
---
|
||||
|
||||
# 25. The design record is read where it is written, never copied to be found
|
||||
|
||||
## Context
|
||||
|
||||
**These documents cannot be found by searching the mesh's memory, and never could.** Checked on
|
||||
2026-08-23 and again on 2026-08-31, against both the symptom-indexed store and the structured
|
||||
archive, using a decision record's full title and a distinctive phrase from a design document: no
|
||||
result, no partial match, no stale copy.
|
||||
|
||||
That matters because of what was promised. The objection to giving this material its own
|
||||
repository was that the mesh already has a knowledge store, and a second one repeats the mistake
|
||||
that store was created to fix. **The answer offered was indexing rather than location** — that
|
||||
these documents would be returned beside everything else in a search, so where they were authored
|
||||
became a separate question. The indexing was never built.
|
||||
|
||||
**The claim has since stopped being load-bearing**, which is why this is a decision rather than an
|
||||
incident. [`README.md`](../README.md) names the gap in the place the claim used to sit, and
|
||||
[ADR 0019](0019-how-this-repository-works.md)'s reasoning rests on cadence, reviewers and scope —
|
||||
none of which depend on being searchable from elsewhere. What remained was an unbuilt capability
|
||||
and an open question, recorded as
|
||||
[`04-ISSUES/006`](../04-ISSUES/006-hq-is-not-indexed-into-the-knowledge-base/00-report.md).
|
||||
|
||||
**A signpost was added on 2026-08-31 and measured.** One entry in the mesh's memory naming what
|
||||
lives here and when to come looking. A search for *design records, decisions, repository* returns
|
||||
it; a search phrased the way somebody actually asks — *why is the mesh built this way* — returns
|
||||
nothing, because the store matches terms and not meaning. **Reachable is not the same as
|
||||
surfacing**, and the measurement is what established which one a signpost buys.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A one-way sync into the mesh's memory.** A job reads this repository on a schedule and writes
|
||||
the documents into the searchable store. It works with what exists today and needs nothing
|
||||
built first. **Rejected**, because it creates a second copy of every document, and the failure
|
||||
mode of a derived copy is the one this repository is least able to tolerate: *the copy that is
|
||||
searched quietly stops matching the copy that is edited*, and the enforced one wins. A design
|
||||
record that has silently diverged from the reasoning it claims to carry is worse than one that
|
||||
cannot be found — the first misleads, the second merely fails.
|
||||
|
||||
2. **Leave the signpost and close nothing.** Honest, free, and it keeps the gap visible.
|
||||
**Rejected as an end state**, though it is what stands until the option below exists. It
|
||||
answers only for a reader who already suspects these documents exist, which is precisely not
|
||||
the reader the mesh's memory is designed for.
|
||||
|
||||
3. **An agent reads this repository directly, and the search consults it.** Nothing is copied.
|
||||
**Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**The design record is read where it is written.** Retrieval is an agent reading this repository,
|
||||
not a copy living in a second store — and a search of the mesh's memory consults that agent, so
|
||||
what it knows appears beside ordinary results rather than only when it is asked.
|
||||
|
||||
Both halves are the decision. The first alone is merely a reader, and would leave this repository
|
||||
reachable but not surfacing — the state measured above. **The second half is what discharges the
|
||||
promise** that these documents are returned beside everything else.
|
||||
|
||||
**There is no copy, and that is the point.** No sync, no schedule, no reconciliation, and nothing
|
||||
that can drift, because there is only ever one of each document. It is also always current,
|
||||
including for work that is not yet committed.
|
||||
|
||||
**The direction of reading is one-way and stays that way.** The agent reads this repository and
|
||||
answers from it. Nothing flows back: this repository is public, the mesh is not, and a return path
|
||||
would be how installation-specific detail arrives into documents that must not carry it
|
||||
([`README.md`](../README.md)).
|
||||
|
||||
> **The mechanism changed — 2026-09-30, by ADR 0153.** What stands: read where it is written, no copy,
|
||||
> one-way. What moved: the reader is a module the mesh assigns (`records`, a checkout at a commit every
|
||||
> answer names) rather than the agent session of design 15, which is not built; and "the search consults
|
||||
> the agent" has no store to consult since the cut-over — the console's tool list is where the record
|
||||
> appears beside everything else. [ADR 0153](0153-the-record-is-read-by-a-module-and-the-console-lists-it.md).
|
||||
|
||||
## Consequences
|
||||
|
||||
**This repository stops being a fourth knowledge system, properly.** The original objection was
|
||||
about adding a knowledge *system*. An agent with read access adds no store at all — which answers
|
||||
the objection more completely than the indexing that was promised, rather than merely as well.
|
||||
|
||||
**ADR 0019's promise is amended, not satisfied.** It said these documents would be *indexed*. They
|
||||
will not be. They will be *read*, and the search will ask. The commitment that survives is the one
|
||||
that mattered — that a searcher finds them without already suspecting they exist — and the
|
||||
mechanism behind it is different from the one named.
|
||||
|
||||
**It is gated on an agent that does not exist yet.** Until it does, the signpost is what stands,
|
||||
and this repository is reachable rather than surfacing. That is a known and stated gap, not a
|
||||
silent one — and the gap is now a build task with a decided shape rather than an open question.
|
||||
|
||||
**The search must degrade honestly.** When the agent cannot be reached, a search has to say that
|
||||
this material was not consulted. A result set that silently omits it looks identical to one where
|
||||
nothing matched, and *silence and success must never look alike*
|
||||
([ADR 0004](0004-a-node-and-how-it-joins.md)) — the rule this repository has now paid for twice.
|
||||
|
||||
**A rule states how it is checked, and this one is checkable.** The check is the measurement that
|
||||
produced this record: search the mesh's memory for a phrase that appears only in a design document
|
||||
here, and require it back. That check fails today, deliberately, and passing it is what closes
|
||||
`04-ISSUES/006`.
|
||||
|
||||
## References
|
||||
|
||||
- [`04-ISSUES/006`](../04-ISSUES/006-hq-is-not-indexed-into-the-knowledge-base/00-report.md) —
|
||||
the gap, the two measurements, and why closing it early was refused
|
||||
- [ADR 0019](0019-how-this-repository-works.md) — the promise this amends
|
||||
- [`README.md`](../README.md) — the objection, and the gap named where the claim used to sit
|
||||
@@ -1,130 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
||||
---
|
||||
|
||||
# 26. The mesh has a session of its own, and it is the node session's mechanism
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md) gives every node a session: one per node, permanent,
|
||||
remembering across callers, its system prompt the node's engram, reachable over the broker like
|
||||
everything else. **Any node can message any node**, and that is called the one part of the system
|
||||
that is genuinely a mesh — symmetric, with no centre.
|
||||
|
||||
**There is no way to address the mesh itself.** A question that spans machines — *what is running
|
||||
across all of this*, *which nodes are behind*, *why is it built this way* — has to be put to some
|
||||
node, which then asks the others. That works, and it makes a mesh-wide question **nobody's
|
||||
question**: every node answers it as a foreigner, from a position where the whole is not in view.
|
||||
|
||||
**Three things independently arrived at the same missing piece.**
|
||||
|
||||
[ADR 0025](0025-the-design-record-is-read-not-copied.md), taken hours before this one, commits to
|
||||
an agent that reads the design repository directly and answers into search. That agent has to
|
||||
exist, run somewhere, and be askable — and nothing in the record says what it is or where it
|
||||
lives.
|
||||
|
||||
[`14-model-access.md`](../03-DESIGN/01-to-be/14-model-access.md) records, as a gap deliberately
|
||||
not half-built: *this worker uses that licence is a binding to an agent, not to a node* — and the
|
||||
provisions model has no consumer identity other than a node. A session that must be assigned a
|
||||
licence is exactly that consumer, and node sessions are already one.
|
||||
|
||||
**And ADR 0004 never said how a session is set up.** It describes behaviour and stops: nothing
|
||||
states how a session starts, where its context lives, how the engram reaches it, or how a message
|
||||
off the broker becomes a prompt. There is no design document for it. That gap was invisible until
|
||||
something had to be built *like* a node session, because describing a second instance of a
|
||||
mechanism requires the mechanism to have been described once.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **No mesh session; keep relaying through a node.** Costs nothing and works today. **Rejected.**
|
||||
It leaves mesh-wide questions belonging to nobody, and it does not survive contact with
|
||||
ADR 0025 — that agent still needs a home, so the thing gets built anyway, unnamed, as an
|
||||
attachment to whichever node happened to host it.
|
||||
|
||||
2. **A new kind of agent, built separately.** Purpose-built for the whole mesh. **Rejected.** It
|
||||
would hold a session, a memory, a licence and broker plumbing — every one of which the node
|
||||
session already has. Two implementations of one mechanism drift, and the vocabulary collision
|
||||
that [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) exists to undo began exactly this
|
||||
way: two things that were nearly the same, built twice, until neither word meant one thing.
|
||||
|
||||
3. **The same mechanism, started in a different context.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh has one session, addressed as the mesh, and it is a node session in every respect but
|
||||
three.**
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **the context it starts in** | the mesh's, not a machine's — this is the whole of what makes it different |
|
||||
| **its engram** | the mesh's system prompt, as a node's engram is that node's |
|
||||
| **its licence binding** | assigned in its own right, not inherited from the machine it runs on |
|
||||
|
||||
Everything else is unchanged and deliberately so: it is permanent, it remembers, it is reachable
|
||||
over the broker, it holds its own tools, and switched off it still answers *I am switched off*
|
||||
rather than falling silent.
|
||||
|
||||
**It runs on the node that holds the control plane** — not for convenience, but because that node
|
||||
is already the one place excepted from *compromise of a node is compromise of that node*
|
||||
(ADR 0004). An agent able to reach everything, placed anywhere else, creates a **second** such
|
||||
place. Putting it where the authority already sits concentrates nothing new.
|
||||
|
||||
**It is an addition to per-node messaging and never a replacement.** Every node remains directly
|
||||
addressable. This is not a preference: ADR 0001 holds that losing the control plane costs *change,
|
||||
not operation*, and a mesh whose only conversational surface lives on that node would lose the
|
||||
ability to ask anything while every machine kept running perfectly. **The front door may not be
|
||||
the single point.**
|
||||
|
||||
**It is not an employee** ([ADR 0003](0003-agents-are-persistent-employees.md)). Nobody hires it,
|
||||
it holds no task queue, it is never drained or reassigned. What it does with work that belongs
|
||||
somewhere else is **dispatch it** — to node sessions, or to workers — which is what a node session
|
||||
already does when asked something it does not have.
|
||||
|
||||
**It is ADR 0025's reader.** The agent that reads the design repository and answers into search is
|
||||
this session, not a second one. One agent, one memory, one place to reach; two would both need
|
||||
that repository and would eventually disagree about what it says.
|
||||
|
||||
**"One per node" is about address, not about process count.** ADR 0004's rule — *two and nothing
|
||||
decides which replies* — forbids ambiguity in who answers when a **node** is addressed. The mesh
|
||||
session answers when the **mesh** is addressed. The control-plane node therefore hosts two
|
||||
sessions and no ambiguity, and stating this here is what stops it reading as a contradiction
|
||||
later.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The node session's setup must now be designed, and it never was.** This decision is expressed as
|
||||
*the same as a node session, elsewhere*, which is only meaningful once that mechanism is written
|
||||
down. The design document covering both is the immediate consequence of this record, not a
|
||||
follow-up to it.
|
||||
|
||||
**A consumer that is not a machine stops being deferrable.** The licence binding above is the gap
|
||||
`14-model-access.md` names, and it now has two consumers rather than a hypothetical one. Until it
|
||||
exists, a session's model access can only be expressed as *this module on this machine*, which
|
||||
cannot say *this node's session uses the personal licence and the mesh's uses the company one* —
|
||||
the thing the binding is for.
|
||||
|
||||
**Symmetry is preserved, and it is worth being precise about why.** ADR 0004's claim is about what
|
||||
a node can reach, and it is untouched: node-to-node messaging is unchanged, nothing is routed
|
||||
through the mesh session, and it is a participant rather than a hop. What arrives is a
|
||||
participant that happens to be the one a person usually addresses.
|
||||
|
||||
**Availability degrades to inconvenience rather than to silence** — but only because of the
|
||||
addition rule above. If that rule is ever relaxed, this consequence inverts, and it inverts
|
||||
quietly: everything keeps working and nobody can ask about it.
|
||||
|
||||
**The surface a person uses is not decided here.** That a board is a good place to talk to it is
|
||||
likely and is not this record's business; the session is reachable over the broker like everything
|
||||
else, and what puts a text box in front of it is a separate choice.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — the node session this extends
|
||||
- [ADR 0025](0025-the-design-record-is-read-not-copied.md) — the reader this session is
|
||||
- [ADR 0003](0003-agents-are-persistent-employees.md) — the vocabulary this is not
|
||||
- [`03-DESIGN/01-to-be/14-model-access.md`](../03-DESIGN/01-to-be/14-model-access.md) — *a
|
||||
consumer that is not a machine*, the gap this makes concrete
|
||||
@@ -1,109 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 27. A provision names what the consumer is coupled to, not the role it plays
|
||||
|
||||
## Context
|
||||
|
||||
Provisions are named after roles. The catalogue and every test fixture built so far say:
|
||||
|
||||
```
|
||||
provides: database
|
||||
requires: database
|
||||
```
|
||||
|
||||
**Nothing distinguishes one engine from another.** A module requiring `database` is satisfied by
|
||||
any module providing `database`, so a module written against PostgreSQL can be matched to a
|
||||
provider of Microsoft SQL Server, resolve as satisfied, deploy, and fail on its first query.
|
||||
|
||||
**The mesh runs several engines** — PostgreSQL, Microsoft SQL Server, MariaDB, and others behind
|
||||
products that expose their own. This is not a hypothetical collision.
|
||||
|
||||
**The failure is in the direction that hides.** Resolution *succeeds*. Nothing is refused, nothing
|
||||
is logged, and the breakage surfaces later as an error inside an application, on a machine, with
|
||||
nothing connecting it back to a match made elsewhere by something that thought it had done its
|
||||
job. **A wrong answer delivered confidently costs more than a refusal**, and the whole point of
|
||||
refusing on ambiguity ([ADR 0009](0009-modules-and-the-graph.md)) was to not do this.
|
||||
|
||||
**How it got in:** every test written for the resolver had exactly one provider of each name, so no
|
||||
mismatch was expressible and none was caught. The fixtures agreed with the design. That is the same
|
||||
fault as [`04-ISSUES/005`](../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)'s
|
||||
imagined output and [`019`](../04-ISSUES/019-a-comment-asserting-a-fact-about-a-machine/00-report.md)'s
|
||||
unchecked comment, at the level of a name rather than a line.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep role names; let the operator assign correctly.** The mesh would refuse ambiguity when two
|
||||
providers exist, so a person picks. **Rejected.** It makes correctness depend on somebody
|
||||
knowing that the module they are assigning speaks a particular dialect — which is exactly the
|
||||
knowledge the provisioning model exists to remove. And with one provider of each name, nothing
|
||||
is ambiguous and nothing is asked.
|
||||
|
||||
2. **A role name plus a `flavour:` or `engine:` qualifier**, matched as a second field.
|
||||
**Rejected.** Two fields that must agree is a constraint the resolver has to enforce and a
|
||||
manifest author has to remember, to express something one field already can. The name is the
|
||||
contract; splitting it invites a requirement that names a role and forgets the qualifier, which
|
||||
then matches everything again.
|
||||
|
||||
3. **The name says what the consumer is coupled to.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**A provision is named for the thing a consumer's code is written against.**
|
||||
|
||||
```
|
||||
provides: postgres-database
|
||||
requires: postgres-database
|
||||
```
|
||||
|
||||
**The test is whether the consumer can tell the difference.** If swapping the provider would break
|
||||
the consumer, the name must say which provider — because a match that breaks the consumer is not a
|
||||
match. If the consumer genuinely cannot tell, a role name is correct and better.
|
||||
|
||||
| provision | | why |
|
||||
|---|---|---|
|
||||
| `postgres-database`, `mssql-database` | **specific** | applications are written against a dialect; a swap breaks them |
|
||||
| `route` | **role** | the consumer wants its name reachable and does not care what proxies it |
|
||||
| `resolver` | **role** | the consumer wants names to resolve |
|
||||
| `artifact-store` | **role** | the consumer fetches by digest over a protocol, and nothing else |
|
||||
|
||||
**`database` is not a provision and may not be provided.** There is no context in which an
|
||||
application talks to a generic database: it talks to PostgreSQL or it talks to SQL Server. A name
|
||||
that cannot be true of any real consumer should not be expressible.
|
||||
|
||||
**This is about coupling, not about products.** Two providers of `postgres-database` — a container
|
||||
on this node and a managed instance elsewhere — are interchangeable and *should* both match. What
|
||||
may not be interchangeable is what the consumer's queries are written in.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Every manifest that names a database changes.** Doing this now costs a rename across a handful of
|
||||
examples. Doing it after modules are migrated costs it across all of them, plus every deployment
|
||||
that resolved against the old name.
|
||||
|
||||
**Wrong requirements now fail loudly, and at the right moment.** A module requiring
|
||||
`postgres-database` where only `mssql-database` is provided is unsatisfiable, so it is **refused at
|
||||
resolution** with both names visible — rather than deployed and broken later. This is the property
|
||||
that was lost, restored.
|
||||
|
||||
**Generic role names are still right, and the rule says when.** This does not push specificity
|
||||
everywhere; it puts it exactly where a consumer is coupled. Naming `route` after a particular proxy
|
||||
would be the same error in the other direction, and would prevent a swap that genuinely changes
|
||||
nothing.
|
||||
|
||||
**It is checked, not merely stated** ([`00-META/how-we-build.md`](../00-META/how-we-build.md) §5).
|
||||
A manifest providing a name known to be engine-generic is refused, naming what to say instead.
|
||||
Without that, this record is a convention, and a convention is what the previous naming was.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — provisions, and refusing on ambiguity
|
||||
- [`03-DESIGN/01-to-be/07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md) — *the
|
||||
provisioning model uses databases, roles and schemas as PostgreSQL means them*, which is this
|
||||
record's point made about the substrate before it was made about modules
|
||||
@@ -1,118 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 28. The substrate supplies the control plane and nothing else
|
||||
|
||||
*Corrects one row of [ADR 0006](0006-the-substrate-and-the-control-plane.md) and makes explicit
|
||||
something it left unsaid. The rest of that record stands.*
|
||||
|
||||
## Context
|
||||
|
||||
ADR 0006 defines the substrate by a circularity: **what the control plane needs in order to run,
|
||||
and cannot ask itself for, because it is not running yet.** Two questions, and both must be
|
||||
answered *yes* for something to be substrate.
|
||||
|
||||
Its membership table admits the object store on this line:
|
||||
|
||||
| role | product | |
|
||||
|---|---|---|
|
||||
| object store | **MinIO** | it cannot grant itself a bucket |
|
||||
|
||||
**That answers the second question and assumes the first.** It is true that a control plane cannot
|
||||
grant itself a bucket. Nothing establishes that it needs one.
|
||||
|
||||
**It does not.** Verified 2026-08-31 against `mesh-control`: no S3 client, no bucket, no object
|
||||
storage of any kind outside comments. Artifacts reach nodes as content-addressed blobs in the OCI
|
||||
registry, and the code records the decision and its reasoning:
|
||||
|
||||
> One store, and it is the registry the bootstrap already pulls from. An OCI registry is a
|
||||
> content-addressed blob store that happens to also understand images… The alternative considered
|
||||
> was a second store beside it — S3-shaped, buckets, signed URLs. It is the right answer for
|
||||
> objects that are *mutable*, or need per-reader access, or are not build output. None of that
|
||||
> describes a digest-pinned archive, and standing up a second service to hold one kind of
|
||||
> immutable blob means two things to run, two things to back up and two ways for an artifact to be
|
||||
> missing.
|
||||
|
||||
**The row is inherited from the system being replaced**, where an object store distributed module
|
||||
tarballs. Here nothing does, and the row was never re-tested against the definition it sits under.
|
||||
|
||||
**A second thing ADR 0006 never says:** whether a substrate service and a module of the same
|
||||
product are the same instance. It says the substrate is *not the control plane* and *not a place
|
||||
for logic*, and stops. The question is not idle — an application wanting a database, on a mesh
|
||||
whose substrate is already running PostgreSQL, has an obvious wrong answer available.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave the object store as substrate, unused.** Harmless-looking. **Rejected.** A membership
|
||||
list that includes something nothing needs is a list that has stopped being derived from its
|
||||
test, and the next member is admitted by precedent instead of argument. It also mandates that
|
||||
every mesh run a service no mesh uses.
|
||||
|
||||
2. **Applications share the substrate's instances.** One PostgreSQL, one of everything.
|
||||
**Rejected**, below.
|
||||
|
||||
3. **The substrate is exactly what the control plane consumes; everything else is a module.**
|
||||
**Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**The object store is not substrate.** It fails the first half of the test: the control plane does
|
||||
not need one. An object store is an ordinary module, required through the module graph like
|
||||
anything else, and a module wanting one depends on a module providing one.
|
||||
|
||||
**The substrate has four members, not five**: a relational store, a message bus, an image registry,
|
||||
and conditionally an identity provider. The registry stays — the control plane genuinely cannot
|
||||
deliver an artifact without somewhere to put it.
|
||||
|
||||
**A substrate service and a module of the same product are different instances, and are not
|
||||
shared.** The mesh's own PostgreSQL and a PostgreSQL a workload was given are two servers, two
|
||||
containers, two lifecycles.
|
||||
|
||||
Three reasons, and the first is the one that matters:
|
||||
|
||||
**The substrate is not in the module graph.** It is raised from the pinned bundle the host carries,
|
||||
before any mesh exists to declare it. A workload depending on it would depend on something the
|
||||
graph cannot see, cannot rotate a credential for, and cannot move — which is every property the
|
||||
provisioning model exists to provide.
|
||||
|
||||
**It would put workload data in the control plane's own store.** The mesh's contexts own their
|
||||
stores exclusively ([ADR 0008](0008-a-context-owns-its-store.md)). An application sharing that
|
||||
server can exhaust it, lock it, or fill its disk, and the failure is the control plane going down
|
||||
— which is the one failure that makes every other one harder to fix.
|
||||
|
||||
**They are bounded differently.** The substrate is sized, backed up and upgraded as part of
|
||||
bootstrapping a mesh. A workload's database follows the workload — moved with it, destroyed with
|
||||
it, restored with it.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Migrating an object store is ordinary module work**, not substrate work. It was previously going
|
||||
to be done as part of completing the substrate, which would have been the wrong shape and would
|
||||
have coupled every mesh to a service the mesh does not use.
|
||||
|
||||
**A mesh with no workload needing one runs no object store at all.** That is the correct outcome
|
||||
and was not previously available.
|
||||
|
||||
**Two PostgreSQL containers on a node that hosts both is expected**, not duplication to be
|
||||
optimised away. Anyone tidying them together should find this record first.
|
||||
|
||||
**"Substrate by role and ordinary by delivery" loses one of its two members.** ADR 0006 uses that
|
||||
phrase of the object store and the registry — things that are substrate but provisioned once a
|
||||
control plane exists. It now describes the registry alone.
|
||||
|
||||
**The definition is applied, not just stated.** Both halves of the circularity test are asked of
|
||||
each member, and *cannot grant itself one* is not sufficient on its own — it is true of almost any
|
||||
service, which is what made it possible to admit a member on that half alone.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — the definition, and the table this
|
||||
corrects one row of
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store exclusively
|
||||
- `mesh-control internal/builder/registry.go` — where artifacts go, and why not S3
|
||||
@@ -1,112 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 29. A network is a shape, because an action cannot be undone
|
||||
|
||||
## Context
|
||||
|
||||
**A module of several containers has no way to let them reach each other by name.** A container
|
||||
declaration carries a `network` field, and it only ever *joins* one that already exists — it was
|
||||
added so the control plane could reach the store and the broker on the machine it was raised on.
|
||||
Nothing in the vocabulary **creates** a network.
|
||||
|
||||
Without one, containers on a machine share the runtime's default bridge, which gives addresses and
|
||||
no name resolution between them. So a module that is several containers can only be written by
|
||||
publishing ports onto the machine and pointing its own parts at the host — which puts a module's
|
||||
private wiring on the machine's own address space, where anything else on the machine can reach it
|
||||
and any other module can collide with it.
|
||||
|
||||
**This is the gap a mail system meets and nothing else so far does**
|
||||
([`00-work-breakdown.md`](../03-DESIGN/01-to-be/00-work-breakdown.md) 3.3). It is being taken now
|
||||
rather than then, because 3.3 is the task most likely to send work back into the declaration
|
||||
language and the least useful place to discover it.
|
||||
|
||||
**Adding a shape is not a small change, and the host says so** — the vocabulary is asserted
|
||||
against a stated number, with the reason written into the failure: *every addition widens what a
|
||||
compromised control plane can express, so a change here is a decision.* The host applies what it
|
||||
is told; the only bound on a hostile control plane is what the language can say
|
||||
([ADR 0004](0004-a-node-and-how-it-joins.md)).
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **An `action` that creates the network.** The vocabulary already has one, the bundle already
|
||||
uses seven of them, and `docker network create` with a `verify` is exactly the shape an action
|
||||
takes. Nothing would need adding. **Rejected**, on removal:
|
||||
|
||||
> An action has no footprint the host can undo — it ran, and whatever it did belongs to
|
||||
> whatever it acted on.
|
||||
|
||||
A network made this way **leaks when the module is unassigned**, and the mesh cannot tell: the
|
||||
record says an action ran, and there is nothing to reverse. Unassigning a module would leave a
|
||||
network behind on every machine it was ever on, and the only way to find them would be to go
|
||||
and look. *A resource the mesh can create and never clean up is one it should not create.*
|
||||
|
||||
There is a second reason, and it is the one that generalises: an action is opaque. **The mesh
|
||||
cannot tell what an action did**, so a network created by one is not a thing the mesh knows
|
||||
about — it cannot be reported, counted, or reasoned about, and a module could not require one.
|
||||
|
||||
2. **Publish ports on the machine instead.** No new shape, and it works today. **Rejected.** It
|
||||
makes a module's internal wiring part of the machine's address space: two modules that each
|
||||
want a database on a fixed port collide, and anything else on the machine can reach what was
|
||||
meant to be private. It also makes the module's manifest depend on what else is installed,
|
||||
which is the thing provisioning exists to remove.
|
||||
|
||||
3. **`network` as a ninth shape.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**`network` joins the vocabulary, and the vocabulary is nine shapes.**
|
||||
|
||||
```
|
||||
{"id": "internal", "type": "network", "name": "mail"}
|
||||
```
|
||||
|
||||
**A name and nothing else.** Not a driver, a subnet, an address range or a gateway: every one of
|
||||
those is a thing a module would have to know about the machine it lands on, and a module that
|
||||
names a subnet is a module that collides with whatever else chose the same one. The runtime picks;
|
||||
the mesh names.
|
||||
|
||||
**It is created if absent and removed when no longer declared** — an ordinary shape, with the same
|
||||
lifecycle as a directory. That is the whole reason it is a shape.
|
||||
|
||||
**Declared before the containers that join it.** Resources are applied in the order the module
|
||||
wrote them, and orphans are removed in **reverse** — so a network written first is created first
|
||||
and removed last, after the containers attached to it are gone. This is not a new rule; it is the
|
||||
existing one, and it happens to be exactly right here. A network written *after* its containers
|
||||
would fail to remove while they still hold it, and that failure is reported rather than silent.
|
||||
|
||||
**What it does not do:** it does not reach across machines. A network is one machine's, like
|
||||
everything else the host applies. Modules on different machines reach each other over the private
|
||||
network the mesh already provides ([ADR 0007](0007-connectivity.md)), and a shape that tried to
|
||||
span machines would be a second overlay with a worse contract.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The vocabulary is nine, and the count moves with a record.** The test that asserts it names this
|
||||
one, so the next person to change it finds the argument rather than a number to edit.
|
||||
|
||||
**A compromised control plane can now create and destroy networks on a machine.** Stated plainly
|
||||
because that is the cost, and the bound is the point: it can create a named network and remove
|
||||
one, and it can do neither to anything it did not declare. It cannot inspect, attach to, or
|
||||
reroute what is already there — those would be different shapes, and are not being added.
|
||||
|
||||
**A multi-container module becomes expressible**, which unblocks 3.3 and, less obviously, makes
|
||||
several smaller modules simpler: anything that is a service plus a sidecar currently has to
|
||||
publish a port to talk to itself.
|
||||
|
||||
**Nothing is required to use it.** A module of one container declares no network and joins none,
|
||||
exactly as now. The substrate keeps using `host`, which is a runtime-provided network and not one
|
||||
the mesh creates.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md) — the host's vocabulary, and why each shape is a decision
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — what may be pushed is bounded by form, not by trust
|
||||
- [`03-DESIGN/01-to-be/00-work-breakdown.md`](../03-DESIGN/01-to-be/00-work-breakdown.md) — 1.3,
|
||||
and the mail system at 3.3 that this is for
|
||||
@@ -1,96 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 30. Data outlives the mesh that declared it
|
||||
|
||||
## Context
|
||||
|
||||
**The conversion runs on live services holding real data**, and starts on the node that holds all
|
||||
of it. Identity, mail, everything. The requirement stated plainly: a data directory may be
|
||||
*moved*, and may never be *lost*.
|
||||
|
||||
**The host deleted them.** A directory that stopped being declared was an orphan, and an orphan
|
||||
directory was removed with `os.RemoveAll` — everything under it — while the report said
|
||||
`removed`. A module unassigned took its database's files with it, and nothing anywhere said what
|
||||
had been in there.
|
||||
|
||||
Reproduced before it was fixed: assign a module, let a service write into its directory, unassign
|
||||
the module, and the file is gone.
|
||||
|
||||
**A directory stops being declared for ordinary reasons**, which is what makes this sharp rather
|
||||
than theoretical. A module unassigned from a node. A manifest edited to move a data folder — the
|
||||
exact operation the conversion needs. A resource renamed. A typo. **Every one of those is a normal
|
||||
day's work, and every one of them was destructive.**
|
||||
|
||||
**The removal order was already right, and that is what makes a fix possible.** Everything the
|
||||
mesh puts inside a directory is itself a declared resource, and orphans are removed in reverse
|
||||
declaration order — so by the time a directory is reached, what the mesh wrote there is already
|
||||
gone. Anything still present was put there by something else.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A `keep` flag on the directory.** A module declares which of its directories hold data, and
|
||||
the host leaves those. **Rejected.** It is safe only when somebody remembered, and the failure
|
||||
of forgetting is total and silent. A rule that protects data only when it was asked to is not
|
||||
a rule about data, it is a rule about attentiveness — and this is the one place in the system
|
||||
where being wrong does not recover.
|
||||
|
||||
2. **Never remove a directory.** Simple and unarguably safe. **Rejected**, narrowly: every module
|
||||
ever assigned would leave its directories behind for ever, and a machine that accumulates
|
||||
things nobody can account for is one where nobody can tell what is still in use. The clean-up
|
||||
that is genuinely the mesh's is worth keeping.
|
||||
|
||||
3. **Remove a directory only when it is empty.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**A directory that still holds anything is kept, and the mesh says so.** An empty one is removed.
|
||||
|
||||
**This is the host's existing line applied to the one shape where getting it wrong is
|
||||
unrecoverable** — *it removes what it made and leaves what it merely configured*
|
||||
([ADR 0005](0005-the-node-host.md)). An empty directory is what the host made. A full one is not.
|
||||
|
||||
**No flag, no declaration, nothing to remember.** Emptiness is the test, and it is derived from
|
||||
the removal order rather than asserted: the mesh's own contents are gone by then, so what remains
|
||||
is by definition something nobody declared.
|
||||
|
||||
**It is reported, not silent.** The outcome is `kept`, naming how many items are inside and saying
|
||||
they are for a person to deal with. A directory quietly left behind is how a machine accumulates
|
||||
things nobody can account for — which is the objection to option 2, and it is answered by saying
|
||||
so rather than by deleting.
|
||||
|
||||
**Files are unchanged.** A declared file is the mesh's own — it wrote it, it owns it, and losing a
|
||||
configuration file is not the failure this is about. The distinction is deliberate: **directories
|
||||
hold what other things produced; files are what the mesh itself put there.**
|
||||
|
||||
## Consequences
|
||||
|
||||
**Moving a data directory is now safe by default.** The manifest changes, the old path stops being
|
||||
declared, and the data stays where it is until somebody has looked at it. That was the operation
|
||||
most likely to destroy something during the conversion, and it is now the operation that does the
|
||||
least.
|
||||
|
||||
**Unassigning a module leaves its data.** Correct, and it means unassignment is no longer a way to
|
||||
clean up — removing data is a person's act, done knowingly. Given what unassignment did before,
|
||||
that is the trade being made and it is the right way round.
|
||||
|
||||
**A machine can accumulate directories nobody removed.** Accepted, and mitigated by saying so
|
||||
every time rather than by a periodic sweep. A sweep would be the deletion this record exists to
|
||||
prevent, on a timer, with nobody watching.
|
||||
|
||||
**It is not a backup, and must not be mistaken for one.** This stops the mesh destroying data. It
|
||||
does nothing about a disk, a mistaken `rm`, or a service corrupting its own store. The conversion
|
||||
still needs backups taken and **restored** before anything is moved — a backup nobody has restored
|
||||
is a belief, not a copy.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md) — the host removes what it made
|
||||
- [`03-DESIGN/01-to-be/00-work-breakdown.md`](../03-DESIGN/01-to-be/00-work-breakdown.md) — the
|
||||
conversion this was found by planning
|
||||
@@ -1,72 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 31. The control plane authenticates nobody, so identity is a module
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0006](0006-the-substrate-and-the-control-plane.md) left one member of the substrate
|
||||
conditional, and said exactly why:
|
||||
|
||||
| role | product | |
|
||||
|---|---|---|
|
||||
| identity provider | — | **conditional**: substrate only if the control plane delegates authentication, which is undecided |
|
||||
|
||||
[`07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md) carried it as an open question —
|
||||
*whether identity is the fifth* — noting it followed from a decision nobody had taken.
|
||||
|
||||
**The decision is taken: the control plane does not delegate authentication.** There is no mesh
|
||||
identity provider.
|
||||
|
||||
**Nothing in the mesh's own machinery ever needed one.** A node proves itself with a keypair it
|
||||
generated, over a broker account issued at enrolment
|
||||
([ADR 0004](0004-a-node-and-how-it-joins.md)). Declarations are verified by signature. None of
|
||||
that touches an identity provider, and the conditional was never about machines — it was only ever
|
||||
about whether a *person* signing in to a mesh surface would be authenticated by something else.
|
||||
|
||||
## Decision
|
||||
|
||||
**Identity is a module**, like the mail system and the forge. It runs *on* the mesh, not *of* it
|
||||
([ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)) — a provider other modules require,
|
||||
which is the ordinary shape and needs nothing new to express.
|
||||
|
||||
**So the substrate is three, and no longer conditional**: a relational store, a message bus, and
|
||||
an image registry. Together with
|
||||
[ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md), which removed the
|
||||
object store, the list is settled and every member is there for the same reason — the control
|
||||
plane needs it and cannot ask itself for it.
|
||||
|
||||
**A mesh that wants no identity provider runs none.** That is now expressible, and was not while
|
||||
it sat in the substrate as a maybe.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The last open question about substrate membership is closed.** Both halves of ADR 0006's test
|
||||
now have an answer for every candidate, and the answer for identity is *the control plane does not
|
||||
need it*.
|
||||
|
||||
**It does not settle how a person signs in to a mesh surface**, and that is deliberately left
|
||||
open. What is settled is that whatever answers it is not part of what must exist before the mesh
|
||||
does — so it can be decided late, changed, or replaced, which is precisely what being substrate
|
||||
would have prevented.
|
||||
|
||||
**It becomes a real test of the module graph.** An identity provider is a module that *other
|
||||
modules require* — the object store already consumes it — so it exercises the provider chain more
|
||||
seriously than anything ported so far, where the provider was written alongside its consumer.
|
||||
|
||||
**Ordering follows from it rather than from preference.** Anything requiring identity has to move
|
||||
after it, which is a dependency the graph can state rather than something a person has to
|
||||
remember.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — the conditional this closes
|
||||
- [ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) — the other member
|
||||
removed, and the test applied properly
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — how a node proves itself, which needs none of this
|
||||
@@ -1,79 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: superseded
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
|
||||
superseded-by: 02-DECISIONS/0034-the-local-account-owns-the-mesh.md
|
||||
---
|
||||
|
||||
# 32. The local account owns the mesh; a surface delegates to a module
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0031](0031-the-control-plane-authenticates-nobody.md) settled that the control plane
|
||||
authenticates nobody, and deliberately left one thing open: **how a person signing in to a mesh
|
||||
surface is authenticated.** This answers it, and answers a question 0031 did not ask — *who owns
|
||||
the mesh at all.*
|
||||
|
||||
**There was no answer, and the absence was invisible** because every operation so far has been run
|
||||
by the person sitting at the machine. Nothing had to say whether that was the design or the
|
||||
circumstance.
|
||||
|
||||
## Decision
|
||||
|
||||
**The account that installed the host owns the mesh on that node.** Authority is a local login,
|
||||
and there is nothing else to hold.
|
||||
|
||||
**No mesh user model.** No accounts, no roles, no grants, nothing to administer. A person with a
|
||||
shell on a node can do anything the mesh can do there, because that is already true and pretending
|
||||
otherwise would be a boundary that does not exist.
|
||||
|
||||
**This follows from what was already decided rather than adding to it.**
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md) says there is no authorisation between nodes — every
|
||||
node is the operator's own, so a message from one is a message from them, and *the mesh boundary
|
||||
is therefore the security boundary*. A user model inside that boundary would guard nothing: anyone
|
||||
who could be stopped by it could equally read the node's key off the disk.
|
||||
|
||||
**The board is different, and the difference is the network.** A surface reachable by a browser
|
||||
has to know who is asking, because the people reaching it are not, by construction, people with a
|
||||
shell on the machine. **So the board delegates to an OAuth provider** — which is a module.
|
||||
|
||||
## What this does not change
|
||||
|
||||
**The identity provider is still not substrate** (ADR 0031). A *surface* delegating
|
||||
authentication is not *the control plane* delegating it. The control plane runs, applies
|
||||
declarations and reaches nodes with no identity provider in existence; only the board needs one,
|
||||
and only to decide whose browser it is talking to.
|
||||
|
||||
The test is unchanged and still answers no: *does the control plane need it in order to run?*
|
||||
|
||||
## Consequences
|
||||
|
||||
**The board depends on a module, and says so.** An ordinary edge in the graph, which means the
|
||||
board cannot come up before the provider it authenticates against — stated as a dependency rather
|
||||
than discovered as an outage.
|
||||
|
||||
**Moving the identity provider takes the board with it.** During that module's own conversion the
|
||||
board is unavailable, and that is acceptable: it is a surface, nothing depends on it, and a brief
|
||||
interruption is the trade already accepted everywhere else. Nothing that keeps a service serving
|
||||
goes through it.
|
||||
|
||||
**Anyone with a shell on a node has full authority there.** Written down rather than left implied,
|
||||
because it is the sentence that decides who gets an account on a machine. The protection is the
|
||||
machine's own login, and the overlay that keeps the machine unreachable from outside
|
||||
([ADR 0007](0007-connectivity.md)).
|
||||
|
||||
**A node cannot be operated by somebody without a login on it.** Deliberate, and the cost of
|
||||
having no user model: there is no way to give a person authority over one node without giving them
|
||||
a shell there. If that is ever wanted, it is a new decision and not a gap in this one.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — the control plane authenticates
|
||||
nobody; this answers what it left open
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — no authorisation between nodes, and why the mesh
|
||||
boundary is the security boundary
|
||||
- [`03-DESIGN/01-to-be/11-a-board.md`](../03-DESIGN/01-to-be/11-a-board.md) — the surface this is
|
||||
about
|
||||
@@ -1,91 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0028-the-substrate-supplies-the-control-plane-and-nothing-else.md
|
||||
---
|
||||
|
||||
# 33. The substrate is a store and a broker
|
||||
|
||||
## Context
|
||||
|
||||
Third correction to one table in one day, all found the same way: by asking whether **both** halves
|
||||
of the substrate test were actually answered for a given member, or only the second.
|
||||
|
||||
The test ([ADR 0006](0006-the-substrate-and-the-control-plane.md)) is *what the control plane needs
|
||||
in order to run, and cannot ask itself for, because it is not running yet.* ADR 0006 admits the
|
||||
image registry on this line:
|
||||
|
||||
| role | product | |
|
||||
|---|---|---|
|
||||
| image registry | **an OCI registry** | it cannot grant itself a repository |
|
||||
|
||||
**That is the second half again.** It is true that a control plane cannot grant itself a
|
||||
repository. Nothing establishes that it needs one *in order to run*.
|
||||
|
||||
**Counted rather than argued.** `substrate-first-node.lock` — the only bundle there is, and what a
|
||||
first node actually becomes — raises twelve resources, and no registry is among them:
|
||||
|
||||
```
|
||||
container runtime · the store · one database per context · the schemas
|
||||
· the broker's certificate · the broker · the control plane
|
||||
```
|
||||
|
||||
The registry arrives afterwards, as an ordinary module the mesh assigns. That is what the lab
|
||||
asserts, in those words: *the mesh runs its own artifact store.*
|
||||
|
||||
**ADR 0006 half-said this already**, calling the registry *substrate by role and ordinary by
|
||||
delivery, provisioned once there is a control plane to do it.* A member that is provisioned by the
|
||||
thing it supposedly precedes is not a member; the phrase was carrying a contradiction rather than
|
||||
resolving one.
|
||||
|
||||
**The registry is a closer call than the object store, and the difference is worth keeping.** The
|
||||
control plane never touches an object store at all — no client, no bucket, ever
|
||||
([ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md)). It genuinely
|
||||
*uses* the registry: the builder pushes to it, hosts pull from it, and nothing reaches a machine
|
||||
without it. **So the registry is a real dependency of the mesh operating, and not of the control
|
||||
plane starting** — and it is the second that the word substrate means.
|
||||
|
||||
## Decision
|
||||
|
||||
**The substrate is two things: a relational store and a message bus.** Both are in the bundle,
|
||||
both must exist before the control plane's first instruction, and neither can be asked for.
|
||||
|
||||
**The registry is an ordinary module.** The mesh cannot deliver anything without one, and it
|
||||
installs one the way it installs everything else. The first node's chicken-and-egg is already
|
||||
solved and needs nothing from this list: it fetches upstream images directly, then runs a registry
|
||||
of the mesh's own.
|
||||
|
||||
**The test is applied to both columns, every time.** *Cannot grant itself one* is true of almost
|
||||
any service and settles nothing on its own. It is what admitted the object store, and then the
|
||||
registry, and both were removed by asking the other question.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The substrate is now exactly what the bundle raises**, which is the strongest form this list can
|
||||
take: it can be checked by counting rather than by reading an argument. A member that is not in
|
||||
the bundle is not substrate, and the two statements cannot drift apart.
|
||||
|
||||
**A mesh that builds nothing still needs a registry** — to receive anything at all — but it needs
|
||||
it as a module, on its own schedule, replaceable. That was already true and was obscured by the
|
||||
list.
|
||||
|
||||
**The word may now be doing too little work.** "Substrate" for *a database and a broker* is a term
|
||||
of art for two things everybody can name. Renaming is not taken here and is worth considering
|
||||
separately; what this record fixes is the membership, not the vocabulary.
|
||||
|
||||
**Three removals from one table in one day is itself the finding.** Each member was admitted on the
|
||||
half of the test that is easy to answer, and the design read plausibly throughout. The rule that
|
||||
comes out of it is not about substrates: **a test with two conditions is a test only when both are
|
||||
asked.**
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — the definition, and the table this
|
||||
corrects a second row of
|
||||
- [ADR 0028](0028-the-substrate-supplies-the-control-plane-and-nothing-else.md) — the object
|
||||
store, removed for the same reason
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity, which was conditional and
|
||||
is now a module
|
||||
@@ -1,81 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
supersedes: 02-DECISIONS/0032-the-local-account-owns-the-mesh.md
|
||||
---
|
||||
|
||||
# 34. The local account owns the mesh, and a web application's login is not that
|
||||
|
||||
*Supersedes [ADR 0032](0032-the-local-account-owns-the-mesh.md), which decided the right thing and
|
||||
described it wrongly. The decision below is unchanged; what it said about the board was an
|
||||
invention.*
|
||||
|
||||
## Context
|
||||
|
||||
ADR 0032 answered *who owns the mesh* — the account that installed the host — and then framed the
|
||||
board as **a surface that delegates authentication**, a category it made up for the occasion. It
|
||||
does not need one.
|
||||
|
||||
**The board is a web application.** It has a login, provided by the identity module, in the way
|
||||
every web application has a login. That is a fact about an application, not a property of the
|
||||
mesh, and giving it a name in the mesh's vocabulary implied a relationship that is not there.
|
||||
|
||||
The cost of the invented category was not cosmetic. It made the identity module look like part of
|
||||
the mesh's own authority — something the mesh *depends on* to know who anybody is — when the truth
|
||||
is that the mesh knows nothing about people at all, and one of the applications running on it has
|
||||
a login.
|
||||
|
||||
## Decision
|
||||
|
||||
**The account that installed the host owns the mesh on that node.** Authority is a local login.
|
||||
There is nothing else to hold, no user model, no roles, and nothing to administer.
|
||||
|
||||
**This follows from what was already decided.**
|
||||
[ADR 0004](0004-a-node-and-how-it-joins.md) says there is no authorisation between nodes — every
|
||||
node is the operator's own, so a message from one is a message from them, and *the mesh boundary
|
||||
is the security boundary.* A user model inside that boundary would guard nothing: anyone it could
|
||||
stop could read the node's key off the disk.
|
||||
|
||||
**A web application's login is its own business.** The board authenticates its users through the
|
||||
identity module. So might anything else the mesh runs. **None of that is mesh authority**, and the
|
||||
mesh does not learn who anybody is from it.
|
||||
|
||||
## The line this draws, which is the reason to write it down
|
||||
|
||||
**Signing in to an application must not, on its own, become authority over the mesh.**
|
||||
|
||||
Today it cannot: the board reads and does not act
|
||||
([`11-a-board.md`](../03-DESIGN/01-to-be/11-a-board.md) — *not the way to change things*). Looking
|
||||
at a page tells you what is true and changes nothing.
|
||||
|
||||
**The moment the board can assign a module, whoever it lets in has mesh authority** — and it would
|
||||
arrive as a feature rather than as a decision. That is the failure this record exists to make
|
||||
visible, because it is the kind that is only obvious afterwards.
|
||||
|
||||
So: **a surface that can change the mesh is a change to who owns the mesh**, and is taken as one.
|
||||
Not forbidden — wanting to manage nodes from a browser is reasonable — but not something that
|
||||
turns up in a pull request titled *add assign button*.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The identity module is not special.** Not substrate ([ADR 0031](0031-the-control-plane-authenticates-nobody.md)),
|
||||
not part of the mesh's authority, and nothing about the mesh stops working when it is down. Some
|
||||
applications cannot be logged into, which is what it means for an application's login provider to
|
||||
be unavailable.
|
||||
|
||||
**Anyone with a shell on a node has full authority there.** Unchanged from ADR 0032, and still the
|
||||
sentence that decides who gets an account on a machine. The protection is the machine's own login
|
||||
and the overlay that keeps it unreachable from outside ([ADR 0007](0007-connectivity.md)).
|
||||
|
||||
**A node cannot be operated by somebody without a login on it.** The cost of having no user model.
|
||||
If that is ever wanted, the paragraph above says what it costs.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0032](0032-the-local-account-owns-the-mesh.md) — superseded; same decision, invented category
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — the mesh boundary is the security boundary
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — the control plane authenticates
|
||||
nobody
|
||||
@@ -1,115 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0034-the-local-account-owns-the-mesh.md
|
||||
---
|
||||
|
||||
# 35. One implementation, several surfaces, and what that costs
|
||||
|
||||
## Context
|
||||
|
||||
The mesh is operated from a command line today. It needs to be operable from a browser and from a
|
||||
model's tools as well, and the three must not be three different systems.
|
||||
|
||||
**The pattern is already in the code and unnamed.** `board` serves HTTP by calling the same
|
||||
functions the CLI calls; it holds nothing and decides nothing. What follows makes that the rule
|
||||
rather than a property of one command.
|
||||
|
||||
**The board is a presentation layer over the control plane.** Not an application beside it holding
|
||||
a database credential — the thing that shows what the control plane knows, and asks it to do what
|
||||
a person asked for.
|
||||
|
||||
## Decision
|
||||
|
||||
**The logic lives once, in the context that owns it. A surface is an adapter with no decisions in
|
||||
it.**
|
||||
|
||||
| surface | for |
|
||||
|---|---|
|
||||
| **command line** | a person on a machine, and the recovery path below |
|
||||
| **HTTP** | the board, and anything else that speaks to the mesh over a network |
|
||||
| **model tools** | an agent asking the mesh to do something |
|
||||
|
||||
**Every surface refuses identically, because the refusal is not in the surface.** An assignment
|
||||
that cannot be satisfied is refused by the same resolution whichever way it arrived. The moment a
|
||||
surface can accept something another would reject, the mesh has two answers to one question and
|
||||
people learn which to trust.
|
||||
|
||||
**Reading and doing are both exposed.** The HTTP surface is not read-only: managing the mesh from
|
||||
a browser is the point. This takes the decision
|
||||
[ADR 0034](0034-the-local-account-owns-the-mesh.md) said had to be taken deliberately —
|
||||
**a browser login now carries authority over the mesh** — and takes it knowingly rather than
|
||||
letting it arrive with a feature.
|
||||
|
||||
**The networked surfaces authenticate through an OAuth2 identity provider.** Named by protocol
|
||||
rather than by product, like every other dependency the mesh takes — AMQP for the bus, S3 for an
|
||||
object store, OCI for the registry
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md)). What fills the role today is a module
|
||||
running Keycloak; what the control plane knows is that it validates a token against a provider
|
||||
speaking OAuth2, and replacing that provider is a migration rather than a redesign.
|
||||
|
||||
**The command line does not authenticate at all**: it is already behind the machine's own login,
|
||||
which is what owns the mesh (ADR 0034).
|
||||
|
||||
## What this is not: a kernel every module imports
|
||||
|
||||
**The shared library is the failure this project was started over**, and the difference has to be
|
||||
stated or it will be rebuilt. The old one is 155 files and 34,636 lines *containing code from
|
||||
every context* — work-domain logic sitting in the kernel every module imports, each piece landing
|
||||
there to avoid a cycle between two modules that both needed it.
|
||||
|
||||
**Shared surfaces are not a shared library.** What is shared here is that three adapters call the
|
||||
same functions. Those functions stay in the context that owns them — provisioning's logic in
|
||||
provisioning, identity's in identity — and no module imports another's. A surface may call many
|
||||
contexts; a context still may not reach into another's store
|
||||
([ADR 0008](0008-a-context-owns-its-store.md)).
|
||||
|
||||
The test, when something is about to be put "somewhere shared": *does this belong to a context, or
|
||||
does it only belong to the surface?* If it belongs to a context it goes there, even if two
|
||||
surfaces want it.
|
||||
|
||||
## The loop this creates, and the way out
|
||||
|
||||
**The control plane's networked surfaces will depend on a module the control plane assigns.**
|
||||
An identity provider is an ordinary module ([ADR 0031](0031-the-control-plane-authenticates-nobody.md)). When
|
||||
it is down, or being migrated, or misconfigured, the HTTP and tool surfaces cannot authenticate
|
||||
anybody — including the person trying to fix it.
|
||||
|
||||
**The command line is the way out, and it is why local ownership matters more rather than less.**
|
||||
It authenticates through nothing, needs no network, and is available on the machine to the account
|
||||
that owns the mesh. **A mesh must always be operable by somebody standing at it.**
|
||||
|
||||
So the rule: **no capability exists only behind an authenticated surface.** Anything the board can
|
||||
do, the command line can do. That is not a courtesy to CLI users; it is the recovery path, and a
|
||||
capability that exists only over HTTP is one that disappears exactly when identity does.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Identity is still not substrate**, and the test still answers no: the control plane runs, applies
|
||||
declarations and reaches nodes with no identity provider in existence
|
||||
([ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md)). What is unavailable without it is two
|
||||
surfaces, not the mesh.
|
||||
|
||||
**Whoever the identity provider admits has authority over the mesh.** That is now a real perimeter
|
||||
with real consequences, where before it guarded a page that only read. Who may log in, and to
|
||||
which realm, becomes a decision about the mesh rather than about an application.
|
||||
|
||||
**A surface must not grow an opinion.** The likely erosion is a validation added to the board
|
||||
because it was quicker there — and then the CLI accepts something the board rejects, or worse the
|
||||
reverse. Adapters hold no decisions.
|
||||
|
||||
**Three surfaces over one implementation is a cost paid three times if it is not one
|
||||
implementation.** The reason to write this down now is that the second surface is the cheapest
|
||||
moment to get it right, and the third is where the drift usually starts.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0034](0034-the-local-account-owns-the-mesh.md) — the local account owns the mesh, and the
|
||||
line this record deliberately crosses
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) — the shared library this must not
|
||||
become
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store, which a surface does
|
||||
not change
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-08-31
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0035-one-implementation-several-surfaces.md
|
||||
---
|
||||
|
||||
# 36. Bootstrap ends at a usable mesh, and the first credential comes from a person
|
||||
|
||||
## Context
|
||||
|
||||
Bootstrap currently ends when the control plane starts
|
||||
([`07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md)). That is a mesh that runs and
|
||||
cannot yet be used by anybody who is not standing at the machine: the networked surfaces need an
|
||||
OAuth2 identity provider ([ADR 0035](0035-one-implementation-several-surfaces.md)), the provider is
|
||||
a module, and no module has been assigned.
|
||||
|
||||
**So bootstrap should go further** — through the identity provider and the first login — and stop
|
||||
at a mesh somebody can actually use.
|
||||
|
||||
**One thing in the way, and it is not incidental.** The mesh has never held a readable secret. The
|
||||
sealing code says what it does and why:
|
||||
|
||||
> Make generates a secret and seals it to both ends, **keeping no readable copy.**
|
||||
|
||||
An initial administrator's credential is the first value a **person must read**. Everything else
|
||||
the mesh generates is something no human ever sees, and everything a human provides is something
|
||||
the mesh immediately stops being able to read.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The mesh generates it and prints it once**, to the terminal of whoever ran the bootstrap.
|
||||
Convenient, and needs no prompt. **Rejected.** It would give the control plane a plaintext
|
||||
secret for the first time — briefly, and only to one terminal, but the capability would then
|
||||
exist. *An exception made for one case does not stay one*: the next credential that is awkward
|
||||
to supply gets printed too, and the property that a copy of the mesh's database is a copy of
|
||||
nothing stops being checkable by reading the code.
|
||||
|
||||
2. **No password: a one-time link that lets the operator set their own.** The nicest to use.
|
||||
**Rejected for now** — it needs a mechanism that does not exist, and the thing it improves is
|
||||
one prompt, once, on a new mesh.
|
||||
|
||||
3. **The operator supplies it.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**Bootstrap runs to a usable mesh**: the substrate, the control plane, the identity provider as an
|
||||
ordinary module, its realm and client provisioned, an administrator able to log in, and the
|
||||
networked surfaces available.
|
||||
|
||||
**The administrator's credential is supplied by the person doing the bootstrap**, on standard
|
||||
input and not echoed — the path that already exists for a model-access key. The mesh seals it and
|
||||
cannot read it afterwards.
|
||||
|
||||
**What is created is an account in the identity provider, not a user of the mesh.** The mesh still
|
||||
has no user model and gains none here ([ADR 0034](0034-the-local-account-owns-the-mesh.md)). What
|
||||
this produces is the first login for the applications that have one.
|
||||
|
||||
**The provisioning is ordinary.** A realm, a client and a first account are what an identity
|
||||
module's provisioner makes from what the mesh granted it — the same shape as a database and a
|
||||
bucket, which are built and proven.
|
||||
|
||||
**The surfaces arrive when their dependency does.** The command API is not started with the
|
||||
control plane and then broken until identity exists; it becomes available once it can authenticate,
|
||||
the way anything else waits for a provider.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The mesh still never holds a readable secret**, and that sentence needs no exception clause.
|
||||
That is the whole reason for the prompt.
|
||||
|
||||
**An unattended bootstrap is still possible, and the value still comes from outside.** Automation
|
||||
supplying the credential is the operator supplying it. What is refused is the *mesh inventing*
|
||||
one — so an unattended bootstrap with no credential provided produces a mesh with no
|
||||
administrator, which is correct rather than broken.
|
||||
|
||||
**Bootstrap gains an interactive step**, and it is the only one. Worth stating because a bootstrap
|
||||
that cannot run without a person is a real constraint on how a node is stood up, and this is
|
||||
deliberate rather than an oversight.
|
||||
|
||||
**The identity provider is still not substrate.** It is assigned by the control plane, so it comes
|
||||
after it, and a thing that comes after cannot be a thing that must exist before
|
||||
([ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md)). Bootstrap running through it does not
|
||||
move it: bootstrap is a sequence, the substrate is a dependency.
|
||||
|
||||
**And the recovery path is unchanged.** When the identity provider is broken later — which is the
|
||||
failure that matters, not the one at first start — the command line still works, because it
|
||||
authenticates through nothing (ADR 0035).
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0035](0035-one-implementation-several-surfaces.md) — the surfaces, and why the command line
|
||||
must keep working
|
||||
- [ADR 0034](0034-the-local-account-owns-the-mesh.md) — the local account owns the mesh; this adds
|
||||
no user model
|
||||
- [ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md) — what must exist before the control
|
||||
plane, which this does not change
|
||||
@@ -1,119 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-01
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
rests-on: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 37. Where a module lives
|
||||
|
||||
## The question
|
||||
|
||||
The mesh's own module descriptions currently sit in `examples/` inside the control plane, beside
|
||||
the small programs that hand out logins. That was fine while there were three of them. It is
|
||||
wrong now, and the name is doing active harm: everything in `examples/` reads as a sketch, and one
|
||||
of them shipped naming a container image that nothing in the repository builds. A directory called
|
||||
*the catalogue* would have made *does this actually work* the obvious question to ask of it.
|
||||
|
||||
So: **one repository holding the modules we ship?** And if so, where does everything that is not
|
||||
ours go?
|
||||
|
||||
## What a module actually is, counted
|
||||
|
||||
The system being replaced has **126 modules** on its main branch. The shape of them is the whole
|
||||
argument, so it is measured rather than assumed:
|
||||
|
||||
| | count | what it is |
|
||||
|---|---|---|
|
||||
| **the module is software** | 47 | its own source tree lives inside the module — a daemon, a service, a library |
|
||||
| **helper scripts only** | 44 | no application of its own; scripts it runs at install time or offers to an agent |
|
||||
| **a description and nothing else** | 35 | a package to install and some files to write |
|
||||
|
||||
**Two thirds of modules contain code.** The largest is a shared library of 182 source files. A
|
||||
speech-capture module carries a complete daemon — audio capture, mixing, transcription, a model
|
||||
runner. Treating a module as *a description of something else* is true of barely a quarter of them.
|
||||
|
||||
That kills the simplest answer. A catalogue cannot be "a folder of manifests" when most modules
|
||||
are programs.
|
||||
|
||||
## The four kinds, which want different homes
|
||||
|
||||
**1. What the mesh is made of.** The control plane, the host, the shared library, the board.
|
||||
These are not modules that happen to be ours; they are the mesh, expressed as modules so it can
|
||||
install itself. They belong in the repositories that build them, which already exist.
|
||||
|
||||
**2. Something the world made, that we describe.** A forge, a mail system, an identity provider,
|
||||
a media server. Nobody upstream ships a description; somebody has to write one, and it is the same
|
||||
description for everybody who runs it. **This is what a catalogue is for.** It is also where the
|
||||
small programs that create accounts belong, because such a program is part of describing that
|
||||
service, not part of the mesh.
|
||||
|
||||
**3. Something we wrote, that runs somewhere.** An application, a site, a side project. The
|
||||
description belongs **with the code, at the root of its own repository**, because the two change in
|
||||
the same commit. A repository that gains an environment variable and a description that gains it
|
||||
elsewhere will drift, and there is no mechanism that could stop it. This is already how it works
|
||||
and it should stay that way.
|
||||
|
||||
**4. A package and some files.** A tool, a font, a shell. Thirty-five of these, and each is a few
|
||||
lines. The catalogue.
|
||||
|
||||
## The proposal
|
||||
|
||||
**A `mesh-catalog` repository** holding kinds 2 and 4: descriptions of software we did not write,
|
||||
and the programs that provision it. Not kind 1, which is the mesh itself. Not kind 3, which lives
|
||||
with its own code.
|
||||
|
||||
**The mesh's list of modules is not this repository.** It is a table in the control plane, filled
|
||||
by adding a description to a running mesh. The catalogue is a *source* to add from — one of
|
||||
several, and the mesh already records which: every module carries where it came from, the branch
|
||||
followed there, and the commit its description was read at. **Nothing needs inventing to support
|
||||
modules from anywhere**; a repository of our own is simply the source we curate.
|
||||
|
||||
**A description is checked by the tool, not by a test that imports the tool.** Today a test in the
|
||||
control plane parses the example manifests by reaching into the control plane's internals, and
|
||||
another reads the control plane's own build file to check every image a module names can be built.
|
||||
Two jobs tangled. A `module check` command on the control plane's binary would let the catalogue
|
||||
hold data validated from outside, and would give the same check to somebody describing their own
|
||||
application in their own repository — which is the case that matters most and currently has no
|
||||
check at all.
|
||||
|
||||
## What this costs, and the argument against
|
||||
|
||||
**It is early.** Ten modules exist, four of them ours. Moving ten files is a morning; moving a
|
||||
hundred is a week — but the hundred is not here yet, and splitting now adds a second repository to
|
||||
release across before there is anything to release.
|
||||
|
||||
The counter is that the tangle is already producing faults rather than merely threatening to. A
|
||||
manifest naming an unbuildable image, and a test reading a build file two directories up, are both
|
||||
symptoms of one repository doing two jobs. And the moment the first module is adopted on a real
|
||||
machine, the descriptions stop being examples and become the thing deployments come from. **That
|
||||
is the moment this becomes urgent, and it is close.**
|
||||
|
||||
## What it does not settle
|
||||
|
||||
**Where a provisioning program's image is published**, and how a description pins it. A description
|
||||
names an image by digest; the image is built from the catalogue; the catalogue must therefore both
|
||||
produce an image and refer to it, which is the same knot the bootstrap has and solves by writing
|
||||
the digest down after building.
|
||||
|
||||
**Whether kind 4 deserves a module at all.** Thirty-five descriptions that say *install this and
|
||||
write these files* may be better as one module with settings than as thirty-five modules. Left
|
||||
open deliberately; it is a question about the shape of the catalogue, not about whether to have one.
|
||||
|
||||
## Accepted, 2026-09-29, against what was built
|
||||
|
||||
*Marked in a grooming pass: the mesh was built to this record and the record still said `proposed`.*
|
||||
|
||||
The proposal is the arrangement that exists. `mesh-catalog` holds descriptions of software we did
|
||||
not write and the programs that provision it, and holds neither the mesh's own components nor an
|
||||
application's own module. The mesh's list of modules is a table in the control plane, filled by
|
||||
`module add`, and every module records the source it came from with the commit it was read at.
|
||||
|
||||
**One half is not built: `module check` as a command on the control plane's binary.** A manifest is
|
||||
still validated by a test that reaches into the control plane's internals — which works for this
|
||||
catalogue and gives nothing at all to somebody describing their own application in their own
|
||||
repository, which this record says is the case that matters most. That is
|
||||
[issue 148](../04-ISSUES/148-a-manifest-outside-this-catalogue-has-no-check/00-report.md).
|
||||
|
||||
@@ -1,82 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-01
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
rests-on: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 38. The mesh assigns the port, and a module does not care
|
||||
|
||||
## The problem, as met
|
||||
|
||||
A database module cannot start on a machine that runs the control plane. The mesh keeps its own
|
||||
store there and holds 5432; the module publishes 5432. Nothing notices until a container runtime
|
||||
three layers down says `port is already allocated`
|
||||
([`028`](../04-ISSUES/028-two-things-want-one-port-and-nothing-says-so/00-report.md)).
|
||||
|
||||
A module cannot fix this by choosing better, because **a module cannot know what else is on the
|
||||
machine.** It is written once and assigned anywhere. Any number it picks is a guess about a
|
||||
machine it has never seen, and two modules guessing the same number is not a mistake either of
|
||||
them made.
|
||||
|
||||
## The number is written three times, and nothing makes them agree
|
||||
|
||||
Every module says its port in three places:
|
||||
|
||||
| where | for | example |
|
||||
|---|---|---|
|
||||
| `listens` | the rule set that lets traffic in | `{port: 5432, from: mesh}` |
|
||||
| `serves` | what a consumer must know to connect | `{port: 5432}` |
|
||||
| a container's `ports` | what the runtime publishes | `"5432:5432"` |
|
||||
|
||||
They agree today because one person wrote all three. Nothing checks it. A module whose `serves`
|
||||
said 5432 and whose container published 5433 would resolve, compose, apply, and hand every
|
||||
consumer a port that answers nothing.
|
||||
|
||||
## The decision
|
||||
|
||||
**The mesh assigns the machine-side port, and the module says only what it needs.** A module
|
||||
declares that a container port must be reachable and what it is for. Which number the machine uses
|
||||
is the mesh's to choose, because the mesh is the only thing that knows what else is there.
|
||||
|
||||
**One source, and the other two are derived.** `serves` carries the assigned port so a consumer is
|
||||
told where to connect without the module having written it down; the rule set is computed from the
|
||||
same assignment. Three copies become one fact.
|
||||
|
||||
**An assignment is made once and kept**, exactly as a credential is. A port that moved on every
|
||||
push would restart both ends each time and would hand consumers a number that was true when it was
|
||||
read.
|
||||
|
||||
## Some ports cannot move, and that is a claim
|
||||
|
||||
Mail is 25, submission is 587, IMAP over TLS is 993. A mail system on a strange port is not a mail
|
||||
system. So a module may say a port is **fixed by the protocol** rather than assigned.
|
||||
|
||||
**A fixed port is exactly a claim** — the thing the mesh already has for what is singular on a
|
||||
machine: one seat, one display server, one artifact store. Two modules wanting 25 on one machine is
|
||||
the same shape as two wanting the seat, and gets the same answer: the second is refused, by name,
|
||||
when it is assigned rather than when it is applied.
|
||||
|
||||
That is why this does not need a new mechanism so much as it needs the existing one pointed at
|
||||
ports.
|
||||
|
||||
## What follows
|
||||
|
||||
- **A module becomes portable in a way it was not.** Two databases on one machine stop being a
|
||||
collision and become two assignments.
|
||||
- **The substrate has to be visible.** The mesh cannot assign around its own store while it has
|
||||
never heard of it. What the bundle holds must be written down somewhere the assignment can read
|
||||
— which the bundle does not say today.
|
||||
- **A refusal can be useful.** *25 is held by the mail system on this machine* is a sentence a
|
||||
person can act on. `port is already allocated` is not.
|
||||
- **`serves` stops being written by hand**, which is a small vocabulary change with a large
|
||||
consequence: what a consumer is told is now derived from what actually happened.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
**Whether a module should publish to the machine at all.** Assignment makes publishing safe; it
|
||||
does not make it necessary. Consumers could instead reach a provider on the module's own network by
|
||||
name, with nothing published — which would make the question moot for anything inside the mesh, and
|
||||
would still leave it for anything reached from outside.
|
||||
@@ -1,108 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 39. What the SDK holds, and what it refuses
|
||||
|
||||
> **The mechanism changed — 2026-10-02, by [ADR 0188](0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md).** The test of this record — frequent *and* cascading does not belong — stands and now applies to one SDK per language. Where *what it holds* names the broker client and the event consumer, read: the local protocol a tools bundle speaks to the node's runtime; the transport lives in the runtime and in no SDK, which is what keeps a bus change from rebuilding any module in any language.
|
||||
|
||||
_Reconciliation note (2026-09-05): supersedes the earlier "repository structure" decision, which the consolidation folded; no standalone record remains to point at, so body references to it now point at the nearest surviving record, [ADR 0015](0015-applications-live-in-their-own-repository.md)._
|
||||
|
||||
## Context
|
||||
|
||||
The earlier "repository structure" decision (folded in consolidation; see the reconciliation note
|
||||
above, and [ADR 0015](0015-applications-live-in-their-own-repository.md) as the nearest survivor)
|
||||
named `mesh-sdk` "contracts shared across tiers:
|
||||
types, not behaviour." That line is superseded here, because it draws the boundary in the wrong
|
||||
place. The boundary that matters is not *types versus behaviour* — it is **how often the thing
|
||||
changes**.
|
||||
|
||||
The current SDK is the cautionary tale, and its failure is precise. `hal/sdk` holds all the
|
||||
code, including a per-module API client for every service (`clients/plex.ts`, `clients/gitea.ts`,
|
||||
…) and a per-module tool implementation for each (`tools/plex.ts`, …). Every module depends on
|
||||
the SDK, so **every edit to any of that per-module code rebuilds every module** — the cascade.
|
||||
The SDK is under constant maintenance precisely because it became the place all the volatile
|
||||
per-module logic accumulated.
|
||||
|
||||
The root cause is worth stating exactly, because the fix follows from it: the pressure was never
|
||||
to share a client *between* modules. It was to share a client between one module's *own features*
|
||||
— plex's tools, its health check and its hooks all wanted the same `PlexClient` — and the only
|
||||
place to share code across a module's features was the global SDK. So **intra-module sharing
|
||||
leaked out as inter-module coupling.**
|
||||
|
||||
## Decision
|
||||
|
||||
The SDK holds the **stable spine** that modules build against, and earns its place by rarely
|
||||
changing. The test for membership is change-frequency, not kind.
|
||||
|
||||
### What it holds
|
||||
|
||||
- The **tool-serving harness** — the worker and registration mechanism, and the tool-definition
|
||||
type. *How* a tool is declared and served is settled; it does not change when an individual
|
||||
tool does.
|
||||
- The **messaging and event framework** — the broker client, the event consumer, the envelope.
|
||||
- The **contracts** — the manifest, declaration, provision and link shapes.
|
||||
- **Core primitives** — sealing and crypto, semver, the shared resolution helpers.
|
||||
|
||||
These change rarely and deliberately. When one of them does change, a rebuild of everything is
|
||||
the *correct* outcome, because the contract every module shares has genuinely changed.
|
||||
|
||||
### What it must not hold — the more important half
|
||||
|
||||
- **A module's API client.** A Plex client, a Gitea client, a MinIO client belong in their
|
||||
module. They change when that service's API or the module's use of it changes, which is often,
|
||||
and which has nothing to do with any other module.
|
||||
- **A module's tool implementations.** Same reason, same place: in the module.
|
||||
- **Anything volatile** — anything that changes when one service's features change.
|
||||
|
||||
The rule, stated so it can be applied without re-deriving it:
|
||||
|
||||
> If editing a thing recompiles unrelated modules **and** it changes often, it does not belong
|
||||
> in the SDK.
|
||||
|
||||
Both conditions are load-bearing. A rare change that cascades is fine — that is a contract, and
|
||||
the cascade is correct. A frequent change that stays local is fine — that is a module minding its
|
||||
own business. Only **frequent *and* cascading** is the disease, and per-module clients and tools
|
||||
are its carriers.
|
||||
|
||||
### Where per-module shared code lives instead
|
||||
|
||||
Code shared among a module's *own* features lives **in the module**. The default is the plainest
|
||||
thing that works: an ordinary shared file the features import — `plex/client.ts`, imported by
|
||||
`plex/tools/`. Within one module, features are files importing sibling files; no package
|
||||
boundary, no ceremony.
|
||||
|
||||
A **module-local SDK** (a sub-package with its own version) is warranted only for the few modules
|
||||
whose shared surface is large enough to version on its own. It is the exception, not the shape.
|
||||
|
||||
Either form gives the property the global SDK could not: editing a module's shared code rebuilds
|
||||
**that module and nothing else**.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The cascade becomes **structurally impossible for module logic**. There is no longer an edge
|
||||
from one module's internals to another, so the only thing that can rebuild everything is a real
|
||||
change to a shared contract in the SDK — which is rare, and when it happens, is right.
|
||||
- The SDK is small and stable **by construction**, not by discipline. Its size is no longer a
|
||||
thing anyone has to police.
|
||||
- **Converting a module from the current system is partly a de-coupling, not just a move.** Its
|
||||
client and its tools are pulled *out* of the shared SDK and *into* the module. A conversion
|
||||
that copied `clients/plex.ts` into the SDK's replacement would rebuild the exact mistake.
|
||||
- The host still does not import the SDK. It depends on nothing
|
||||
([ADR 0005](0005-the-node-host.md)) and **mirrors** the contracts rather than
|
||||
importing them, exactly as its apply-shapes table already does deliberately. The SDK is shared
|
||||
by the tiers that *can* share code; the host is not one of them.
|
||||
|
||||
## References
|
||||
|
||||
- The earlier "repository structure" decision — named the repositories; its `mesh-sdk`
|
||||
description ("types, not behaviour") is superseded by this record (folded in consolidation;
|
||||
nearest survivor [ADR 0015](0015-applications-live-in-their-own-repository.md)).
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) — the decomposition this serves: code
|
||||
belongs to the boundary that owns it.
|
||||
- [ADR 0005](0005-the-node-host.md) — why the host mirrors the contracts instead of
|
||||
importing the SDK.
|
||||
@@ -1,103 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 40. What a module is
|
||||
|
||||
_Reconciliation note (2026-09-05): supersedes the earlier "grouped by domain" decision, which the consolidation folded into how-we-build.md; no standalone record remains to point at._
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0009](0009-modules-and-the-graph.md) settled that everything is a module, but never said what a
|
||||
module *is* beyond "a directory the mesh processes." That gap let the catalogue's breadth read as a
|
||||
smell: a module can carry a container, a built image, tools, a provisioner, migrations, health,
|
||||
config, seat claims, requires and provides — so much that the unit seemed ill-defined.
|
||||
The earlier "grouped by domain" decision (folded in consolidation; see the reconciliation note
|
||||
above) tried to organise modules by domain, which is the wrong axis. This record states what a module is, drawn from the cases that
|
||||
stress-tested it: the shell, i3-vs-sway, umami, and "database."
|
||||
|
||||
## Decision
|
||||
|
||||
**A module is one self-contained piece of software the mesh installs and manages** — everything
|
||||
needed to make that one thing real and integrable: what runs, the seats it claims, what it provides
|
||||
to other modules, what it requires from them, and what operates it.
|
||||
|
||||
The **software is the module's identity.** Capabilities, seats and provisioned resources are the
|
||||
**relationships *between* modules**, not what a module is — and that is what binds a module into one
|
||||
thing. umami is bound by *being umami*: its container runs umami, its provisioner creates umami sites,
|
||||
its tools query umami, its `requires` gets umami a database. Every feature serves the one software.
|
||||
|
||||
### The three relationships
|
||||
|
||||
1. **Shared seat** — several modules fulfil a capability and coexist; one may be default. bash, zsh
|
||||
and fish all join `shell`.
|
||||
2. **Exclusive seat** — modules contend for a single slot; one holds it. i3 (needs x11) and sway
|
||||
(needs wayland) contend for `display-session`.
|
||||
3. **Provide / require** — a provider ships the **provisioner** that creates instances of the
|
||||
resource it offers and returns sealed credentials; a consumer requires it and the mesh wires the
|
||||
credential in. Symmetric: umami requires a database *and* provides analytics.
|
||||
|
||||
### Interfaces are mesh-owned; providers adapt to them
|
||||
|
||||
The mesh **defines the interface** for a capability — the provider-neutral contract of what a
|
||||
consumer receives and how it integrates. Both sides conform: a provider's provisioner **adapts** its
|
||||
software's real API to the mesh contract; a consumer depends on the **interface**, never on a
|
||||
provider. Swap one provider for another and the consumer does not change.
|
||||
|
||||
### The naming rule — draw the interface at the consumer's real coupling
|
||||
|
||||
Name a `provides`/`requires` at the **widest boundary across which the consumer genuinely does not
|
||||
care which implementation serves it**:
|
||||
|
||||
- Where the consumer's coupling is thin — an analytics embed snippet and dashboard, opaque to it —
|
||||
the mesh defines a neutral interface (`analytics`) and providers (umami, amumi) adapt. Swappable
|
||||
across vendors.
|
||||
- Where the consumer **speaks a protocol** — a database's wire protocol and query dialect — the
|
||||
interface *is* the protocol: `postgres-database`, `mssql-database`, `mongodb-database`. Swappable
|
||||
only among protocol-compatible implementations, **never across**, because the application cannot
|
||||
cross it either. "database" is not a capability; the protocol is.
|
||||
- **Never false genericity.** A name must not promise a swap the contract cannot deliver
|
||||
([research 005](../01-RESEARCH/005-domain-grouping/analysis.md)).
|
||||
|
||||
This is [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md)'s rule made general — "names the protocol, not
|
||||
the product; a database names the engine because the app targets it" — with the reason stated: the
|
||||
contract sits where the coupling is.
|
||||
|
||||
### What is not a module
|
||||
|
||||
- A **library** (built against, never deployed — [ADR 0039](0039-what-the-sdk-holds-and-refuses.md)).
|
||||
- A **control-plane context** (the mesh itself — [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md)).
|
||||
|
||||
A **swappable machine mechanism** (a firewall — ufw, nftables) *is* a module implementing a
|
||||
capability. The host hardcodes no firewall, supervisor, package manager or runtime; it owns only the
|
||||
generic apply primitives and platform detection, so it runs where none of those exist — an Android
|
||||
phone has no ufw, systemd, pacman or Docker.
|
||||
|
||||
> **The mechanism changed — 2026-10-02, by [ADR 0176](0176-the-login-shell-is-a-node-seat-and-execute-is-its-contract.md).** The shell example above — *bash, zsh and fish all join `shell`; one may be default* — is read as *installed is not holding*: the three may all be installed, and the `login-shell` seat is node-scoped and held by exactly one. The decision — what a module is, and the three relationships — stands; [ADR 0173](0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md) applies it to the operator's whole machine.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Supersedes the earlier "grouped by domain" decision** (folded in consolidation; see the
|
||||
reconciliation note above). Modules are
|
||||
organised by their relationships (seats, provisions), not grouped into domain folders.
|
||||
- **Refines [ADR 0009](0009-modules-and-the-graph.md).** Everything the mesh runs and integrates is
|
||||
a module — but a module is defined by the *software it delivers*, not by being a bucket of features.
|
||||
- The target is a **self-fulfilling mesh**: declared wants bound to swappable modules, provisioners
|
||||
wiring credentials, nothing hardcoded. The control plane's whole job is the binding.
|
||||
- Converting a module from the old system includes pulling its per-module code out of the shared SDK
|
||||
([ADR 0039](0039-what-the-sdk-holds-and-refuses.md)) and shipping its provisioner as an adapter to a
|
||||
mesh interface — a de-coupling, not just a move.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — everything is a module; this says what one is.
|
||||
- [ADR 0001](0001-mesh-brokers-nodes-host-agents-think.md) — contexts are the mesh, not modules.
|
||||
- The earlier "grouped by domain" decision — superseded (folded in consolidation; see the note above).
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — protocol-not-product, generalised here.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — per-module code lives in the module.
|
||||
- [research 011](../01-RESEARCH/011-the-module-graph/00-overview.md) — the graph of these relationships.
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0040-what-a-module-is.md
|
||||
---
|
||||
|
||||
# 41. Events are a relationship, the lighter sibling of provisioning
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0040](0040-what-a-module-is.md) names two relationships between modules — seats and
|
||||
provide/require (provisioning). A third is latent in the mesh and worth making first-class: the
|
||||
broker every node already runs ([ADR 0002](0002-nodes-communicate-over-a-broker.md)) can carry a
|
||||
module's activity as **events**, which any other module reacts to. A logger that writes an audit
|
||||
trail, a module that acts when another module acts, observability — all of it is one mechanism, and
|
||||
today it is ambient rather than declared.
|
||||
|
||||
## Decision
|
||||
|
||||
**A module emits events and consumes events, and both are declared** — parallel to `provides` /
|
||||
`requires`, so the mesh knows the event graph the same way it knows the provisioning graph.
|
||||
|
||||
### Events are provisioning's lighter sibling
|
||||
|
||||
| | provisioning | events |
|
||||
|---|---|---|
|
||||
| shape | **1:1**, a provider creates a resource *for* one consumer | **1:many**, a module emits, any number listen |
|
||||
| credential | yes — sealed, per consumer | none — it is broadcast |
|
||||
| machinery | a provisioner (the reconcile adapter) | nothing but the broker's topic routing |
|
||||
| declared as | `provides` / `requires` | `emits` / `consumes` |
|
||||
|
||||
Because an event is broadcast and credential-free, there is no provisioner and no per-consumer
|
||||
setup — only a subscription. That is why it is the *lighter* relationship, and why most
|
||||
inter-module reaction should be an event, not a provision.
|
||||
|
||||
### An event carries what an audit needs
|
||||
|
||||
Every event carries its **type** (a dotted topic key, so listeners match by prefix), its **source**
|
||||
module, the **node** it came from, and the **time**. A body follows. The metadata is not optional:
|
||||
a reaction may only need the body, but an audit trail needs to know who did what, where and when,
|
||||
and an event that cannot answer that is not auditable.
|
||||
|
||||
### The audit logger is just a consumer of everything
|
||||
|
||||
A logger that records the whole mesh's activity is **not a privileged component** — it is an
|
||||
ordinary module that consumes `#` (every event) and writes them down. It holds no special access;
|
||||
it only listens widely. That it falls out of the model with no new machinery is the check that the
|
||||
model is right.
|
||||
|
||||
### `consumes` is validated like `requires`
|
||||
|
||||
A `consumes` for an event that **nothing** `emits` is a dangling edge, and the mesh refuses it
|
||||
before deploy — the same rule that catches a `requires` for a resource nothing provides
|
||||
([research 011](../01-RESEARCH/011-the-module-graph/00-overview.md)). A listener waiting for an
|
||||
event that can never arrive is a silent failure, and this repository's whole discipline is against
|
||||
silent failure.
|
||||
|
||||
### One runtime serves all three
|
||||
|
||||
The per-node module runtime that serves a module's tools also wires its `consumes` (subscribe,
|
||||
dispatch to the handler) and lets its code `emit`. Tools are *invoked* (request/reply), resources
|
||||
are *provisioned* (1:1, credentialed), events are *emitted and consumed* (1:many, broadcast) —
|
||||
three relationships, one broker, one runtime, all declared on the manifest.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The mesh gains a declared **event graph** alongside the provisioning graph — visible, validated,
|
||||
reasoned over.
|
||||
- **Reaction becomes the default coordination**: a module acts on another's event without either
|
||||
knowing the other, and without a credentialed link. Coupling drops.
|
||||
- An **audit trail** is a module, not a platform feature — and can be swapped, extended or run more
|
||||
than once (a file logger and a queryable one) with no change to anything that emits.
|
||||
- The runtime must dispatch a module's event handlers as well as its tools; that generalisation is
|
||||
small (both arrive by importing the module's entrypoint) but it is real work.
|
||||
|
||||
## Progressive insight
|
||||
|
||||
> **Progressive insight — 2026-09-26.** *"No provisioner and no per-consumer setup" was a fact
|
||||
> about the transport, and the transport changed.* This record's table says an event's machinery is
|
||||
> "nothing but the broker's topic routing", and the text that an event needs "no per-consumer setup
|
||||
> — only a subscription". That was true of a topic exchange, where a binding cost nothing and the
|
||||
> broker fanned out. On NATS
|
||||
> ([ADR 0106](0106-the-bus-is-nats.md)) a subscription is a **durable consumer**: a real object
|
||||
> with a name, an ack policy, a delivery limit and its own ack subject, created when a module is
|
||||
> assigned and removed when it is not. Per-consumer setup exists, and the controller does it.
|
||||
>
|
||||
> The decision is untouched — events are declared on both sides, 1:many, credential-free, and
|
||||
> still provisioning's lighter sibling; the lightness is now relative rather than absolute.
|
||||
> [ADR 0126](0126-a-module-declares-its-own-seats.md) adds the relationship this record's two
|
||||
> columns had no room for: work addressed to a role, where exactly one holder must act.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0002](0002-nodes-communicate-over-a-broker.md) — the broker events ride.
|
||||
- [ADR 0040](0040-what-a-module-is.md) — the relationships this extends.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — `emit`/`on` are stable sdk surface; the
|
||||
broker binding and the runtime are not.
|
||||
- [research 011](../01-RESEARCH/011-the-module-graph/00-overview.md) — the graph these edges join.
|
||||
@@ -1,116 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0041-events-are-a-relationship.md
|
||||
---
|
||||
|
||||
# 42. The shape of an event on the wire
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0041](0041-events-are-a-relationship.md) made events a relationship — `emits`/`consumes`, the
|
||||
graph, the audit logger. It did not say what an event *is* on the broker: the exchanges, the
|
||||
routing keys, the headers, the queues and their configuration. That shape is a contract every
|
||||
emitter and consumer conforms to, exactly as [ADR 0010](0010-delivery.md)
|
||||
is for declarations — and it was being decided ad-hoc in code. This settles it, so the sdk and the
|
||||
runtime implement one contract and a module never reinvents it.
|
||||
|
||||
## Decision
|
||||
|
||||
### Two exchanges, kept apart
|
||||
|
||||
- **`mesh.events`** — a durable topic exchange. Every event rides it: module, mesh and node.
|
||||
- **`mesh.rpc`** — a durable topic exchange. Tool invocations (request/reply) ride it.
|
||||
|
||||
Kept separate because RPC is not an event: a `#` subscription on `mesh.events` is then a complete
|
||||
audit of what happened, with none of the invocation traffic.
|
||||
|
||||
### The routing key is the event type, namespaced by origin
|
||||
|
||||
Dotted and hierarchical — `<origin>.<name>.<event…>` — with three reserved origins:
|
||||
|
||||
- `module.<module>.<event>` — `module.umami.site.created`
|
||||
- `mesh.<context>.<event>` — `mesh.delivery.deployed`, `mesh.provisioning.granted`
|
||||
- `node.<node>.<event>` — `node.anchor.joined`, `node.anchor.unreachable`
|
||||
|
||||
Topic matching gives a consumer `node.*.joined`, `module.umami.#`, or `#`. The origin roots are
|
||||
reserved; everything after is the emitter's own namespace.
|
||||
|
||||
### Metadata in headers, payload in the body
|
||||
|
||||
An event's identity and provenance are AMQP **headers**, so a consumer — or the broker, or an
|
||||
audit tool — reads who/when/what without parsing the body, and the body is only the domain payload.
|
||||
|
||||
**Required headers**
|
||||
|
||||
| header | meaning |
|
||||
|---|---|
|
||||
| `x-event-id` | a unique id — for dedup and audit (delivery is at-least-once, below) |
|
||||
| `x-source` | the emitter: the module, context or node name |
|
||||
| `x-node` | the node it was emitted from |
|
||||
| `x-time` | emit time, RFC-3339 |
|
||||
| `content-type` | `application/json` |
|
||||
|
||||
**Optional headers**
|
||||
|
||||
| header | meaning |
|
||||
|---|---|
|
||||
| `x-causation-id` | the event or command that caused this one — tracing |
|
||||
| `x-schema` | a version of the body's shape, so a body evolves without silent misreads |
|
||||
|
||||
The routing key already carries the type; it is not duplicated as a header. An **unknown `x-`
|
||||
header is ignored, not refused** — unlike a declaration, an event is observed by parties that need
|
||||
not all understand every header, and refusing would couple every consumer to every emitter's
|
||||
additions.
|
||||
|
||||
### Messages are persistent
|
||||
|
||||
Events are published persistent (delivery-mode 2). An audit trail that loses events on a broker
|
||||
restart is not one, and the cost is disk the broker already spends on everything durable.
|
||||
|
||||
### Queues: one per consumer, durable, dead-lettered
|
||||
|
||||
- **A consumer's queue** is `<node>.<module>.events`, durable, bound to that module's consumed
|
||||
patterns. Durable so a restart does not drop what arrived while it was down. **Manual ack** after
|
||||
the handler succeeds — at-least-once.
|
||||
- **Prefetch** bounds in-flight work (default 32) so one slow consumer does not pull the whole
|
||||
backlog into memory.
|
||||
- **A dead-letter exchange** `mesh.events.dead` receives a message rejected past a redelivery limit,
|
||||
so a poison event is set aside for inspection rather than looping forever or vanishing silently.
|
||||
- **The audit logger's queue** `<node>.audit-logger.events`, bound to `#`, is the same shape —
|
||||
durable, persistent, dead-lettered — because completeness is its whole job.
|
||||
- **RPC reply queues** are exclusive, auto-delete and server-named; **RPC serve queues**
|
||||
`serve.<key>` are durable and shared, so several runtimes serving one tool key compete rather than
|
||||
each answer.
|
||||
|
||||
### At-least-once, and consumers are idempotent
|
||||
|
||||
A handler may see an event twice — a redelivery after a crash between doing the work and acking.
|
||||
Consumers must be idempotent, and `x-event-id` is what makes dedup possible. **Exactly-once is not
|
||||
offered**: it is a promise no broker keeps honestly, and saying so is better than pretending.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The event shape is a versioned, enforced contract, not conventions each module reinvents. The
|
||||
sdk's `emit`/`on` and the runtime's AMQP binding implement it; a module never sees an exchange or
|
||||
queue name.
|
||||
- Metadata-in-headers means the body is exactly the domain payload, and a consumer that only wants
|
||||
provenance never parses it.
|
||||
- Adding a header or an origin root widens the contract and is reviewed as one — the discipline
|
||||
[ADR 0010](0010-delivery.md) applies to the
|
||||
declaration vocabulary.
|
||||
- The sdk's first cut carried source/node/time in the *body*; this supersedes that — they move to
|
||||
headers. That is code to align, in `mesh-sdk` (`emit`/`on`) and `mesh-tools` (the binding, queue
|
||||
config, dead-letter).
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — events as a relationship; this is their wire shape.
|
||||
- [ADR 0010](0010-delivery.md) — the precedent: a wire
|
||||
contract, versioned, additions reviewed as security.
|
||||
- [ADR 0002](0002-nodes-communicate-over-a-broker.md) — the broker.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — `emit`/`on` are stable sdk surface; the
|
||||
binding, queue config and dead-letter are the runtime's, not the sdk's.
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0041-events-are-a-relationship.md
|
||||
---
|
||||
|
||||
# 43. A module's broker account is scoped by what it emits and consumes
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0041](0041-events-are-a-relationship.md) made events a relationship — `emits` and `consumes`
|
||||
on the manifest. [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) gave them a wire shape — the
|
||||
`mesh.events` exchange, the durable per-consumer queue, the reserved routing-key origins. Neither
|
||||
said how a module *reaches* the broker: what account it holds, and what that account is allowed to
|
||||
do.
|
||||
|
||||
As the code stands, there is no answer. The mesh can provision a **node** account (at enrolment)
|
||||
and a **builder** account (scoped to the build queue), and it can *deliver* any module a sealed
|
||||
own-secret at a declared path — but it has no way to provision a broker **account** for a general
|
||||
module. A module that declares `own-secrets: {broker: …}` and nothing more receives thirty-two
|
||||
random bytes, not a credential. So on the broker, `emits` and `consumes` are enforced by nothing: a
|
||||
running module could bind any queue, consume any pattern, and publish under any origin, and the
|
||||
manifest that says otherwise would be describing a boundary no code draws — the exact shape of fault
|
||||
[04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) records, a scope
|
||||
declared in manifests and read by nothing.
|
||||
|
||||
This settles it, so a module's place on the bus is a thing the broker enforces rather than a thing
|
||||
the manifest merely claims.
|
||||
|
||||
## Decision
|
||||
|
||||
### A module gets a broker account when it is assigned, and its permissions are the manifest
|
||||
|
||||
When the mesh assigns a module to a node it provisions a broker account for that module on that node,
|
||||
sealed to the node ([ADR 0004](0004-a-node-and-how-it-joins.md)) and delivered as the
|
||||
module's `own-secrets` broker — `amqps://` with the mesh's fingerprint, the shape
|
||||
[ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) already carries. The account's permissions are
|
||||
derived from the manifest, and are exactly these:
|
||||
|
||||
- **What it consumes.** Read on `mesh.events`, and configure-and-read on its own queue
|
||||
`<node>.<module>.events` bound to the patterns in `consumes`. It cannot bind or read another
|
||||
module's queue. A module that consumes nothing gets no read on the events exchange at all.
|
||||
- **What it emits.** Write to `mesh.events`, restricted to routing keys under its own origin,
|
||||
`module.<name>.*`. It cannot publish as another module, and cannot publish under the reserved
|
||||
`mesh.*` or `node.*` origins — those belong to the mesh and the host (ADR 0042). A module that
|
||||
emits nothing gets no write.
|
||||
- **Nothing else.** The events account reaches `mesh.events` and that module's own queue, and no
|
||||
more. Tool serving and calling over `mesh.rpc` is a separate grant on the same principle — a
|
||||
module serves the tool keys it declares and calls the ones it is bound to — and is scoped the same
|
||||
way rather than folded in here.
|
||||
|
||||
### Consuming everything is a privilege, granted deliberately
|
||||
|
||||
`consumes: ["#"]` — the audit logger — is read across the whole bus: every module's events, the
|
||||
mesh's, every node's. That is not a pattern like any other; it is the power to see everything, and
|
||||
the account is where it becomes visible. The grant that lets one module read the entire bus is one
|
||||
the mesh issues on purpose and can be audited — the answer to *who can read everything* is a row, not
|
||||
a guess — rather than a breadth any manifest acquires by typing a single character. A `#` consume is
|
||||
a reviewed grant, not a default one.
|
||||
|
||||
### The account is how the declaration is enforced
|
||||
|
||||
Because the account can do only what `emits` and `consumes` name, the broker itself refuses a module
|
||||
that tries to consume a queue it did not declare or emit under an origin it does not own. That is what
|
||||
makes an event relationship a rule and not a comment — the discipline that a stated rule says how it
|
||||
is checked. A manifest that over-declares grants more than the module uses, which is visible and
|
||||
reviewable; one that under-declares makes the module fail closed at the broker, which is the safe
|
||||
direction to be wrong in.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The control plane gains a **generic module broker-account**, derived from the manifest. The
|
||||
builder stops being a special case: its access to the build queue becomes an ordinary expression of
|
||||
what it consumes and serves, not a bespoke account method. One rule, and the builder is an instance
|
||||
of it.
|
||||
- The runtime reads its credential from a file (the broker own-secret), `amqps://` verified against
|
||||
the mesh's fingerprint. The `guest` account is for raising the substrate, never for a module — a
|
||||
module documented as holding its own credential and handed the broker's administrative one is worse
|
||||
than one with no credential story at all.
|
||||
- `emits` and `consumes` stop being advisory. They are the module's authority on the bus, so the
|
||||
manifest is now a security boundary and is reviewed as one, the discipline
|
||||
[ADR 0010](0010-delivery.md) applies to the declaration
|
||||
vocabulary.
|
||||
- *Who can read the whole bus* becomes an answerable question, because `#` is a grant and not an
|
||||
accident.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — events are a relationship; this scopes the account
|
||||
by that relationship.
|
||||
- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) — the wire this account secures: the queue,
|
||||
the origins, the `amqps` credential shape.
|
||||
- [ADR 0004](0004-a-node-and-how-it-joins.md) — the link is the security boundary; a
|
||||
module's account is sealed to its node the same way a node's is.
|
||||
- [ADR 0010](0010-delivery.md) — a declaration is owned
|
||||
and its additions reviewed; a module's broker permissions are that discipline applied to the bus.
|
||||
- [04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) — a scope declared
|
||||
in manifests and enforced by no code: the fault this decision closes for events.
|
||||
@@ -1,98 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 44. A public name is provisioned, not registered by hand
|
||||
|
||||
> **The mechanism changed — 2026-10-06, by [ADR 0226](0226-the-private-network-is-assigned-by-its-own-name-and-the-proxy-names-its-public-issuer.md).** The interface stands; its one provider,
|
||||
> `cloudflare-dns`, left the catalogue, assigned nowhere and required by nothing. A mesh that needs a
|
||||
> public name made for it adds a provider of `public-dns` again.
|
||||
|
||||
## Context
|
||||
|
||||
The mesh names and resolves its own machines internally: the overlay generates
|
||||
`<service>.<node>.<suffix>` wildcards, dnsmasq answers them (`wildcard-resolution`), and the mesh
|
||||
issues a certificate for each internal name. A service reachable at a *public* domain —
|
||||
`plex.example.com`, not `plex.anchor.internal` — needs three things that machinery does not give it:
|
||||
|
||||
- a **public DNS record** at a registrar or DNS provider, so the name resolves on the internet;
|
||||
- a **publicly-trusted certificate** for it, because the mesh's own authority is trusted by nobody
|
||||
outside the mesh;
|
||||
- and routing from that name to the module — which the reverse proxy already does: a module
|
||||
`requires` the `route` capability and the proxy provides it, routing by the host it was asked for.
|
||||
|
||||
The routing exists. The public DNS record does not: the mesh has no way to make a name resolve on
|
||||
the public internet, so today that is a step someone does by hand at a DNS provider, outside the
|
||||
mesh, remembered nowhere. A public name is therefore the one part of reaching a service that the
|
||||
declaration graph cannot grant or withdraw — which means it is created once and outlives whatever it
|
||||
was for, the shape of drift this project exists to remove.
|
||||
|
||||
## Decision
|
||||
|
||||
### A public name is a capability, requested like any other
|
||||
|
||||
A module reachable at a public host declares `requires: ["public-dns"]` and contributes the hostname
|
||||
it wants — beside `requires: ["route"]`, which exposes it through the proxy. The name is then
|
||||
provisioned on declaration ([ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md)): created
|
||||
when the module is assigned, removed when it is withdrawn, reconciled like every provision.
|
||||
|
||||
### The interface is neutral; the providers are the registrars
|
||||
|
||||
`public-dns` is drawn at the consumer's coupling: the consumer wants *a public name that resolves to
|
||||
me*, and does not care whether Cloudflare, Route 53 or a registrar's own API puts the record there.
|
||||
So the interface is neutral and the providers are provider-scoped — `cloudflare-dns`,
|
||||
`route53-dns`, `porkbun-dns` — each implementing the one `public-dns` contract, the same way a
|
||||
neutral database coupling is answered by `postgres-database` and `mssql-database`. A module names
|
||||
`public-dns`; it never names a registrar.
|
||||
|
||||
### The record points at the mesh's public ingress, not at the node
|
||||
|
||||
What the name resolves to is the address the reverse proxy answers on, not the consuming machine's.
|
||||
A public service is reachable only *through* the proxy — the proxy holds the `route` grant and routes
|
||||
by host to the module — so the public name must resolve to the proxy. `public-dns` and `route` are
|
||||
the two halves of one public exposure: the name, and what the name reaches.
|
||||
|
||||
### The record is a fact, not a secret
|
||||
|
||||
A DNS record is public by definition, so the grant returns the fully-qualified name and its TTL and
|
||||
nothing sealed. The only secret is the provider's own API credential, which is the provider module's
|
||||
own-secret and never leaves it — the module that wanted the name never sees it.
|
||||
|
||||
### Events
|
||||
|
||||
The provider emits `module.<provider>.record.created` and `module.<provider>.record.removed`
|
||||
([ADR 0041](0041-events-are-a-relationship.md)), so *which names the mesh publishes, and where* is a
|
||||
question answered from the event trail and the grants, not from a folder of records edited at a
|
||||
provider.
|
||||
|
||||
### The public certificate is the proxy's, and is named here only to pair it
|
||||
|
||||
A public name without a publicly-trusted certificate is reachable and not trusted — the same pairing
|
||||
the internal name and the mesh-issued certificate already have. Obtaining that certificate (ACME
|
||||
against the now-resolving public name) is the reverse proxy's to do, and its mechanism is its own
|
||||
decision; it is named here so the pairing is not forgotten, not resolved here.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A public name is created and torn down with the module, so it cannot outlive it, and the mesh can
|
||||
say which public names it publishes without anyone reading a registrar's dashboard.
|
||||
- Adding a registrar is adding a provider that answers `public-dns`; the modules that want names do
|
||||
not change.
|
||||
- Public exposure of a service is a trio of separate, declared, enforced relationships: the firewall
|
||||
opens the proxy's public port ([ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)),
|
||||
`route` routes the host to the module, and `public-dns` makes the host resolve.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — a capability is provisioned on
|
||||
declaration; a public name is one.
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the firewall, the other
|
||||
half of the reachability question this was asked with.
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — the provider's record events.
|
||||
- [ADR 0040](0040-what-a-module-is.md) — a provider and its interface; the neutral-interface,
|
||||
scoped-provider naming this follows.
|
||||
@@ -1,98 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 45. A machine's firewall is the sum of what its modules listen on
|
||||
|
||||
> **Superseded in part — 2026-09-28, by [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md).**
|
||||
> That a listen's `from` "is the whole of public-versus-internal" no longer holds: an assignment states each
|
||||
> endpoint's reach, a statement that also governs names and certificates, and `from` is the packet filter's
|
||||
> share of it. The firewall derived and applied on the node, not a provider, stands as decided here.
|
||||
|
||||
## Context
|
||||
|
||||
The reverse proxy is a *provider*: a module `requires` the `route` capability and a running proxy
|
||||
provides it, routing traffic by name and reaching back to the consumer. A fair question follows —
|
||||
is the firewall the same shape? Should a module *register* a port with a firewall provider the way
|
||||
it requests a route?
|
||||
|
||||
It should not, and the difference is the point. A reverse proxy is a service another component
|
||||
performs; a firewall is a property of the machine — a packet filter the host applies to itself.
|
||||
Modelling it as a provider would invent a credential and a reach-back for something that has neither.
|
||||
|
||||
And the mesh already has the registration: a module declares `listens: [{ port, from }]` — the port
|
||||
it accepts connections on, and from where. That *is* how a service says it wants a port open. What is
|
||||
missing is not a model but enforcement. [04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md)
|
||||
records that a `scope:` key five manifests carry is read by no code: a manifest can appear to
|
||||
restrict a port and restrict nothing — the exact fault
|
||||
[how-we-build.md](../00-META/how-we-build.md) names, *an unenforced rule is indistinguishable from a
|
||||
wrong one*, made worse because the declaration reads as a restriction.
|
||||
|
||||
## Decision
|
||||
|
||||
### The firewall is derived and host-applied, not a provider
|
||||
|
||||
A machine's firewall is the sum of what the modules assigned to it declare they listen on, computed
|
||||
by the host and applied as one of its owned resources ([ADR 0005](0005-the-node-host.md):
|
||||
the host applies, it does not decide; [ADR 0010](0010-delivery.md):
|
||||
the declaration is owned resources). It is not a capability, not a per-consumer grant — opening a
|
||||
port is a declarative fact about a machine, so it is computed and applied, not requested and
|
||||
credentialed.
|
||||
|
||||
### `from` is the whole of public-versus-internal
|
||||
|
||||
The distinction the question is really about lives in `from`:
|
||||
|
||||
- `listens: [{ port: 5432, from: mesh }]` — open to the private overlay only.
|
||||
- `listens: [{ port: 443, from: anywhere }]` — open to the public internet.
|
||||
|
||||
A module registers a port on the firewall by listening on it and saying from where. There is no
|
||||
separate firewall capability, because the firewall is not a thing that reaches back or holds a
|
||||
secret; it is the machine's own filter over the ports its modules named.
|
||||
|
||||
### The host enforces it both ways, and unknown keys are refused
|
||||
|
||||
A port a module listens on is opened to exactly the scope it named; a port nothing declares is
|
||||
closed. And a key the firewall does not read — the `scope:` of issue 003 — is refused at the
|
||||
manifest, not accepted and ignored, so a declaration that reads as a restriction is one. This is the
|
||||
discipline [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) applied to the
|
||||
broker account, applied here to the packet filter: the declaration is the enforcement, or it is a
|
||||
comment.
|
||||
|
||||
### A public service is exposed through the proxy, not by opening its own port
|
||||
|
||||
Reaching the public internet is normally not `from: anywhere` on the service's own port. The service
|
||||
listens `from: mesh` — only the proxy reaches it — and `requires: route`, so the sole machine with a
|
||||
public opening is the one running the reverse proxy, and the service is exposed by name through it.
|
||||
`from: anywhere` is the deliberate direct-exposure case, for a service that is its own front door.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Issue 003 is closed: the firewall is computed from `listens` and enforced, so a declared scope is
|
||||
real and an undeclared port is shut. Rejecting unknown manifest keys is the general fix, of which
|
||||
the `scope:` key was one instance.
|
||||
- The firewall and the reverse proxy stop being confused for one model: the firewall is the machine's
|
||||
filter (host-derived from `listens.from`); `route` is a name-router (a provider); the public DNS
|
||||
name is a third thing ([ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md)). A
|
||||
public service uses all three.
|
||||
- The modelling question is answered: a module registers a port by declaring `listens`, and reaches
|
||||
the public internet by name through `route` + `public-dns` — never by the firewall being a
|
||||
provider.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0005](0005-the-node-host.md) — the host applies; the firewall is one of
|
||||
the things it applies.
|
||||
- [ADR 0010](0010-delivery.md) — the firewall is a derived
|
||||
owned resource, not a grant.
|
||||
- [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) — the same discipline:
|
||||
a declaration is enforced, or it is a comment.
|
||||
- [ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md) — the public name, the other
|
||||
half of the reachability question this was asked with.
|
||||
- [04-ISSUES/003](../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) — the unenforced
|
||||
`scope:` this closes.
|
||||
@@ -1,88 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 46. A module's configuration is its assignment's, not its manifest's
|
||||
|
||||
## Context
|
||||
|
||||
A module is assigned to a node — `assign <node> <module>`, always to a machine; there is no
|
||||
assignment to the mesh. "Mesh" is a *scope*, not a place: a `provides` or a `claim` scoped `mesh`
|
||||
reaches the whole mesh, but the module still runs on a node. So the two kinds of thing a module can
|
||||
carry are the manifest (what the module *is*) and, separately, what it should do *here* — which
|
||||
differs by deployment and by node.
|
||||
|
||||
The mesh already has the second: **settings**. `settings set <module> [--node <node>]` — with a node
|
||||
it is that machine's, without it the whole mesh's — layered over what the module declares and applied
|
||||
at resolution, changeable without editing the module and without a rebuild. That is the surface a
|
||||
meshboard would edit.
|
||||
|
||||
But settings today reach only a module's **config-file content** (a mergeable file the module owns).
|
||||
Configuration that is not a file has been landing in the manifest instead, statically — a registrar's
|
||||
zone and domain, the address public names point at, and, most sharply, `listens.from`. That last one
|
||||
is the tell: whether a port is open to the private overlay or to the public internet is a
|
||||
*per-node deployment choice* — the same database internal on one machine and public on another — and
|
||||
a value fixed in the manifest is one value for every machine, so it cannot be. Static configuration in
|
||||
the manifest is configuration in the wrong place: it cannot vary per node, and it cannot change
|
||||
without a new module version.
|
||||
|
||||
## Decision
|
||||
|
||||
### The manifest is identity and defaults; the assignment's settings are the configuration
|
||||
|
||||
A module's manifest declares what it is — what it provides, requires and claims, the shape of its
|
||||
resources — and, for anything configurable, a **default**. The values that make a running instance
|
||||
*this* instance are settings, carried by the assignment: per-node, or mesh-wide when no node is named,
|
||||
applied over the defaults at resolution. Change one and the next reconcile carries it; nothing is
|
||||
edited on a machine and nothing is rebuilt.
|
||||
|
||||
### Settings drive the configurable fields the manifest marks, not only file content
|
||||
|
||||
Settings extend beyond a config file's content to the manifest fields a module declares settable —
|
||||
foremost:
|
||||
|
||||
- **`listens.from`**: a module declares its safe default (`from: mesh`), and a per-node setting
|
||||
raises or lowers it. postgres declares `listens: [{ port: 5432, from: mesh }]`; on the machine that
|
||||
should expose it, a setting makes that port `from: anywhere`. Same module, different exposure, and
|
||||
the firewall ([ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md)) is computed
|
||||
from the effective value, so the packet filter follows the setting.
|
||||
- **A provider's own configuration**: a registrar's zone, domain and the ingress its names point at
|
||||
([ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md)) are mesh-wide settings, not
|
||||
manifest constants — one mesh's Cloudflare zone is not another's, and the module description is the
|
||||
same for both.
|
||||
|
||||
### Unset is the default, and an unknown setting is refused
|
||||
|
||||
A field with no setting keeps the manifest's default, so a module runs correctly configured by nobody.
|
||||
A setting that matches no settable field — like a config value that reaches no file today — is named,
|
||||
not silently dropped, so a misspelled setting is found rather than believed (the discipline of
|
||||
`UnusedSettings`, and of [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md):
|
||||
a declaration is enforced or it is a comment).
|
||||
|
||||
## Consequences
|
||||
|
||||
- The postgres case works: one module, `from: mesh` by default, `from: anywhere` where a setting says
|
||||
so — internal on ace, public on novox, changeable live.
|
||||
- Provider modules stop carrying a mesh's specifics: `cloudflare-dns` describes *a Cloudflare
|
||||
registrar*, and *which* zone and ingress is a setting, so the same module serves every mesh.
|
||||
- Configuration becomes a thing a meshboard manages — set per node or mesh-wide, applied on the next
|
||||
reconcile — rather than a manifest edit and a rebuild ([ADR 0011](0011-managed-files-are-generated-never-edited.md):
|
||||
the way you change a managed thing is not by editing it).
|
||||
- What a manifest may not do is grow a value that differs per machine; if it differs per machine it is
|
||||
a setting, and the manifest holds only the default.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — what is provisioned on
|
||||
declaration; its per-instance values are the assignment's.
|
||||
- [ADR 0011](0011-managed-files-are-generated-never-edited.md) — a managed thing is changed through
|
||||
the mesh, not by editing it; settings are that, for configuration.
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the firewall follows the
|
||||
effective `listens.from`, so making `from` a setting makes exposure a setting.
|
||||
- [ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md) — the provider whose zone and
|
||||
ingress are settings, not manifest constants.
|
||||
@@ -1,97 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md
|
||||
---
|
||||
|
||||
# 47. A module runs its code as its own process, with its own account
|
||||
|
||||
> **The mechanism changed — 2026-10-02, by [ADR 0175](0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md).** A module's *tools* are no longer served by the module's own process under its own account: one tool runtime per node, on the host side, serves every assigned module's bundle. A tool is still served on its own subject and only the module that serves it answers; what moved is the process and the account.
|
||||
|
||||
## Context
|
||||
|
||||
A module is one self-contained thing ([ADR 0040](0040-what-a-module-is.md)), and it gets a broker
|
||||
account scoped to what it emits and consumes ([ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md)).
|
||||
The catalogue now gives modules **tools** and **events** — real code, in the module ([ADR 0039](0039-what-the-sdk-holds-and-refuses.md)) —
|
||||
but nothing has said what *runs* that code. The audit-logger showed one shape and was treated as an
|
||||
exception: a container running the tool runtime carrying the module's compiled code, holding the
|
||||
module's own scoped account. Every module with tools or events needs the same, and the tempting
|
||||
alternative does not work.
|
||||
|
||||
**A node-wide runtime that loaded every assigned module's code cannot hold a per-module account.** It
|
||||
would run under one account with the union of every module's permissions — able to emit as any of
|
||||
them and read any of their queues — which is exactly the isolation ADR 0043 exists to draw. So the
|
||||
runtime is per-module, not per-node, and treating the audit-logger as special left the other
|
||||
modules' code with nothing to run it: the conversion produced tools and events that, as it stands,
|
||||
never execute.
|
||||
|
||||
> **The hosting form is settled elsewhere — 2026-09-30.** Where this record says "a container", read
|
||||
> [ADR 0150](0150-a-modules-own-code-runs-as-supervised-processes-under-one-account.md): a module's own
|
||||
> code runs as supervised processes under this record's one account. Nothing else here changes — the
|
||||
> per-module runtime, the per-tool key and the single scoped account are the argument this record made
|
||||
> and they are why 0150 goes the way it does. The note is here because two design documents chose the
|
||||
> other form without knowing this record existed
|
||||
> ([issue 117](../04-ISSUES/117-a-modules-own-code-is-a-container-and-a-process/00-report.md)).
|
||||
|
||||
## Decision
|
||||
|
||||
### A module with tools or events runs a process of its own
|
||||
|
||||
A module that has tools or events runs a **runtime process** — a container, the tool runtime carrying
|
||||
that module's compiled code — assigned and started like the module it is, holding the single broker
|
||||
account the mesh scoped to it (ADR 0043). One module, one process, one account.
|
||||
|
||||
### It serves its tools, each on its own key
|
||||
|
||||
A tool is served on its own key (`serve.<tool>`), and a caller invokes a named tool. Only the module
|
||||
that serves it answers, and the module's account is scoped to exactly its tool keys — so one module
|
||||
cannot answer another's calls, the isolation ADR 0043 gives events extended to tools. This supersedes
|
||||
a single `tools.invoke` endpoint that dispatched by name: that shape assumed one runtime for the
|
||||
whole node, and per-module runtimes competing on one key would each be handed calls for tools they do
|
||||
not have.
|
||||
|
||||
### It runs its events in the same process, under the same account
|
||||
|
||||
Emitting under the module's own origin and consuming its own queue ([ADR 0042](0042-the-shape-of-an-event-on-the-wire.md))
|
||||
happen in that same process, with that same account — not a second one to scope and seal. A module's
|
||||
tool code, its event code and, for a provider, its provisioner are the one module's code and run as
|
||||
the one module's process.
|
||||
|
||||
### The runtime image is the tool runtime plus the module's code
|
||||
|
||||
Built from the module's source like any module image — the audit-logger's shape, made the rule, not
|
||||
the exception. The module declares a `container` for it carrying `MESH_BROKER_FILE` (its sealed
|
||||
credential, ADR 0043) and its compiled code. A module with **neither** tools nor events runs no such
|
||||
process: a plain service module — the plex *server*, dnsmasq the resolver — is its service and files
|
||||
and nothing more. A module that is both a service and code declares both containers: the service, and
|
||||
the runtime beside it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The catalogue's tools and events become runnable: each tools-or-events module gains a runtime
|
||||
container with its scoped credential, and the audit-logger stops being special. Until this, the
|
||||
converted modules held code with nothing to execute it.
|
||||
- A process, and a small image, per tools-or-events module. That is the cost of ADR 0043's isolation:
|
||||
one account per module means one process per module. It is paid deliberately — a shared runtime is
|
||||
cheaper and cannot be scoped, and a mesh where any module can emit as any other is not one worth the
|
||||
saving.
|
||||
- `serve.<tool>` per key replaces the single `tools.invoke` dispatch. The sdk's serving and a module's
|
||||
account scope both come to name tools individually.
|
||||
- **A provider's provisioner is a runtime process too.** It already runs as its own container; its
|
||||
events (`bucket.created`, `database.provisioned`) belong to *that* process and need the same
|
||||
credential. So a provisioner that emits carries `MESH_BROKER_FILE` and its scoped account like any
|
||||
runtime — or it does not emit. (This is the fix for provisioners that emit today with no broker
|
||||
bound: the emit is a runtime's, and the provisioner is a runtime.)
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0040](0040-what-a-module-is.md) — a module is one self-contained thing; its code runs as one
|
||||
process.
|
||||
- [ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md) — the scoped account
|
||||
this process holds, and the isolation that makes it per-module.
|
||||
- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) — the events this process runs, and the
|
||||
`serve.<key>` queue tools now use.
|
||||
- [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) — the code lives in the module; this runs it.
|
||||
@@ -1,131 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 48. A provider creates the credential the mesh minted, and seals nothing
|
||||
|
||||
## Context
|
||||
|
||||
A provider module stands up a per-consumer resource — a database, a cache bucket, an object
|
||||
store user — and the consumer must end up holding a credential that authenticates against it.
|
||||
Building the module runtime (ADR 0047: a module runs its own code as its own process under its
|
||||
own account), the provider's provisioner was run for the first time as a delivered thing, and
|
||||
it did not work. It reads a seal key from the environment that nothing sets, and it seals every
|
||||
credential it produces to that key with a symmetric passphrase.
|
||||
|
||||
Tracing the credential's path turned up something larger than a missing key. **The provisioner
|
||||
harness the whole catalogue is built on describes a credential flow the mesh does not have, and
|
||||
duplicates — incorrectly — one it does.**
|
||||
|
||||
What the sdk's `runProvisioner` does today:
|
||||
|
||||
- reads request files named `*.grant.json` — which nothing in the mesh writes;
|
||||
- calls an adapter whose `create` **generates its own password** and returns it;
|
||||
- seals that password with a symmetric key (`$MESH_SEAL_KEY`) and writes a `*.credential`
|
||||
file — which nothing in the mesh reads, and no consumer ever unseals.
|
||||
|
||||
What the mesh already does, and has wired end to end:
|
||||
|
||||
- The control plane mints one password per (consumer, provider) pair (`Inventory.SecretFor` →
|
||||
`secrets.Make`) and seals it to **both** node keys asymmetrically — a copy the consumer's
|
||||
host can open and a copy the provider's host can open. No shared symmetric key exists
|
||||
anywhere, on purpose: a key both ends hold is a key the mesh would have to distribute, which
|
||||
is the same problem one level down, and the control plane's own code refuses it.
|
||||
- The provider is handed, at the path its `receives` names, one contribution per consumer:
|
||||
the **login to create** (`As`, derived by the mesh so the two ends agree by construction),
|
||||
the consumer's address and requested values, and a **`Secret` file** holding that consumer's
|
||||
password sealed to the provider and unsealed onto the machine by its host.
|
||||
- The consumer is handed the *same* password, as plaintext its own host wrote by unsealing its
|
||||
copy and substituting it into a config file. The consumer never unseals anything itself and
|
||||
holds no key.
|
||||
|
||||
So the password a provider's provisioner invents is not even the password the consumer was
|
||||
given: a consumer authenticating with the mesh's password against a resource the provisioner
|
||||
created with its own would simply fail. The symmetric seal is not an incomplete feature to
|
||||
finish delivering a key for. It is a second, contradictory credential model bolted beside the
|
||||
real one, and it cannot be made to work without building the very thing the mesh was designed
|
||||
not to have.
|
||||
|
||||
This is a decision and not a patch because the harness is the **provider contract**. Every
|
||||
provider — the four that exist and the many a real mesh grows — is built on
|
||||
`runProvisioner(resource, adapter)`. Whatever it says a provider is, they all inherit; and
|
||||
changing it later is one migration per provider. It is cheaper and more honest to settle what
|
||||
a provider is now.
|
||||
|
||||
## Decision
|
||||
|
||||
**A provider is handed the credential; it does not make one, does not seal one, and does not
|
||||
hand one back.** The provisioner's only job is to make the mesh's grants true in its own
|
||||
software.
|
||||
|
||||
Concretely, for the sdk harness and the adapter contract:
|
||||
|
||||
- The harness reconciles the **contributions the mesh delivers** to the provider's `receives`
|
||||
path — the list of consumers, each with its login name (`As`), address, requested values,
|
||||
and the path to its unsealed password (`Secret`). It does not read `*.grant.json` and it
|
||||
does not write `*.credential`.
|
||||
- For each consumer present, the harness reads the password from that consumer's `Secret` file
|
||||
and calls the adapter to bring the resource into being under the given login. For each
|
||||
consumer no longer present — the mesh drops it from the contributions file when its consumer
|
||||
goes away — the harness calls the adapter to withdraw it.
|
||||
- The adapter shrinks to the per-software half and nothing else. It is given the login, the
|
||||
password, and the values, and it makes the resource exist or removes it. It generates no
|
||||
password, derives no name, seals nothing, and returns no credential:
|
||||
roughly `create({ as, password, values })` and `remove({ as })`, both returning nothing.
|
||||
- `$MESH_SEAL_KEY`, the symmetric `seal()`/`writeSealedCredential` path, and the `*.grant.json`
|
||||
/ `*.credential` files are removed from the provisioning path entirely. The credential
|
||||
reaches the consumer through the mesh's own asymmetric channel, which already crosses node
|
||||
boundaries and holds no shared secret.
|
||||
|
||||
Identity stays the mesh's to say. The login the provider creates is the name the mesh derived
|
||||
and gave the consumer to present; the provider never invents a name, because a name the
|
||||
consumer cannot learn is a name it cannot authenticate with.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A provider module becomes smaller and unable to be wrong in this way: with no password to
|
||||
generate and no key to seal to, the class of bug where the two ends hold different secrets
|
||||
cannot be written. A provider added after this inherits the corrected contract and has no
|
||||
seal to reintroduce.
|
||||
- The four current providers (redis, postgres, minio, umami) each lose their `generatePassword`
|
||||
+ seal code and gain a `create` that takes the password it is given. Their teardown becomes
|
||||
"withdraw the login named `As`".
|
||||
- The symmetric `seal()`/`unseal()` primitive loses its only caller and leaves — checked, not
|
||||
assumed: nothing else in the sdk or the catalogue called it, so it is removed with the
|
||||
provisioner it belonged to.
|
||||
- **How this is verified:** redis is assigned as a provider in the lab, the contributions and
|
||||
the unsealed password the mesh would deliver are put in its `receives` path, and a client
|
||||
authenticates as that consumer with the mesh's password and gets PONG — where a provider that
|
||||
invented its own password answers WRONGPASS — with `$MESH_SEAL_KEY` set nowhere and no
|
||||
`.credential` file written. Proven: `provider-uses-mesh-credential` is green.
|
||||
|
||||
**What this does not cover — credential provisions, not data provisions.** This decision is about a
|
||||
provision whose credential is a *secret the mesh mints* — a login and password (redis, postgres,
|
||||
minio). A provider that instead *generates* the thing the consumer needs, and that thing is not a
|
||||
secret — umami's `analytics`, where the consumer wants back a `siteId` umami assigned — does not fit,
|
||||
because a contract that returns nothing has no way to hand that data back. The seal-key fault was
|
||||
never umami's (it sealed no password; it returned a public id), so removing the seal does not break
|
||||
it further, and it still reconciles its sites off the mesh's contributions. But delivering
|
||||
provider-generated data back to a consumer is a *return path* the mesh does not have and this
|
||||
decision does not build — a separate shape, left to a separate decision.
|
||||
- Teardown beyond "remove the login" — data an object store leaves behind when a consumer
|
||||
leaves — is named by each provider's adapter, not by the harness, and is out of scope here
|
||||
except to say the contract must leave room for it.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/032](../04-ISSUES/032-provider-runtime-has-no-seal-key/00-report.md) — the
|
||||
observation and the cross-repo trace this decision rests on.
|
||||
- ADR 0047 (the module runtime) — what first ran a provider's provisioner as a delivered
|
||||
process and exposed this; link to be filled when 0047 lands on the trunk.
|
||||
- Control-plane mechanisms this relies on already existing: `mesh-control` —
|
||||
`internal/inventory/secrets.go` (`SecretFor`, `SecretsFrom`), `internal/secrets/seal.go`
|
||||
(`Make`, the two-blob asymmetric sealing), `cmd/mesh-control/plan.go` (`grantsFor`, the
|
||||
`Grant.Sealed = ForProvider` delivery), `internal/catalogue/declaration.go` (the `receives`
|
||||
contribution: `As`, `At`, `Values`, `Secret`).
|
||||
- The path being removed: `mesh-sdk` — `src/provisioner/index.ts` (`runProvisioner`, `sealKey`,
|
||||
`writeSealedCredential`) and the symmetric `src/primitives/index.ts` `seal()`/`unseal()`.
|
||||
@@ -1,138 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 49. A consumer's identity is bounded by the tightest backend that must accept it
|
||||
|
||||
## Context
|
||||
|
||||
The mesh says who a consumer is, once, and hands the same name to the provider (to create) and the
|
||||
consumer (to present), so the two ends agree by construction rather than by two conventions (the
|
||||
principle behind `ConsumerIdentity`, 04-ISSUES/023). The name is `mesh_<node>_<module>`, cleaned to
|
||||
lower-case letters, digits and underscore.
|
||||
|
||||
Proving the provider contract per backend (ADR 0048) turned up 04-ISSUES/034: redis and postgres
|
||||
create that name verbatim, but **minio refuses it** — an S3 access key is capped at 20 characters,
|
||||
and `mesh_anchor_bucketuser` is 22. The provisioner then retries for ever, per consumer, and the
|
||||
consumer holding that same too-long name could never present it either.
|
||||
|
||||
Two things about the existing derivation decide most of this:
|
||||
|
||||
- **The charset is already right.** `[^a-z0-9_]` is deliberately conservative, and its own comment
|
||||
says it reaches "a PostgreSQL role, a MinIO access key, an LDAP uid and a Keycloak client without
|
||||
quoting." That much is true.
|
||||
- **The length is wrong.** `CheckIdentity` refuses names over `identityLimit = 63`, commented as
|
||||
"the shortest identifier limit among the systems these names reach: PostgreSQL's". It is not the
|
||||
shortest — S3's 20 is shorter — so the guard that was meant to catch exactly this lets it through,
|
||||
and the failure lands at provision time as a silent retry instead of at assignment as a refusal.
|
||||
|
||||
So this is a small wrong constant with a real cost attached: whatever bound we set, `mesh_` (5) plus
|
||||
a node name plus `_` plus a module name has to fit inside it.
|
||||
|
||||
## The options
|
||||
|
||||
**A — Bound the identity by the true minimum, and refuse early.** Lower `identityLimit` to the real
|
||||
shortest (20, S3's), so `CheckIdentity` refuses an over-long name *at assignment* with a clear
|
||||
message, the way it already refuses over-63 names. The derivation does not change; long names are
|
||||
simply rejected before anything is provisioned.
|
||||
- *For:* smallest change; keeps "the mesh says the identity once, verbatim" intact; the failure
|
||||
moves from a per-consumer provision-time retry to an up-front, legible refusal — which is what
|
||||
`CheckIdentity` exists to do.
|
||||
- *Against:* a hard budget. `mesh_` + node + `_` + module ≤ 20 means node + module ≤ 14 characters.
|
||||
`anchor` + `bucketuser` (16) is already over. It pushes the constraint onto how machines and
|
||||
modules are named, which is a real limitation on legible names.
|
||||
|
||||
**B — Keep the readable name when it fits, compact it when it does not.** Below the bound, the name
|
||||
is `mesh_<node>_<module>` as today; over it, the mesh substitutes a deterministic short form (e.g.
|
||||
`mesh_` + a truncated hash of node+module) — still one derivation, so both ends still agree.
|
||||
- *For:* no naming constraint; short backends always satisfied; the common case stays legible.
|
||||
- *Against:* some identities become opaque, and a provisioner tracing "whose login is this" loses
|
||||
the answer for exactly the consumers that overflowed. The mesh now owns a fallback format and its
|
||||
collision properties (a truncated hash is not free of collisions at 15 characters).
|
||||
|
||||
**C — Let each interface declare its identifier bounds, and derive within the tightest a consumer
|
||||
reaches.** `s3-bucket` states `identifier: { max: 20 }`; `postgres-database` states 63; the mesh
|
||||
derives a name that fits the **minimum** bound across the providers a given consumer is granted.
|
||||
- *For:* the most precise — each provision gets exactly the room it has, and a database consumer
|
||||
keeps long legible names while an S3 consumer gets a short one; the constraint lives where the
|
||||
fact does (on the interface).
|
||||
- *Against:* the most work, and a consumer of two interfaces with different bounds must satisfy the
|
||||
smaller — so its name shortens for both, reintroducing B's opacity in a narrower case. It also
|
||||
means one consumer can hold **different** identities per provision, which the "said once" model
|
||||
currently forbids.
|
||||
|
||||
**D — Let the provider generate a backend-valid identity and hand it back (rejected).** minio mints
|
||||
its own access key and returns it to the consumer. This is the data-provision return path this era
|
||||
keeps meeting — but it directly contradicts 023 and ADR 0048: the identity would no longer be the
|
||||
mesh's single derivation the two ends share, it would be a value one side invents and the other must
|
||||
be told. Listed for completeness; not recommended.
|
||||
|
||||
**E — A module (and a node) may declare a short slug; the identity is built from it.** The identity
|
||||
becomes `mesh_<node-slug|node-name>_<module-slug|module-name>`: where a slug is declared it is used,
|
||||
otherwise the cleaned name. A slug is a deliberately short, operator-chosen identifier — `kc` for
|
||||
keycloak, `wkstn` for a workstation. It is optional: short names (`anchor`, `redis`) need none.
|
||||
- *For:* this is the escape hatch B wanted to be, without the opacity. The name stays legible — a
|
||||
provisioner can read `mesh_wkstn_kc` and know who is asking — because a person chose it, not a
|
||||
hash function. And it makes an early refusal *palatable*: if even the slug-built identity overflows,
|
||||
the refusal points at the slug, a field made for exactly this, rather than at the machine's name.
|
||||
Both ends still derive it from one declared thing, so they agree by construction.
|
||||
- *Against:* a new optional manifest field, and someone must pick the slug — but only for names that
|
||||
would otherwise overflow, and picking a short legible identifier is a better job than being handed
|
||||
a hash.
|
||||
|
||||
## What implementing A revealed
|
||||
|
||||
A was tried first. At `identityLimit = 20`, the readable budget is `mesh_` (5) + node + `_` + module
|
||||
≤ 20, i.e. **node + module ≤ 14 characters** — far tighter than it looked. The catalogue's own
|
||||
existing tests use `workstation`+`keycloak` (25), which compacts to `mesh_dbbc02f8dde34d3`; common
|
||||
mesh names (`home-server`, `the-build-node`, `workstation`) blow the budget with any module. So B's
|
||||
compact fallback would fire for the *common* case, not the rare overflow — which inverts A+B: most
|
||||
identities would be opaque hashes. A alone (hard refusal at 20) would refuse most realistic names.
|
||||
This is what moved the recommendation to E: the problem is not the limit, it is that the *readable
|
||||
name* is the wrong source when it is long, and a slug is a better source than either a hash or a ban.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**E, over a per-consumer bound (start with the global minimum, 20).** Build the identity from an
|
||||
optional slug, keep it when it fits, and refuse at assignment with "declare or shorten `<module>`'s
|
||||
slug" when it does not — no hash, no lost legibility, and the fix is a first-class field. Set the
|
||||
bound to the true minimum (20) now; it needs no per-interface machinery to unblock S3, and a module
|
||||
that consumes S3 simply declares a short slug. Graduate to **C** (per-interface bounds) later if it
|
||||
turns out that non-S3 consumers are paying for S3's limit often enough to mind — E and C compose:
|
||||
slugs are the mechanism, per-interface bounds refine where the ceiling sits. **B is dropped**: a
|
||||
declared slug is a strictly better escape hatch than an opaque hash. **D stays rejected.**
|
||||
|
||||
## Consequences (of E)
|
||||
|
||||
> **The mechanism changed — 2026-10-06, by [ADR 0225](0225-a-consumers-identity-is-bounded-by-the-provision-it-requires.md).**
|
||||
> Option C, named above as the later refinement, is taken: each offer states the longest identity its
|
||||
> backend keeps, and a consumer is bounded by the provision it requires rather than by 20 everywhere.
|
||||
> 20 stays the bound of the object store and of a provider that is told its consumers and does not
|
||||
> say. The overflow is refused before merge by the catalogue check, and at composition the consumer
|
||||
> is left out of its provider's grants and reported — never the provider's machine refused. What
|
||||
> stands: the identity is said once, the slug is the remedy, nothing is hashed or truncated.
|
||||
|
||||
- A module manifest gains an optional `slug`; a node may carry one too. `ConsumerIdentity` prefers
|
||||
the slug over the cleaned name for each half. `identityLimit` becomes 20 (the true minimum), and
|
||||
`CheckIdentity` refuses at `module add` / assignment — now with a message naming the slug to set.
|
||||
- The common case stays legible; only names that overflow the budget need a slug, and what they get
|
||||
is a name a person chose, not a hash.
|
||||
- Existing modules/nodes whose names overflow declare a slug once — a migration cost paid as a clear
|
||||
refusal with an obvious remedy, not a silent hash or a silent truncation.
|
||||
- minio (04-ISSUES/034) is unblocked: an S3 consumer declares a short slug and its access key fits.
|
||||
- **How it is checked:** the minio grant e2e — a consumer whose (slugged) identity fits reaches its
|
||||
bucket with the credential the mesh delivered — plus unit tests that a slug is preferred, that an
|
||||
un-sluggable over-long identity is refused (naming the slug), and that two consumers never collide.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/034](../04-ISSUES/034-mesh-login-exceeds-s3-access-key-limit/00-report.md) — the
|
||||
observation.
|
||||
- ADR 0048 — a provider creates the credential the mesh minted; the identity it creates it under is
|
||||
the one this decision bounds.
|
||||
- `mesh-control` `internal/catalogue/identity.go` — `ConsumerIdentity`, `identityUnusable`,
|
||||
`identityLimit`, `CheckIdentity` — where the constant and the check live.
|
||||
@@ -1,217 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0024-model-access-is-a-provision.md
|
||||
---
|
||||
|
||||
# 50. Model access is vendor-agnostic, and a vendor is an adapter
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0024](0024-model-access-is-a-provision.md) settled that model access is a provision and that
|
||||
a licence is a named thing an operator uses. What shipped, and runs, is a single vendor: the mesh's
|
||||
"claude" feature. A read-only trace of that feature (2026-09-05, in the code workspace) was made to
|
||||
answer whether the model-access provision is Anthropic-shaped or genuinely general. The finding is
|
||||
that **the vendor-agnostic layer already largely exists**, and the Anthropic specifics are a thin
|
||||
band around it that a per-vendor adapter can hold.
|
||||
|
||||
**What is already general, with evidence.** `mesh-control internal/licences` models
|
||||
`licence(name, vendor, serves)` and `licence_holder(licence, node, module, sealed)`, and each
|
||||
holder's credential is sealed per-holder through `internal/secrets`. `serves` carries the
|
||||
non-secret facts (a base URL, a model) and is not vendor-specific. The `accept` verb
|
||||
([ADR 0024](0024-model-access-is-a-provision.md), and
|
||||
[`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md)) already takes an operator-supplied
|
||||
value, seals it to each holder and discards the plaintext. A model the mesh runs itself answers
|
||||
`model-access` at node scope with no licence at all. None of that mentions Anthropic.
|
||||
|
||||
**What is Anthropic-specific.** The credential is not a static key: it is a subscription OAuth grant
|
||||
— an hourly access token plus a refresh token. That shape drags four things behind it that a static
|
||||
key does not need: **central rotation** (one manager node refreshes under a lease and publishes the
|
||||
new token), **delivery that strips the refresh token** so a consuming node holds only an access token,
|
||||
an **identity guard** that reads the credential to catch a mis-binding, and a **usage** reading with
|
||||
Anthropic's own `utilization%` semantics. Most vendors are a single static key, which the sealed-key
|
||||
model already handles and which needs none of these four.
|
||||
|
||||
**The tension at the centre of this.** A refreshable credential cannot be both *sealed so the mesh
|
||||
cannot read it* and *rotated centrally*. Central rotation means some node in the mesh holds the
|
||||
refresh token in readable form, because that is what refreshing requires. Per-holder sealing means no
|
||||
node but the holder can read the credential. For a static key the two never meet — there is nothing to
|
||||
rotate. For a refreshable grant they collide directly, and this record exists to say which gives way,
|
||||
and by how much.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep Anthropic special-cased in the core.** Leave the three binding columns and the `claude_*`
|
||||
schema, and add other vendors beside them the same way. **Rejected.** It is exactly what
|
||||
[ADR 0024](0024-model-access-is-a-provision.md) ruled against: a module that names a vendor cannot
|
||||
be moved onto another model without editing it, and moving it is the point. It also grows the core
|
||||
by one band per vendor, when the bands are the same shape.
|
||||
|
||||
2. **One provision, and refuse to hold any refresh token — re-seal only.** Make every credential
|
||||
purely sealed per-holder, including refreshable ones; let each holder refresh its own grant.
|
||||
**Rejected.** It throws away the hard half [ADR 0024](0024-model-access-is-a-provision.md) says
|
||||
already works — the lease, the single-refresher, the switch-on-exhaustion — and replaces it with N
|
||||
nodes each holding a refresh token, which is the very thing today's delivery strips on the stated
|
||||
ground that *a node never holds a refresh token*. A refresh token is the long-lived secret; spraying
|
||||
it across every holder is strictly worse than keeping one copy on one node.
|
||||
|
||||
3. **One provision, and abandon central rotation entirely** for refreshable vendors — treat the grant
|
||||
as opaque and let it expire. **Rejected.** For a subscription-seat vendor an expired access token is
|
||||
a dead licence; without rotation the feature that works today stops working. This is option 2's cost
|
||||
without option 2's autonomy.
|
||||
|
||||
4. **One vendor-blind provision, plus a per-vendor adapter, with a bounded carve-out for the
|
||||
refreshable case.** **Adopted**, below.
|
||||
|
||||
## Decision
|
||||
|
||||
**`model-access` is one consumer-facing, vendor-blind provision.** A consumer declares
|
||||
`requires: model-access`, and is coupled to *reaching a model* — a base URL, a model name, a key —
|
||||
and not to which vendor answers. That is the coupling the name is drawn at
|
||||
([ADR 0040](0040-what-a-module-is.md)'s rule: name the interface at the widest boundary across which
|
||||
the consumer does not care which implementation serves it). Where a consumer were genuinely coupled to
|
||||
a specific wire API it could not swap across, the same rule would split the name — but the consumers
|
||||
that exist reach their model through a CLI or SDK that hides the vendor, so `model-access` is the true
|
||||
coupling and stays one name. This extends [ADR 0024](0024-model-access-is-a-provision.md) and
|
||||
[ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) without changing them.
|
||||
|
||||
**A vendor is an adapter, keyed by the licence's `vendor` field.** The lifecycle a licence needs is
|
||||
vendor-specific and lives in a per-vendor adapter selected by `licence.vendor`, exactly as
|
||||
`public-dns` is one neutral interface answered by registrar-scoped providers —
|
||||
`cloudflare-dns`, `route53-dns` ([ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md)).
|
||||
A consumer names `model-access` and never a vendor, the same way a module names `public-dns` and never
|
||||
a registrar.
|
||||
|
||||
**The field is named `vendor`, not `provider`.** The inventory already uses "provider" for the
|
||||
provider-pin — *which node answers a brokered provision*. Reusing it for *which company sells this
|
||||
licence* would collide two unrelated facts on one word. `vendor` is the licence's, and is separate.
|
||||
|
||||
### The adapter's capabilities, all but one optional
|
||||
|
||||
An adapter declares:
|
||||
|
||||
- **`shape`** — `static-key` or `refreshable-grant`. This is the switch the carve-out below turns on.
|
||||
- **`accept(value) → sealed`** — take an operator-supplied credential and seal it to the holders, the
|
||||
`accept` verb [ADR 0024](0024-model-access-is-a-provision.md) already defines.
|
||||
- **`refresh(licence)`** — refreshable-grant only: the lease / rotate / publish machinery.
|
||||
- **`identity(credential) → account-id`** — the mis-binding guard, for a vendor whose credential
|
||||
carries an identity worth checking.
|
||||
- **`usage(licence) → normalised rows`** — the vendor's usage reading, mapped to the common shape below.
|
||||
- **`deliver`** — the credential *value* only; the destination path is the consumer's, not the
|
||||
adapter's.
|
||||
|
||||
**A static-key vendor implements almost nothing** — `shape: static-key`, `accept` is the generic
|
||||
seal, `deliver` is the value, and `refresh`, `identity` and `usage` are absent or trivial. The
|
||||
abstraction earns its keep by making the common vendor small, not the rare one clever.
|
||||
|
||||
### The carve-out — the one place the guarantee is relaxed, said plainly
|
||||
|
||||
The mesh's standing principle is that it cannot read what it stores: `accept` seals to the holders and
|
||||
discards the plaintext ([ADR 0024](0024-model-access-is-a-provision.md)), and a provider seals nothing
|
||||
because the credential travels the mesh's own asymmetric channel
|
||||
([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)). A `refreshable-grant`
|
||||
credential cannot honour that principle and be centrally rotated at the same time, and central rotation
|
||||
is the working half [ADR 0024](0024-model-access-is-a-provision.md) is explicit about keeping.
|
||||
|
||||
**So, for `refreshable-grant` vendors only:**
|
||||
|
||||
- the **manager node holds the refresh token encrypted at rest** — readable by that node, because
|
||||
rotation requires it. This is the bounded exception.
|
||||
- **access tokens are still sealed per-holder**, as every credential is; a holder reads its own and no
|
||||
other node reads it.
|
||||
- the **refresh token is stripped on delivery** — it never reaches a consuming node. *A node never
|
||||
holds a refresh token* stays true for every node but the one manager.
|
||||
|
||||
**Static-key vendors keep the full guarantee.** There is no token to rotate, so there is nothing to
|
||||
hold readably, so `accept` discards the plaintext and the carve-out never fires. The majority of
|
||||
vendors are static-key, and the majority therefore lose nothing.
|
||||
|
||||
The exception is stated rather than hidden because a relaxed guarantee that is not written down is
|
||||
indistinguishable from a broken one. It is bounded on three axes at once: **refreshable-grant vendors
|
||||
only, the refresh token only, the manager node only.**
|
||||
|
||||
### The settled details this record also fixes
|
||||
|
||||
- **Usage is normalised to `(licence, consumer, period, metric, value)` plus the raw response as
|
||||
jsonb.** The metric is vendor-defined — Anthropic's `utilization%` is one metric, a token count is
|
||||
another — and no common unit is forced across vendors. The raw response is kept so a reading can be
|
||||
re-derived if the normalisation is later found wrong.
|
||||
- **Binding is explicit per consumer, and an unchosen consumer is refused — no implicit fallback.**
|
||||
This is the direction the resolver already takes, and it is the safe one: a mesh with several ways
|
||||
to reach a model refuses a consumer that has not said which, naming the candidates and the command,
|
||||
rather than silently choosing one ([ADR 0024](0024-model-access-is-a-provision.md), and
|
||||
[`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md)).
|
||||
- **Subscription-seat authentication lives entirely inside the adapter**, never in the generic core.
|
||||
So does an interactive `/login` — an adapter-specific "adopt" origin for a credential a person must
|
||||
produce in a browser; the generic `licence key <name>` covers the static-key case.
|
||||
- **Anthropic is the first `refreshable-grant` adapter**, carrying the OAuth refresh, the usage
|
||||
reading, the identity guard and access-token-only delivery. **`anthropic-api-key` is a
|
||||
`static-key` adapter for the same vendor's plain API keys**, and is the early second case that
|
||||
proves the abstraction is not a single vendor wearing a coat: it exercises the whole path with the
|
||||
carve-out switched off.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The carve-out is the mesh's one deliberate relaxation of "it cannot read what it stores."** It is
|
||||
bounded to refreshable-grant vendors, to the refresh token, and to the manager node; the static-key
|
||||
majority keep the full guarantee unchanged. This is the open risk the analysis carried here, and it
|
||||
is recorded as an exception rather than pretended away.
|
||||
- **Anthropic collapses from special case to adapter.** The three binding columns
|
||||
(`nodes.node_license`, `nodes.hal_claude_account`, `agents.claude_account`) become three ordinary
|
||||
consumers of `model-access`; the `claude_*` schema becomes the generic licence tables plus one
|
||||
adapter. What was hardcoded becomes data keyed by `vendor`.
|
||||
- **Adding a vendor is adding an adapter, and a static-key vendor is nearly free.** The modules that
|
||||
want a model do not change when a vendor is added — they named `model-access`, not a vendor.
|
||||
- **The refresh-token concentration is now a stated property to defend, not an accident.** The manager
|
||||
node is a place a long-lived secret lives readably, and losing it or compromising it is a bounded,
|
||||
named blast radius rather than a surprise.
|
||||
|
||||
### How each claim here is checked
|
||||
|
||||
- **Vendor-blind provision, static-key path.** A lab scenario binds an `anthropic-api-key` licence to
|
||||
a consumer; the consumer resolves, receives a key sealed to its node, and reaches a model — and the
|
||||
key is **nowhere in the control plane's database** nor in anything that crossed the broker. This is
|
||||
the `licence_holder` sealed-per-node check that [`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md)
|
||||
already runs, now asserted for a second vendor.
|
||||
- **The carve-out is exactly as narrow as stated.** For a `refreshable-grant` licence, a test asserts
|
||||
the refresh token exists (encrypted) **only on the manager node**, is **absent from every holder's
|
||||
delivery**, and that the delivered credential is access-token-only — and that for a `static-key`
|
||||
licence no refresh token is stored anywhere.
|
||||
- **Adapter selection is keyed by `vendor`.** A scenario with two vendors on two licences verifies each
|
||||
licence's lifecycle runs its own adapter, and that a consumer naming `model-access` never names a
|
||||
vendor to get one.
|
||||
- **Refuse-if-unchosen.** Already checked in [`14-model-access`](../03-DESIGN/01-to-be/14-model-access.md):
|
||||
a consumer with more than one candidate is refused with the candidates and the command named.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0024](0024-model-access-is-a-provision.md) — model access is a provision, a licence is a named
|
||||
thing, and `accept`; this record generalises its single vendor and keeps its working central
|
||||
rotation.
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — a provision names the
|
||||
coupling; `model-access` is drawn at the consumer's.
|
||||
- [ADR 0040](0040-what-a-module-is.md) — the naming rule and the neutral-interface / scoped-provider
|
||||
shape a vendor adapter follows.
|
||||
- [ADR 0044](0044-a-public-name-is-provisioned-like-any-capability.md) — registrar-scoped `public-dns`
|
||||
providers, the precedent a `vendor`-scoped adapter mirrors.
|
||||
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) — the mesh seals credentials
|
||||
and holds no readable copy; the carve-out here is the bounded, named exception to that for a
|
||||
refreshable grant.
|
||||
- [`03-DESIGN/01-to-be/14-model-access.md`](../03-DESIGN/01-to-be/14-model-access.md) — the design this
|
||||
record extends, amended to describe the adapter generalisation.
|
||||
- The read-only vendor-agnostic analysis, 2026-09-05 (code workspace) — the inventory and the decisions
|
||||
taken on the open questions this record encodes.
|
||||
|
||||
> **The mechanism changed — 2026-10-02, by [ADR 0183](0183-the-anthropic-licence-manager-is-a-module-and-hands-tokens-to-the-agent-over-the-bus.md).**
|
||||
> What stands: `model-access` is one vendor-blind provision for the consumers that do not care which
|
||||
> vendor answers; a vendor's lifecycle is an adapter's; the carve-out that one node holds a refreshable
|
||||
> grant's refresh token readably. What moved: the Anthropic adapter is no longer a part of the
|
||||
> controller's licences context but a module, `claude-licence-manager`, holding the seat
|
||||
> `anthropic-licence-manager`, with the grants in its own store encrypted with a key the vault made for
|
||||
> it; and the agent at a terminal is not a consumer of `model-access` — it is coupled to an Anthropic
|
||||
> grant and uses the seat. The consequence above that the three binding columns *become three ordinary
|
||||
> consumers of `model-access`* therefore no longer describes the agent's bindings; they are the
|
||||
> manager's. The static-key adapters and the vendor-blind records stay where this record put them.
|
||||
@@ -1,166 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md
|
||||
---
|
||||
|
||||
# 51. Shared data is the operator's, and a module is granted access to it
|
||||
|
||||
## Context
|
||||
|
||||
**Eight modules declared one filesystem as eight private ones.** The media stack — a library
|
||||
server, the acquisition managers for films, series, music and books, a subtitle fetcher and two
|
||||
download clients — shares directories on one machine: the download clients write into
|
||||
`/services/media/downloads` and the managers read it; the managers write into the libraries and
|
||||
the library server reads them. That sharing is the entire point of the stack. Yet each module
|
||||
declared every shared directory it touched as its own `directory` resource, with an owner and a
|
||||
mode. `/services/media/downloads` was written seven times, as seven private directories that
|
||||
happen to be the same path.
|
||||
|
||||
**The resolver refuses exactly that, and is right to.** Two modules declaring one path on one node
|
||||
are refused by name, with no exemption for identical content and no merge — because two owners of
|
||||
one path is the class of fault this repository keeps recording ([04-ISSUES/036](../04-ISSUES/036-six-modules-own-what-they-must-share/00-report.md)).
|
||||
So the stack as written refuses its own only sensible assignment: all of it on one machine,
|
||||
sharing one filesystem. It passed today only because no test co-resolves any two of the eight. The
|
||||
first machine assigned two of them together is where the refusal would have surfaced.
|
||||
|
||||
**The vocabulary had one word for two intentions, and this was already seen.**
|
||||
[04-ISSUES/026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md) found the
|
||||
same gap from the other side and named it precisely: two kinds of mount are spelled identically —
|
||||
*the directory my data lives in*, which the mesh creates and owns, and *a facility I was granted*,
|
||||
which already exists and the mesh only reaches. That issue deferred inventing a field to tell them
|
||||
apart, because doing so is a design decision and it declined to make one to get a check green. This
|
||||
is that decision.
|
||||
|
||||
**A `directory` resource is owned, on every axis.** The host creates it, sets its owner and mode,
|
||||
and removes it when it is empty and no longer declared — it *removes what it made and leaves what
|
||||
it merely configured* ([ADR 0005](0005-the-node-host.md), [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)).
|
||||
A shared media library is none of that. It existed before the mesh, several modules read and write
|
||||
it at once, and losing it is the one failure that does not recover. It is not any module's
|
||||
resource; it is the operator's, and a module only needs to be let at it.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Make the stack one module with several containers**, the way the mail system already is. The
|
||||
issue raises it directly: is a set of modules that must share a filesystem really one module?
|
||||
**Rejected.** The eight are independently assignable and independently useful — a person may run
|
||||
the download client without the library server, or the film manager without the music one — and
|
||||
folding them into a single module to express a shared directory would make *what a module is*
|
||||
turn on an incidental filesystem contract. It also does not generalise: the next pipeline of
|
||||
modules handing files to each other on one machine (an ingest folder, a spool, a drop directory)
|
||||
would face the same wall and the same wrong remedy.
|
||||
|
||||
2. **One module owns the directories and the rest `require` them.** **Rejected**, and this is the
|
||||
heart of the decision. Nobody owns shared operator data. The library predates the mesh and
|
||||
outlives any one module, so making the library server or a manager its owner means unassigning
|
||||
that module orphans everyone else's access — and the owner would set the owner and mode of a
|
||||
tree it did not create. Ownership is the wrong relationship to model, because the true owner is
|
||||
not a module at all.
|
||||
|
||||
3. **A flag on a directory resource** — `external: true`, or an owner of `operator`. **Rejected.**
|
||||
It overloads one shape with a boolean that inverts every one of its semantics: created becomes
|
||||
*must already exist*, owned becomes *touch nothing*, removed-when-empty becomes *never removed*.
|
||||
That is the *two-kinds-spelled-identically* trap of 04-ISSUES/026 reintroduced with a single
|
||||
quiet field — a reviewer reading `type: directory` would have to check one boolean elsewhere to
|
||||
know whether the mesh owns the thing at all.
|
||||
|
||||
4. **A distinct `accesses` declaration, separate from resources.** **Adopted.**
|
||||
|
||||
## Decision
|
||||
|
||||
**Shared, pre-existing data is operator-owned and external. The mesh does not create it, does not
|
||||
set its owner or mode, does not reconcile it and does not remove it.** A media library, a download
|
||||
spool, an ingest directory is the operator's, and the mesh is a guest in it.
|
||||
|
||||
**A module declares that it needs *access* to such a path, not that it owns a resource there.** The
|
||||
manifest field is `accesses`: a list of `{path, mode}`, where mode is `read` or `read-write` and
|
||||
absent narrows to `read` — the safe default, because the danger with an access is being given more
|
||||
than was meant, not less. A module's own configuration and state directories stay owned
|
||||
`directory` resources; only the shared, pre-existing paths become accesses.
|
||||
|
||||
**The host mounts an accessed path and owns nothing about it.** It reaches the machine as a new
|
||||
declaration shape, `access`, distinct from `directory`. The host confirms the path is present and
|
||||
does nothing else — no create, no chown, no mode, no removal.
|
||||
|
||||
**An accessed path absent at apply time is refused, clearly, not created.** The mesh does not own
|
||||
it, so conjuring it would be a lie the host then acts on — and specifically the lie 04-ISSUES/026
|
||||
records, where a bind mount whose source does not exist is made by the container runtime as root
|
||||
with the wrong ownership. The host says the operator must provide the path instead.
|
||||
|
||||
**Several modules accessing one path is normal, and never refused.** The duplicate-path refusal is
|
||||
about *ownership*, not *use*: it applies to resources a module owns and to those alone. An access
|
||||
is not a resource and never enters the check, so the eight-module stack co-resolves. What stays
|
||||
refused is genuine rivalry — two modules owning one path — and the new contradiction it exposes: a
|
||||
path one module owns while another merely accesses it, because that asserts both that the mesh owns
|
||||
the directory and that the operator does.
|
||||
|
||||
This is a decision and not a patch because it settles *what a module may say about a path it did
|
||||
not make*, which every co-located file-handoff in the catalogue now and later depends on — and
|
||||
because it draws the ownership line [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)
|
||||
started: the host owns what it made and keeps what it merely configured, and this adds the third
|
||||
case it did not have a word for — what it neither made nor configured, and must not touch.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **The stack co-resolves.** A control-plane unit test assigns two modules that declare access to
|
||||
one path on one node and asserts no refusal — the exact case the resolver refuses when the same
|
||||
path is owned. The mirror test, two modules *owning* one path, still refuses, so the sharing
|
||||
vocabulary does not weaken the rule it sits beside.
|
||||
- **Ownership and access cannot both be claimed of one path.** A unit test asserts the resolver
|
||||
refuses a path one module owns and another accesses, naming both.
|
||||
- **Absent is refused, not created.** A host unit test applies an access to a path that does not
|
||||
exist and asserts a clear refusal that names the operator, and that nothing was created.
|
||||
- **Present is confirmed and nothing moves.** A host unit test applies an access to an existing
|
||||
directory and asserts the apply reports no change and disturbs nothing.
|
||||
- **Undeclaring never removes.** A host unit test drops a previously declared access and asserts
|
||||
the operator's directory and its contents are left exactly as they were — the data-loss failure
|
||||
[ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md) exists to prevent, on a directory the
|
||||
mesh never made.
|
||||
- **Every media manifest is corrected.** No `/services/media/*` path is an owned `directory`
|
||||
resource in any of the eight; each is an `accesses` entry, and each module's own config and state
|
||||
directories remain owned. Checked by the control-plane manifest parser, which now understands
|
||||
`accesses` and refuses a malformed one.
|
||||
|
||||
## Consequences
|
||||
|
||||
**A shared filesystem between co-located modules now has a vocabulary**, and it is not the media
|
||||
stack's alone: any pipeline handing files to a neighbour on one machine — an ingest directory, a
|
||||
spool, a drop folder — says *I access this operator path* rather than *I own this directory*, and
|
||||
several of them may say it of one path.
|
||||
|
||||
**Unassigning a module that reached shared data leaves the data.** Correct, and the same trade
|
||||
[ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md) made for owned directories: removing
|
||||
data is a person's act, done knowingly, not a side effect of unassignment.
|
||||
|
||||
**The operator must provision the shared paths before the stack is applied**, and a machine that
|
||||
lacks one is told plainly which. That is a real new obligation, and it is the right one: the mesh
|
||||
cannot own what predates it, so it cannot create it either, and saying so at apply time beats a
|
||||
directory conjured as root and a service that half-works.
|
||||
|
||||
**The host vocabulary grew by one shape**, which is a cost — every added shape widens what a
|
||||
compromised control plane can express ([ADR 0005](0005-the-node-host.md)). It is a narrow one: an
|
||||
`access` is confirmed by a stat and grants the host no new action. It earns its place by letting
|
||||
the host refuse to create what it must not own, which no existing shape could say.
|
||||
|
||||
**A path can be both owned and accessed only by refusal.** If a future manifest declares one path
|
||||
as an owned directory in one module and an access in another, the resolver refuses it rather than
|
||||
guessing which is meant — the two assertions about who owns the data cannot both hold.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/036](../04-ISSUES/036-six-modules-own-what-they-must-share/00-report.md) — six (in
|
||||
fact eight) modules own what they must share; the problem this resolves
|
||||
- [04-ISSUES/026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md) — the two
|
||||
kinds of mount spelled identically, which deferred this field to a decision
|
||||
- [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md) — data outlives the mesh; the host
|
||||
keeps what it did not make. This record adds the case it had no word for
|
||||
- [ADR 0005](0005-the-node-host.md) — the host removes what it made and leaves what it merely
|
||||
configured; the vocabulary is finite and every shape is a security decision
|
||||
- [ADR 0040](0040-what-a-module-is.md) — what a module is; an access is a new thing a module may
|
||||
say about the machine it lands on
|
||||
- mesh-control `feat/shared-data-access`, mesh-catalog `feat/media-access-not-ownership`,
|
||||
mesh-host `feat/mount-operator-owned` — the mechanism, the corrected manifests, and the host
|
||||
shape
|
||||
@@ -1,198 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-05
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0005-the-node-host.md
|
||||
---
|
||||
|
||||
# 52. An init step is a container run once to completion, gating what follows
|
||||
|
||||
## Context
|
||||
|
||||
**A module can declare things that exist; it cannot declare a step that runs.** The host owns a
|
||||
finite vocabulary of shapes — `file`, `directory`, `service`, `package`, `container`, `action` —
|
||||
and every one but `action` describes *state*: a thing that should be present, with content or a
|
||||
mode or an image, which the host reconciles toward ([ADR 0005](0005-the-node-host.md)). That is
|
||||
right for what it covers. But a real class of modules needs, once, to *run their own code at a
|
||||
point in their own lifecycle* — and the vocabulary has no word for it
|
||||
([04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md)).
|
||||
|
||||
**mosquitto is the sharp case, and it fails silently without this.** Its Dynamic Security plugin
|
||||
loads at broker start and refuses to come up unless `dynamic-security.json` already holds an admin
|
||||
client. Seeding that file is a step that must happen *after* the data directory exists and *before*
|
||||
the broker container starts. The manifest can declare the directory, the config file and the broker
|
||||
container; it cannot declare "seed this, once, before that container starts." Written as it is
|
||||
today, the broker starts against an unseeded store and the plugin aborts — and the next reconcile
|
||||
does not fix it, because nothing in the declaration ever seeds the file.
|
||||
|
||||
**It is not one module's defect.** The database providers need the same to run a first-boot
|
||||
migration, an extension enable, or a health gate before they are announced ready; today that works
|
||||
only where the *image* happens to seed itself from an environment variable, and anything the mesh
|
||||
must run once against the server has no home. This is the timing face of the same gap
|
||||
[04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md) records
|
||||
from the content side: the manifest needed *the file to exist before first start*, and had only
|
||||
*the file has this content, forever*.
|
||||
|
||||
**The obvious answer is the one that already went wrong.** An earlier mesh had exactly this as a
|
||||
feature — event-driven hooks that ran custom code at phases of build, publish and deploy. It was
|
||||
powerful and it was *complex to set up and flaky*, and that fragility, not the need, is the content
|
||||
of the issue. Whatever this becomes must not rebuild that engine.
|
||||
|
||||
**The ground has shifted since that engine, in a way that makes a much smaller answer possible.** A
|
||||
module with tools or events now runs a **process of its own** — a container carrying the module's
|
||||
compiled code, holding the single broker account the mesh scoped to it, isolated from every other
|
||||
module ([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The
|
||||
code that must run at first boot is *already in that image, under that account*. So the mesh does
|
||||
not need a way to run a module's code — it has one. It needs a way to say **run this container to
|
||||
completion, and start the one that depends on it only after it has.**
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A host `run` shape — a command the host executes on the machine.** The direct reading of
|
||||
"run code at a lifecycle phase." **Rejected.** The host's vocabulary is finite and every added
|
||||
shape is a security decision, because it widens what a *compromised control plane* can express
|
||||
([ADR 0005](0005-the-node-host.md)). A general "run this command as the host" is the largest
|
||||
such widening there is: the blast radius is the whole machine, as root. The mesh already drew
|
||||
this exact line for `action` — it runs a command, and so it is *permitted from the bundle and
|
||||
refused from the link*, because the bundle arrives with the binary and the link is a separate
|
||||
party with an unbounded reach. A new host-command shape usable by an ordinary module would be an
|
||||
`action` from the link by another name, which is precisely what is refused.
|
||||
|
||||
2. **Per-phase lifecycle hooks on a module** — `pre-start`, `post-start`, `pre-remove`, and their
|
||||
build/publish cousins, each naming code the mesh runs at that phase. The general answer, and the
|
||||
old feature. **Rejected for now.** It is the flaky engine the issue warns against, and most of
|
||||
its phases have no present need. Deciding the full set of phases, where each one's code runs, and
|
||||
how each is made idempotent is a large design taken to buy capability nothing yet asks for. The
|
||||
three blocked modules all need one phase — *before a container starts* — and a mechanism narrow
|
||||
enough to be obviously correct beats a general one that is not.
|
||||
|
||||
3. **A distinct one-shot resource type** — a new shape, sibling to `container`, that names an image
|
||||
and runs it once. **Rejected.** It grows the host vocabulary by a whole shape (a `Type`, a
|
||||
struct, an applier, a place in every host's shape list) to express something a `container` almost
|
||||
already is. A one-shot *is* a container — a pinned image, an account, volumes, an environment —
|
||||
that happens to exit. Spending a new shape on the difference is the cost of option 1 in smaller
|
||||
type, for a capability the existing shape can carry with one modifier.
|
||||
|
||||
4. **A modifier on the existing `container` shape: this container runs once, to completion, and the
|
||||
host gates the apply on it.** **Adopted.** It reuses the shape the host already has, adds no new
|
||||
host action, and leans on two guarantees the host already gives — *apply in declared order,
|
||||
never sorted*, and *a failed step fails the apply* — to turn "before that container starts" into
|
||||
an emergent property of ordering rather than a dependency graph the host must resolve.
|
||||
|
||||
## Decision
|
||||
|
||||
**A run-once step is an ordinary `container`, marked to run to completion.** The manifest sets
|
||||
`run-once: true` on a container resource. Everything else about it is a container as before — a
|
||||
digest-pinned image ([ADR 0006](0006-the-substrate-and-the-control-plane.md)), volumes, an
|
||||
environment, and for a module's own code the same scoped account its runtime already holds
|
||||
([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The mesh adds
|
||||
no way to run code; it marks a container the host must **run to completion and require to exit 0**,
|
||||
rather than start and leave running.
|
||||
|
||||
**The gate is declaration order, not a named dependency.** The host applies a declaration in the
|
||||
order it is given, does not sort, and does not resolve dependencies — ordering is a decision, and it
|
||||
is the control plane's ([ADR 0005](0005-the-node-host.md),
|
||||
[04-ISSUES/013](../04-ISSUES/013-a-file-arrives-after-the-service-that-needs-it/00-report.md)). A
|
||||
run-once step is placed *before* the container that depends on it, and **a failed run-once step
|
||||
halts the apply**, exactly as a failed `action` does — so everything the declaration places after
|
||||
it, the broker included, is never reached until the step has completed. "Before the broker starts"
|
||||
is therefore expressed by list position plus completion, and the host cross-references nothing.
|
||||
|
||||
**Completion is recorded, and a re-apply does not re-run it.** The host records what it applied only
|
||||
after the fact, as the digest of the declaration that produced it
|
||||
([ADR 0018](0018-a-picture-is-read-from-what-runs.md)) — a run-once step no differently. Because the
|
||||
step leaves nothing running to inspect, that persisted digest, not a live container, is the marker
|
||||
that it happened. On a later apply the host finds the digest already recorded for this exact
|
||||
declaration and does nothing; it re-runs only when the declaration's digest has changed, and a step
|
||||
that exited non-zero recorded nothing and so is retried next apply. This is the reconcilable,
|
||||
idempotent discipline the state shapes get for free, made explicit for a step.
|
||||
|
||||
This is a decision and not a patch because it settles **what a module may say about running its own
|
||||
code**, which the whole catalogue of providers — a seed before start, a first-boot migration, a
|
||||
health gate — now and later depends on, and because it draws the line the issue asked for: the
|
||||
narrowest sound mechanism that unblocks the three modules without rebuilding the hook engine whose
|
||||
fragility is the warning.
|
||||
|
||||
### The security bound, stated plainly
|
||||
|
||||
**The host gains no new action and no new shape.** `run-once` is a boolean modifier on the
|
||||
`container` shape that already exists. A run-once container is strictly *less* powerful than an
|
||||
`action`: it cannot run an arbitrary host command, only a digest-pinned image under an account the
|
||||
mesh scoped — which is exactly the capability `container` already grants from the link. A control
|
||||
plane that is compromised can express nothing through `run-once` it could not already express by
|
||||
declaring an ordinary `container`. The dangerous expansion of option 1 — a command the host runs on
|
||||
the machine — is not made.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A run-once step runs to completion and its exit 0 is required.** A host unit test applies a
|
||||
run-once container whose image exits 0, asserts the host ran it to completion (not detached, not
|
||||
left running) and reported it done; a sibling test applies one that exits non-zero and asserts
|
||||
the apply fails, naming the step.
|
||||
- **A failed run-once step gates what follows.** A host unit test places a run-once container that
|
||||
exits non-zero before another container and asserts the second is never started and the apply is
|
||||
reported gated — the mirror of the existing test that a failed action stops what follows.
|
||||
- **It is not re-run once it has completed.** A host unit test applies a run-once step, then applies
|
||||
the identical declaration again with the first run's record present, and asserts the second apply
|
||||
runs nothing and reports the step unchanged.
|
||||
- **A changed declaration re-runs it.** A host unit test applies a run-once step, then applies one
|
||||
whose image or environment differs, and asserts it runs again — the digest moved, so the marker no
|
||||
longer matches.
|
||||
- **The vocabulary carries the field end to end.** A control-plane unit test resolves a module whose
|
||||
manifest marks a container `run-once` and asserts the rendered host declaration carries the field,
|
||||
in author order before the container it gates; the manifest parser refuses a `run-once` that is
|
||||
not boolean.
|
||||
- **mosquitto seeds before the broker.** mosquitto's manifest declares a run-once init container,
|
||||
before the `server` (broker) container, that writes the admin client into `dynamic-security.json`
|
||||
and exits — checked by the control-plane resolver, which now understands the field, and by the
|
||||
ordering of the rendered declaration. The end-to-end proof that it runs exactly once, at the right
|
||||
phase, and converges on re-apply is owed to a lab scenario ([04-ISSUES/037] open question), which
|
||||
this record does not close.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The three blocked modules gain a home for their step.** mosquitto seeds its dynsec admin before
|
||||
the broker; a provider that must migrate or health-gate at first boot declares a run-once step in
|
||||
its own runtime image, under its own account, before the container that depends on it.
|
||||
- **The general lifecycle hook is deferred, deliberately.** Only *before a container starts* is
|
||||
bought here. `post-start`, `pre-remove` and the build/publish phases remain unbuilt, and the day
|
||||
one is genuinely needed it is decided then, against a need, not speculatively — the same restraint
|
||||
that kept this from being the old engine.
|
||||
- **The seed-then-mutate file is safe if the step is written to be.** A run-once seed writes
|
||||
`dynamic-security.json` only when it is absent and never reconciles it, so what the running plugin
|
||||
grows in that file afterward is never wiped
|
||||
([04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)).
|
||||
The host's marker guarantees the step is not re-run; the step's own code guarantees it does not
|
||||
clobber on the pass it does run.
|
||||
- **The host vocabulary did not grow, and that is the point.** The cost of a run-once step is one
|
||||
boolean and a completion path in the container applier, not a new shape and not a new action. The
|
||||
mesh expresses ordering and completion; the module runs its own code, where it already runs it.
|
||||
- **A run-once step that never converges is a stuck apply, loudly.** A step that exits non-zero
|
||||
every time halts the apply every time, and the container it gates never starts — which is the
|
||||
correct failure, reported, rather than a broker that half-starts against an unseeded store and a
|
||||
reconcile that reports success. It is failed forward, not failed silent.
|
||||
|
||||
## References
|
||||
|
||||
- [04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md) — a
|
||||
module cannot run its own code at a lifecycle phase; the gap this resolves, and the warning about
|
||||
the old hook engine
|
||||
- [04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md) — the
|
||||
content face of the same gap: a file needed before first start, that the running program then
|
||||
mutates
|
||||
- [04-ISSUES/013](../04-ISSUES/013-a-file-arrives-after-the-service-that-needs-it/00-report.md) — the
|
||||
order of a declaration is the control plane's, and the host applies it as given; the gate rests on
|
||||
this
|
||||
- [ADR 0005](0005-the-node-host.md) — the host's finite vocabulary, the ordered declaration it does
|
||||
not sort, and `action` as the shape a command already is and why it is refused from the link
|
||||
- [ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md) — a module runs
|
||||
its code as its own process under its own account; a run-once step is that process, run to
|
||||
completion
|
||||
- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — what was applied is recorded after it works;
|
||||
the completion marker is that record
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — a container is pinned by digest; a
|
||||
run-once container no differently
|
||||
- mesh-control `feat/lifecycle-run-once`, mesh-host `feat/apply-run-once`, mesh-catalog
|
||||
`feat/mosquitto-bootstrap` — the vocabulary, the apply support, and mosquitto's seeded broker
|
||||
@@ -1,181 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-06
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md
|
||||
---
|
||||
|
||||
# 53. A scheduled step is a container run on a recurring schedule
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0052](0052-a-step-that-runs-once-before-a-container.md) gave the mesh a step that runs *once*;
|
||||
a real class of modules needs one that runs *again and again*.** kometa reconciles a media library
|
||||
against its lists on a timer; a ticketing integration polls its source for new work every few minutes;
|
||||
a backup, a cache warm, a metrics roll-up all recur. The host's vocabulary describes *state* — a file,
|
||||
a directory, a container that should be running — and 0052 added *a step that happens once and is
|
||||
done*. Neither says *this should happen every night at 3, forever*
|
||||
([04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md) named
|
||||
the lifecycle gap; 0052 closed the once-before-start face of it and left the recurring face open).
|
||||
|
||||
**The modules that need it already run their own code.** As with run-once, the ground has shifted
|
||||
since the old mesh's flaky hooks: a module with tools or events runs a **process of its own** — a
|
||||
container carrying the module's compiled code under the single scoped account the mesh gave it
|
||||
([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The work that
|
||||
must recur is already in that image, under that account. The mesh does not need a way to run a
|
||||
module's code on a timer — it has the code and the account. It needs a way to say **run this
|
||||
container again on this cadence**.
|
||||
|
||||
**The shape of the answer is already decided, one modifier over.** 0052 rejected a host `run` command,
|
||||
per-phase hooks, and a distinct one-shot resource type, and adopted *a modifier on the `container`
|
||||
shape the host already has*, because a one-shot is a container that happens to exit. A scheduled step
|
||||
is the same container that happens to exit — run repeatedly. Reusing that shape keeps the host
|
||||
vocabulary flat and inherits 0052's security bound whole. The only thing 0052's `run-once` does not
|
||||
carry is *when to run it again*.
|
||||
|
||||
**One difference from run-once changes a rule, and it is the reason this is its own record.** A
|
||||
run-once step **gates the apply**: it is placed before the container that depends on it, and a failure
|
||||
halts everything after it, because "seed the store before the broker starts" is a correctness
|
||||
precondition ([ADR 0052](0052-a-step-that-runs-once-before-a-container.md)). A scheduled step is the
|
||||
opposite: it runs *after* the machine is up and converged, on its own clock, and a single failed run
|
||||
is an ordinary operational event — the next run comes anyway. A scheduled step that halted the apply,
|
||||
or that a failed run marked the node not-current over, would make a routine poll into a reason the
|
||||
whole machine reads as broken. So the gating rule 0052 established is exactly the rule this record
|
||||
must **not** inherit.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A host `cron`/`timer` shape — the host installs a system timer that runs a command.** The direct
|
||||
reading. **Rejected**, for the reason 0052 rejected a host `run` shape: it widens what a
|
||||
*compromised control plane* can express toward "run this command on the machine, forever," which is
|
||||
the largest widening there is, and it is `action`-from-the-link by another name
|
||||
([ADR 0005](0005-the-node-host.md)). A recurring command is worse than a one-off, because it
|
||||
persists.
|
||||
|
||||
2. **Per-phase lifecycle hooks** — `on-schedule` joining `pre-start`/`post-start` as named code the
|
||||
mesh runs. **Rejected for now**, as in 0052: it is the flaky hook engine the issue warns against,
|
||||
and the three modules that need this need one thing — *run this container on a cadence* — which a
|
||||
narrow modifier expresses without deciding a whole hook vocabulary.
|
||||
|
||||
3. **A distinct `scheduled` resource type**, sibling to `container`. **Rejected**, as 0052 rejected a
|
||||
distinct one-shot type: it spends a whole new host shape (a `Type`, a struct, an applier, a place
|
||||
in every host's shape list) on something a `container` already almost is — a scheduled task *is* a
|
||||
container (pinned image, account, volumes, environment) that runs on a clock.
|
||||
|
||||
4. **A modifier on the existing `container` shape: `schedule`, a cron expression the host runs the
|
||||
container on.** **Adopted.** It reuses the shape the host has, adds no new host action, and sits
|
||||
beside `run-once` as its recurring twin — the same container, exited, run again.
|
||||
|
||||
## Decision
|
||||
|
||||
**A scheduled step is an ordinary `container`, marked with a `schedule`.** The manifest sets
|
||||
`schedule: "<cron>"` on a container resource — a standard five-field cron expression. Everything else
|
||||
about it is a container as before: a digest-pinned image
|
||||
([ADR 0006](0006-the-substrate-and-the-control-plane.md)), volumes, an environment, and for a module's
|
||||
own code the same scoped account its runtime already holds
|
||||
([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). The mesh adds no
|
||||
way to run code; it marks a container the host must **run on that cadence, each time to completion**,
|
||||
rather than start once and leave running (a service) or run once and gate (a run-once step).
|
||||
|
||||
**A run is fired by the clock, not by the apply, and does not gate it.** Applying the declaration
|
||||
installs the schedule; it does not run the step. The machine converges — reports applied and current —
|
||||
as soon as the schedule is installed, exactly as it does for a service that is running. Thereafter the
|
||||
host fires the container when the cron expression is due. This is the deliberate inversion of 0052:
|
||||
a scheduled step is downstream of convergence, not a precondition of it.
|
||||
|
||||
**A failed run is recorded and the next run still comes; it never marks the node not-current.** A run
|
||||
that exits non-zero is logged against the module — visible, auditable — but it does not fail the apply,
|
||||
does not halt other resources, and does not flip the node's reported state. A poll that fails at 03:00
|
||||
and succeeds at 03:05 is the system working, not a machine that needs attention. A step that fails
|
||||
*every* time is a loud, repeating log entry, which is the correct signal for "this recurring job is
|
||||
broken" — distinct from "this machine did not converge."
|
||||
|
||||
**Runs do not stack.** If a run is still going when the next is due, the host skips the due run rather
|
||||
than starting a second copy, and logs the skip. A slow nightly job that occasionally overruns must not
|
||||
spawn a growing pile of concurrent containers competing for the same account and volumes — the failure
|
||||
mode that made the old timers dangerous.
|
||||
|
||||
**Each run is independent and idempotent by the module's own code.** The mesh guarantees only *the
|
||||
container is run on the cadence*; that a run does the right thing when the previous one half-finished
|
||||
is the module's contract, the same discipline a run-once seed owes
|
||||
([ADR 0052](0052-a-step-that-runs-once-before-a-container.md),
|
||||
[04-ISSUES/035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)).
|
||||
|
||||
This is a decision and not a patch because it settles **what a module may say about running its own
|
||||
code on a cadence**, which every recurring provider job — a sync, a poll, a roll-up — now and later
|
||||
depends on, and because it draws the line 0052 left: the recurring twin of run-once, with the gating
|
||||
rule deliberately reversed so a routine job's failure is never a machine's failure.
|
||||
|
||||
### The security bound, stated plainly
|
||||
|
||||
**The host gains no new action and no new shape.** `schedule` is a string modifier on the `container`
|
||||
shape that already exists. A scheduled container is strictly *less* powerful than an `action`: it runs
|
||||
a digest-pinned image under an account the mesh scoped, which is exactly what `container` already
|
||||
grants from the link, and it cannot run an arbitrary host command. A compromised control plane can
|
||||
express nothing through `schedule` it could not already express by declaring a `container` — the cron
|
||||
string only says *how often*, not *what*. The dangerous expansion of option 1 — a command the host
|
||||
runs on the machine on a timer — is not made.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A scheduled step runs when the schedule is due.** A host unit test installs a container with a
|
||||
schedule that is due immediately (or advances a injected clock to when it is due) and asserts the
|
||||
host ran it to completion; a sibling test with a schedule not yet due asserts it has not run.
|
||||
- **Installing it does not run it, and the node is current without a run.** A host unit test applies a
|
||||
scheduled container and asserts the apply reports current *before* any run has fired — the schedule
|
||||
is state that is present, not a step that gated.
|
||||
- **A failed run does not fail the apply or the node.** A host unit test fires a scheduled container
|
||||
that exits non-zero and asserts the failure is recorded against the module, the apply is not failed,
|
||||
and the node stays current — the mirror of the run-once test where a non-zero exit *does* halt.
|
||||
- **Runs do not stack.** A host unit test fires a scheduled container whose run outlasts its next due
|
||||
time and asserts the host skipped the due run and logged the skip, rather than starting a second
|
||||
container.
|
||||
- **The vocabulary carries the field end to end.** A control-plane unit test resolves a module whose
|
||||
manifest sets `schedule` on a container and asserts the rendered host declaration carries the field;
|
||||
the manifest parser refuses a `schedule` that is not a valid cron expression, and refuses a container
|
||||
that is both `run-once` and `schedule` (a step is one or the other, never both).
|
||||
- **A real module recurs in the lab.** A converted module declaring a scheduled step (kometa's library
|
||||
sync, or a poller) is assigned in a lab scenario, and the scenario asserts the scheduled container
|
||||
fires on its cadence and its account and volumes are the module's — the end-to-end proof this record
|
||||
owes, as 0052 owed its run-once lab proof.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The recurring providers gain a home for their cadence.** kometa reconciles on its schedule; a
|
||||
poller polls; a roll-up rolls up — each a scheduled step in its own runtime image, under its own
|
||||
account, on the cron it declares.
|
||||
- **The general lifecycle hook is still deferred.** Only *run once before* (0052) and *run on a
|
||||
cadence* (this) are bought. `post-start`, `pre-remove` and the build/publish phases remain unbuilt,
|
||||
decided when a real need arrives, not speculatively — the restraint that kept both from being the old
|
||||
engine.
|
||||
- **A recurring job's failure is loud but not fatal.** The node stays current while a scheduled step
|
||||
fails and retries; a step that fails forever is a repeating log entry, not a machine marked broken.
|
||||
This is the correct separation — a machine's convergence and a job's success are different questions —
|
||||
and it is why this could not simply be `run-once` without the schedule.
|
||||
- **The host vocabulary did not grow, again, and that is the point.** The cost of a scheduled step is
|
||||
one string field and a cron loop in the container applier, not a new shape and not a new action. The
|
||||
mesh expresses cadence; the module runs its own code, where it already runs it.
|
||||
- **`run-once` and `schedule` are exclusive and complete for now.** A container runs once and gates, or
|
||||
runs on a cadence and does not, or runs and stays up (a service). A manifest that asks for two of
|
||||
these at once is refused, because the three are distinct answers to "how does this container run."
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md) — the run-once step; this is its
|
||||
recurring twin, reusing the `container`-modifier shape and inheriting its security bound, and
|
||||
reversing its gating rule
|
||||
- [04-ISSUES/037](../04-ISSUES/037-a-module-cannot-run-code-at-a-lifecycle-phase/00-report.md) — a
|
||||
module cannot run its own code at a lifecycle phase; 0052 closed the once-before-start face, this
|
||||
closes the recurring face
|
||||
- [ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md) — a module runs
|
||||
its code as its own process under its own account; a scheduled step is that process, run on a cadence
|
||||
- [ADR 0005](0005-the-node-host.md) — the host's finite vocabulary, and why a host command (even on a
|
||||
timer) is refused from the link
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md) — a container is pinned by digest; a
|
||||
scheduled container no differently
|
||||
- [ADR 0018](0018-a-picture-is-read-from-what-runs.md) — what was applied is recorded after it works;
|
||||
the installed schedule is state, a fired run is an event
|
||||
- mesh-control `feat/schedule-container`, mesh-host `feat/apply-schedule`, and the converted module
|
||||
that first declares a scheduled step — the vocabulary, the apply support, and the recurring proof
|
||||
@@ -1,178 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-06
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0050-model-access-is-vendor-agnostic.md
|
||||
---
|
||||
|
||||
# 54. Model usage is a vendor-neutral record, produced by the adapter, at two grains
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0050](0050-model-access-is-vendor-agnostic.md) fixed the *shape* of a usage reading and left
|
||||
its *home* open.** It decided that an adapter may expose `usage(licence) → normalised rows`, that a
|
||||
row is `(licence, consumer, period, metric, value)` plus the raw vendor response as jsonb, and that the
|
||||
metric is vendor-defined (Anthropic's `utilization%` is one metric, a token count another). It did not
|
||||
say where those rows are stored, how they get produced on a cadence, or whether the only grain is the
|
||||
whole licence — and the mesh's first real consumer needs all three answered.
|
||||
|
||||
**The predecessor mesh recorded usage at two grains, and both are wanted.** It polled the vendor for a
|
||||
licence-level reading (an account's `utilization%`), and it also attributed **per-session** token and
|
||||
cost — which model, how many input and output tokens, what it cost — parsed from the agent's own
|
||||
transcript, including the case where a long session switched the account it billed against mid-way. The
|
||||
licence-level reading answers "how close is this subscription to its cap"; the session-level reading
|
||||
answers "what did this piece of work cost, and against which account." A mesh that kept only the first
|
||||
could not bill a project or notice a runaway session; keeping only the second could not see a cap
|
||||
approaching. Both are load-bearing and neither subsumes the other.
|
||||
|
||||
**A session is already a consumer, so the second grain needs no second vocabulary.**
|
||||
[ADR 0026](0026-the-mesh-has-a-session-of-its-own.md) and design 15 establish that an agent session is
|
||||
a thing in its own right, and that **model access is bound to the session, not to the machine** — a
|
||||
session is a *consumer* of `model-access`, identified by `(node, module)` and its own id. That is
|
||||
exactly the `consumer` column ADR 0050 already put in the usage row. So the two grains are not two
|
||||
schemas; they are the same row at two consumer resolutions: the holding module for the licence grain,
|
||||
the session for the finer one. The design's own still-open worker-naming gap (design 14) is the same
|
||||
gap here and is left where it is — a session id distinguishes what `(node, module)` cannot.
|
||||
|
||||
**The pieces this needs already exist.** A periodic reading is a **scheduled step**
|
||||
([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)) — the adapter's `usage()` poll is a container the
|
||||
mesh runs on a cadence, which is precisely what 0053 was built for. A durable audit of what happened is
|
||||
**an event the audit trail records** ([ADR 0041](0041-events-are-a-relationship.md),
|
||||
[ADR 0042](0042-the-shape-of-an-event-on-the-wire.md)); the `audit-logger` module already consumes every
|
||||
event. And a queryable current picture is **a context store**, the same shape the licences themselves
|
||||
live in ([ADR 0008](0008-a-context-owns-its-store.md)). Nothing new in kind is required; what is missing is
|
||||
the decision to point them at usage.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Licence grain only — a single `utilization%` poll, nothing per session.** Rejected: it cannot
|
||||
attribute cost to a piece of work or catch a session that is burning an account down, which is half
|
||||
of why usage is recorded at all.
|
||||
|
||||
2. **A bespoke `sessions` schema mirroring the old mesh's `*_sessions` / `*_session_account_usage`
|
||||
tables.** Rejected: it reintroduces a second, vendor-shaped vocabulary for something the mesh
|
||||
already names — a session is a consumer, and its usage is a usage row. A parallel schema would drift
|
||||
from the `model-access` vocabulary and force every reader to learn two.
|
||||
|
||||
3. **Store usage only as raw vendor blobs, normalise later.** Rejected as the *whole* answer (kept as a
|
||||
fallback within the chosen one): a reader that must parse Anthropic's response shape to answer "what
|
||||
did this cost" has the vendor coupling the whole feature exists to remove. The raw blob is kept
|
||||
beside the normalised row (0050 already requires this), not instead of it.
|
||||
|
||||
4. **One vendor-neutral usage record at two consumer grains, produced by the adapter, recorded as
|
||||
both an event and a queryable row.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**Model usage is one vendor-neutral record — `(licence, consumer, period, metric, value)` plus the raw
|
||||
response — recorded at two grains that differ only in the `consumer`.** At the **licence grain** the
|
||||
consumer is the holding module and the metric is the vendor's own account reading (Anthropic:
|
||||
`utilization%`). At the **session grain** the consumer is the agent session — `(node, module)` and its
|
||||
session id ([ADR 0026](0026-the-mesh-has-a-session-of-its-own.md)) — and the metrics are the ones a
|
||||
session bills: input tokens, output tokens, model, and cost. The row shape is 0050's, unchanged; the
|
||||
grain is which consumer the row is *for*.
|
||||
|
||||
**The adapter is the only thing that knows the vendor, and it produces both grains.** Reading an
|
||||
account's cap is the adapter's `usage(licence)` verb ([ADR 0050](0050-model-access-is-vendor-agnostic.md));
|
||||
attributing a session's cost is the adapter reading that vendor's transcript or usage API and emitting
|
||||
rows keyed to the session. The mesh defines the row and the plumbing; the adapter fills it from whatever
|
||||
the vendor exposes, and a static-key vendor that exposes nothing simply produces no rows — usage is an
|
||||
optional reading, not a requirement of holding a licence.
|
||||
|
||||
**A reading is taken on a schedule, not on a request.** The licence-grain poll is a **scheduled
|
||||
container** ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)) the adapter runs on a cadence; the
|
||||
session-grain rows are produced as sessions progress, from the transcript the session already writes.
|
||||
Neither blocks anything: a poll that fails is a logged, retried scheduled run (0053's rule), and a
|
||||
session whose cost cannot yet be attributed is a row not yet written, never a session refused.
|
||||
|
||||
**Usage is recorded two ways, for two audiences.** Each reading is **emitted as an event**
|
||||
([ADR 0041](0041-events-are-a-relationship.md)) — an immutable "this was observed at this time" that the
|
||||
`audit-logger` already records, so the history of what an account did is in the audit trail by default,
|
||||
under nobody's special arrangement. And the **current** picture — the latest reading per
|
||||
`(licence, consumer, period, metric)` — is upserted into a **usage context store**, so "how close is
|
||||
this cap" and "what has this project spent this month" are a query, not a fold over the event log. The
|
||||
event is the record of what happened; the store is the answer to what is true now.
|
||||
|
||||
**Usage is not a credential, and is recorded in the clear.** The one thing the mesh must not read is the
|
||||
sealed key ([ADR 0050](0050-model-access-is-vendor-agnostic.md), the refresh-token carve-out aside). A
|
||||
token count and a cost are not secrets; they are the operator's own operational facts, recorded openly
|
||||
so they can be queried, audited, and charged against. This is the deliberate opposite of the credential
|
||||
rule, and stating it prevents a later reader assuming usage inherits the key's secrecy and hiding it
|
||||
from the person who is paying.
|
||||
|
||||
This is a decision and not a patch because it settles **where a model's usage lives and at what grain**,
|
||||
which every reader — a bill, a cap alarm, a per-project report — depends on, and because it closes the
|
||||
half [ADR 0050](0050-model-access-is-vendor-agnostic.md) explicitly left open, reusing the session
|
||||
([ADR 0026](0026-the-mesh-has-a-session-of-its-own.md)), the schedule
|
||||
([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)), the event ([ADR 0041](0041-events-are-a-relationship.md))
|
||||
and the store ([ADR 0008](0008-a-context-owns-its-store.md)) the mesh already has rather than inventing a
|
||||
vocabulary beside them.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A usage row is vendor-neutral and carries the raw beside it.** A unit test constructs an Anthropic
|
||||
`utilization%` reading and a token/cost reading and asserts both render to
|
||||
`(licence, consumer, period, metric, value)` with the vendor response preserved in the raw column;
|
||||
a reader that answers "what did this cost" touches only the normalised columns.
|
||||
- **The two grains differ only in the consumer.** A unit test records a licence-grain row (consumer =
|
||||
the module) and a session-grain row (consumer = a session id) for one licence and asserts both are
|
||||
the same shape and both are returned when the licence's usage is asked for, distinguishable by
|
||||
consumer.
|
||||
- **A poll is a scheduled run and its failure is not fatal.** The adapter's usage container declares a
|
||||
`schedule` ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md)); a host test (0053's) already proves a
|
||||
scheduled step runs on cadence, does not gate, and logs rather than fails on a non-zero run — the poll
|
||||
inherits this and adds nothing to check.
|
||||
- **Each reading is an event the audit trail records.** An integration check asserts a usage reading
|
||||
emits an event that the `audit-logger` receives (it consumes `#`), so the history is present without
|
||||
the usage module and the audit module knowing about each other beyond the event.
|
||||
- **The current picture is a query.** A store test upserts two readings for one
|
||||
`(licence, consumer, period, metric)` and asserts the later replaces the earlier, so "what is true
|
||||
now" is one row, while the event log keeps both.
|
||||
- **A static-key vendor with no usage reading records nothing, and that is fine.** A test resolves a
|
||||
`static-key` model-access consumer whose adapter has no `usage` verb and asserts the licence works and
|
||||
no usage rows or poll are required — usage is optional, holding a licence is not conditioned on it.
|
||||
- **Usage is readable in the clear; the key is not.** A test asserts a usage row is stored unsealed and
|
||||
is returned to an ordinary query, while the licence key remains sealed and absent from the same
|
||||
surfaces — the deliberate inversion of the credential rule.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A bill and a cap alarm are both queries.** "What did project X spend this month" reads the
|
||||
session-grain rows; "how close is account Y to its cap" reads the latest licence-grain metric — both
|
||||
from the usage store, neither a fold over events or a call to the vendor.
|
||||
- **The session becomes the unit of cost, which is what it already is.** Because a session is the
|
||||
consumer, attributing cost needs no new identity — and the worker-granularity gap
|
||||
([design 14](../03-DESIGN/01-to-be/14-model-access.md)) surfaces here exactly as it does for access,
|
||||
to be closed once, for both, when a session id is threaded through.
|
||||
- **The adapter carries the vendor's usage quirks alone.** Anthropic's `utilization%`, its transcript
|
||||
shape, a mid-session account switch — all live in the Anthropic adapter; the mesh, the store, and
|
||||
every reader see only rows. A second vendor adds a second adapter and no new table.
|
||||
- **Usage history is durable and tamper-evident by reuse, not by a new mechanism.** It rides the event
|
||||
trail the mesh already keeps, so an operator who wants the whole history has it, and one who wants the
|
||||
current number has the store — without usage owning either mechanism.
|
||||
- **The refresh-token carve-out is untouched by this.** Usage is read *from* an authenticated adapter;
|
||||
it neither holds nor exposes the credential, so the one place the mesh reads what it stores
|
||||
([ADR 0050](0050-model-access-is-vendor-agnostic.md)) is not widened by recording what that credential
|
||||
was spent on.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0050](0050-model-access-is-vendor-agnostic.md) — model access is vendor-agnostic; fixes the
|
||||
usage row shape and the `usage(licence)` adapter verb, and leaves its home open — which this closes
|
||||
- [ADR 0026](0026-the-mesh-has-a-session-of-its-own.md) — the mesh has a session of its own; a session
|
||||
is the consumer the finer grain attributes to
|
||||
- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md) — a scheduled step; the licence-grain poll is one
|
||||
- [ADR 0041](0041-events-are-a-relationship.md) — events are a relationship; a usage reading is one, and
|
||||
the audit-logger records it
|
||||
- [ADR 0042](0042-the-shape-of-an-event-on-the-wire.md) — the shape of an event on the wire; the form a
|
||||
usage reading takes to reach the audit trail
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store; the current usage picture
|
||||
lives in one
|
||||
- [ADR 0024](0024-model-access-is-a-provision.md) — model access is a provision; usage is a reading of
|
||||
what that provision was used for
|
||||
- [03-DESIGN/01-to-be/14-model-access.md](../03-DESIGN/01-to-be/14-model-access.md) — the model-access
|
||||
design; this fills its usage section and shares its open worker-naming gap
|
||||
- [03-DESIGN/01-to-be/15-the-agent-session.md](../03-DESIGN/01-to-be/15-the-agent-session.md) — the agent
|
||||
session; the consumer the session grain is keyed to
|
||||
@@ -1,146 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-07
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0050-model-access-is-vendor-agnostic.md
|
||||
---
|
||||
|
||||
# 55. Model access is answered by a licence, or by a node that hosts the model
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0024](0024-model-access-is-a-provision.md) made model access a provision, and
|
||||
[ADR 0050](0050-model-access-is-vendor-agnostic.md) fixed what answers it: a record — a licence — that
|
||||
an adapter turns into a sealed vendor credential.** A consumer requires `model-access`, is put on a
|
||||
licence, and is delivered a key (a static API key, or an access token a manager refreshes). Every
|
||||
answer so far has been a credential to reach a vendor's API across the internet.
|
||||
|
||||
**But a model need not come from a vendor. A node in the mesh can host one.** An operator with a GPU
|
||||
runs Ollama or vLLM, which serves an OpenAI-compatible API on that node. A consumer that wants that
|
||||
model does not need a vendor credential — it needs the model server's **endpoint**: the base URL and
|
||||
the model name, and a key only if the server is configured to want one. This is model access answered
|
||||
by a **node**, not by a record.
|
||||
|
||||
**The mesh already knows how a node answers a provision — it is the ordinary provider/consumer path.**
|
||||
A provider `provides` a provision at a scope, `serves` its connection facts, and the mesh fills the
|
||||
consumer's bound facts with the provider's `at`/`port` and each served fact — exactly how a Postgres
|
||||
consumer learns where its database is. Nothing reserved `model-access` to records: a node offering
|
||||
`provides: ["model-access"]` resolves through this path, and the resolver already **prefers a local
|
||||
answer over a licence** — its own comment names the case, "a model the mesh runs itself." So the
|
||||
capability exists; what is missing is the decision to use it, and the statement of what a node-answer
|
||||
delivers and where it stops.
|
||||
|
||||
**A node-answer and a record-answer are the same provision with two shapes of answer.** This is the
|
||||
same move [ADR 0050](0050-model-access-is-vendor-agnostic.md) already made for shapes within the
|
||||
vendor path (static-key vs refreshable-grant): one provision, more than one way it is answered. A
|
||||
consumer written against `model-access` should not care whether the model behind it is a vendor's or
|
||||
the mesh's own — it asks for model access and is given what reaches a model.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A separate provision for the local case (`local-model`, `model-endpoint`).** A node answers that;
|
||||
`model-access` stays record-only. Rejected: it splits "where my model comes from" into two
|
||||
provisions a consumer must choose between in its manifest, when the mesh already models a
|
||||
record-answer and a node-answer to **one** provision. A consumer would have to know, at authoring
|
||||
time, whether its model will be a vendor's or the mesh's — the exact coupling the provision was
|
||||
meant to remove. It is the safer implementation (see the limitation below) but the worse interface.
|
||||
|
||||
2. **Model access answered by either a licence or a node, under the one provision.** A consumer
|
||||
requires `model-access`; the operator answers it with a licence (a vendor) or by assigning a
|
||||
node that hosts a model. Adopted: one interface, and the answer is an operator's deployment choice,
|
||||
not a consumer's authoring choice.
|
||||
|
||||
## Decision
|
||||
|
||||
**Model access is one provision answered two ways: by a licence (a record, turned into a sealed vendor
|
||||
credential by an adapter — [ADR 0050](0050-model-access-is-vendor-agnostic.md)) or by a node that hosts
|
||||
the model (a provider that serves an endpoint).** A consumer requires `model-access` and is delivered
|
||||
whichever the operator assigned; it does not name the kind.
|
||||
|
||||
**A node-answer delivers an endpoint, not a credential.** The provider `provides: ["model-access"]`
|
||||
and `serves` its connection facts — the port it listens on and the model it runs — and the mesh fills
|
||||
the consumer's bound facts with the provider node's `at`, the served `port`, and the served `model`,
|
||||
the same way every provider consumer learns where its provider is. The consumer assembles a base URL
|
||||
(`http://<at>:<port>/v1`) and points an OpenAI-compatible client at it. If the local server wants a
|
||||
key, the provider mints one the ordinary way (a per-consumer secret, sealed and host-unsealed); if it
|
||||
does not — the common Ollama case — the consumer lists `model-access` under `binds` and **not** under
|
||||
`secrets`, and no key is delivered. Secret delivery and fact delivery are already independent, so a
|
||||
keyless endpoint is expressed by asking for the facts and not a secret.
|
||||
|
||||
**A node-answer uses no adapter.** The adapter registry ([ADR 0050](0050-model-access-is-vendor-agnostic.md))
|
||||
is the vendor-credential machinery — accept-and-seal, refresh, usage. A node-hosted model has no vendor
|
||||
secret to seal; its endpoint is served, and its key (if any) is minted like any provider's. The adapter
|
||||
is consulted only for the record/vendor answer. So the vendor-agnostic decision is untouched, and the
|
||||
node-answer adds no vendor logic anywhere.
|
||||
|
||||
**The resolver prefers a local answer.** When a node's own set answers `model-access` — a model the
|
||||
mesh runs itself — a licence for it is not consulted. This is already the resolver's behaviour and is
|
||||
made a decision here: a mesh that runs a model uses it, and a licence is the answer for a consumer that
|
||||
has no local model, not a competitor to one that does.
|
||||
|
||||
**One limitation, stated so it is not found as a bug.** The local-preference above is exact for a
|
||||
**node-scope** provider co-located with its consumer, and for any node-answer in a mesh that holds no
|
||||
`model-access` licence. It is *not* yet exact for a **mesh-scope** model server — one node serving the
|
||||
model to others — **while a licence for `model-access` also exists in the same mesh**: the record pass
|
||||
that turns a licence into an answer keys on same-node satisfaction and would still demand the licence be
|
||||
used, double-answering. Until that pass is taught to stand down when a brokered node need already
|
||||
answers, a mesh-scope local model and a vendor licence must not both answer `model-access` in one mesh.
|
||||
A node-scope local model has no such constraint. This is named because an unstated limitation is
|
||||
indistinguishable from a bug, and costs more.
|
||||
|
||||
This is a decision and not a patch because it settles **what may answer model access** — a question
|
||||
every model-access consumer's meaning depends on — and because it lets the mesh's own hosted models sit
|
||||
behind the same provision as the vendors', which is what makes "the mesh can run its own model" a
|
||||
deployment choice rather than a second interface to build against.
|
||||
|
||||
### How each claim is checked
|
||||
|
||||
- **A node answers model access without a licence, and is preferred over one.** A resolver test
|
||||
assigns a consumer and a module that `provides: ["model-access"]` at node scope on the one node, with
|
||||
a licence also present, and asserts no resolved need is answered by the record — the local model
|
||||
answers and the licence is ignored. (This test exists; the decision adopts what it proves.)
|
||||
- **The consumer is delivered an endpoint, not a credential.** A mesh bed assigns a model-server
|
||||
provider and a consumer that binds `model-access` and does not list it under `secrets`, and asserts
|
||||
the consumer's config carries `OPENAI_BASE_URL` built from the provider's served `at`/`port`, and
|
||||
that no key file was delivered to its secret path.
|
||||
- **The local endpoint is reachable through what the consumer was given.** The bed makes a request to
|
||||
the base URL the consumer wrote and asserts the model server answers — the wiring, not a model's
|
||||
output, is what is proven (the server may be a stub; a real model is not needed to prove the mesh
|
||||
routed the consumer to it).
|
||||
- **A node-answer consults no adapter.** A node-answered `model-access` need is resolved with the
|
||||
vendor registry never read — asserted by the absence of any vendor on a node-answered need and the
|
||||
ordinary served-facts delivery.
|
||||
- **The scope limitation holds where stated.** The node-scope case is what the bed and the resolver
|
||||
test exercise; the mesh-scope-plus-licence collision is recorded here and left for the resolver
|
||||
change that reconciles the two answer passes, not worked around in a module.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The mesh can run its own model, and a consumer reaches it through the same `model-access` it uses
|
||||
for a vendor.** One interface, two answers; a consumer moves between a vendor and a local model by an
|
||||
operator reassigning its provision, not by a code change.
|
||||
- **A local model is keyless by default and keyed by the ordinary path when it must be.** Nothing new
|
||||
is invented for the local server's credential: it either has none, or mints one the way every
|
||||
provider does.
|
||||
- **The vendor path is untouched.** Adapters, the refresh carve-out, and usage
|
||||
([ADR 0054](0054-model-usage-is-recorded-at-two-grains.md)) are the record answer's business; a
|
||||
node-answer neither uses nor changes them. A local model that exposes usage would serve it as facts,
|
||||
not as an adapter's usage verb.
|
||||
- **The two answer passes meet in one place, and must be reconciled there.** The mesh-scope limitation
|
||||
is the single point where a node-answer and a record-answer to the one provision can collide; it is
|
||||
named, and its fix is a resolver change, not a per-module workaround.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0024](0024-model-access-is-a-provision.md) — model access is a provision; this decides a node
|
||||
may answer it, not only a record
|
||||
- [ADR 0050](0050-model-access-is-vendor-agnostic.md) — model access is vendor-agnostic; the record
|
||||
answer and its adapters, which the node answer sits beside and does not use
|
||||
- [ADR 0054](0054-model-usage-is-recorded-at-two-grains.md) — model usage; a vendor's business on the
|
||||
record path, served as facts (if at all) on the node path
|
||||
- [ADR 0008](0008-a-context-owns-its-store.md) — a context owns its store; a licence is a record in one,
|
||||
a node-answer needs none
|
||||
- [03-DESIGN/01-to-be/14-model-access.md](../03-DESIGN/01-to-be/14-model-access.md) — the model-access
|
||||
design, which this extends with the node answer
|
||||
@@ -1,131 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-09
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0007-connectivity.md
|
||||
---
|
||||
|
||||
# 66. Public routing is name-agnostic, its names are resolved inside the mesh, and an internal authority can certify them
|
||||
|
||||
> **Superseded in part — 2026-10-03, by [ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md).**
|
||||
> One clause of the decision below no longer holds: *publishing a granted name into internal resolution,
|
||||
> mesh-wide*. A public name now resolves publicly, and only names under the mesh's own suffix get a private answer.
|
||||
> Inside the mesh a route is reached and certified by its internal name
|
||||
> ([ADR 0151](0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md)). The label, the
|
||||
> node's public domain and their composition stand as decided here.
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0007](0007-connectivity.md) and [connectivity §3](../03-DESIGN/01-to-be/08-connectivity.md)
|
||||
made a public route a grant: a workload that must be reachable requires a route, the proxy provides
|
||||
it, the consumer contributes the name it wants and the port it listens on.** What was never pinned
|
||||
is **what that name is** — and building a whole mesh in the lab showed the gap costs more than it
|
||||
looks.
|
||||
|
||||
**The catalogue shipped each route as a full domain.** A module that needed a public name carried
|
||||
that name, in full, as a literal in its manifest. Running the same catalogue against a different
|
||||
domain — a lab standing in for production, or a second operator's mesh — meant overriding that
|
||||
literal on every routed module, per node. The mesh was, in effect, carrying a **map of names to
|
||||
services**: the one thing it should never hold, because a name is the operator's choice (one runs
|
||||
the forge at `git`, another at `code`) and the domain is the node's, and neither is the mesh's to
|
||||
know.
|
||||
|
||||
**And a second gap surfaced the moment an internal issuer tried to certify those names.**
|
||||
[Connectivity §5](../03-DESIGN/01-to-be/08-connectivity.md) already states the issuer must be
|
||||
configurable and that the lab runs its own ACME authority. With that authority wired to the proxy,
|
||||
issuance still could not complete: the authority accepted the order and offered a challenge, then
|
||||
**could not connect to the validation target.** Nothing inside the mesh resolved the public route
|
||||
name. The mesh publishes each `<node>.internal` name into every container, but not the public names
|
||||
the proxy serves — so a validator living in the mesh had no address to reach, and a name the mesh
|
||||
cannot resolve is a name it cannot have certified.
|
||||
|
||||
**The two are one problem.** A name the mesh can *compose* from parts it is given, and *propagate*
|
||||
to whoever needs to resolve it, is exactly a name it can also have *certified* — and the reverse:
|
||||
without the composition and the propagation, neither the routing nor the certificate is the
|
||||
operator's to move between meshes.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Keep the full domain in the manifest, override per node.** The status quo. It works, and it is
|
||||
wrong in the specific way this repository cares about: the catalogue holds a domain map, lab and
|
||||
production differ by an override on every routed module rather than one fact, and a module manifest
|
||||
names something — the public domain — that belongs to the node, not the module. An unowned name in
|
||||
the wrong place is the shape of a leak.
|
||||
|
||||
**2. A module declares a label; the node declares its public domain; the mesh composes.** The route
|
||||
contribution carries a subdomain the operator chose, the node carries its public domain as
|
||||
node-level configuration, and the mesh joins `<label>.<public-domain>` and grants exactly that. The
|
||||
mesh interprets nothing. Lab-versus-production becomes one node setting. Chosen.
|
||||
|
||||
**3. For certification, issue only from a publicly reachable node against a public authority.**
|
||||
This is already true for public meshes and stays true. It is not an option for a lab or an
|
||||
internal-only mesh: there is no public authority to answer, and no public reachability to validate
|
||||
against. An internal issuer is required there — and an internal issuer must be able to *validate*,
|
||||
which it cannot do unless the routed name resolves and is reachable **inside** the mesh. So the
|
||||
resolution gap is not optional to close; it is what makes an internal authority possible at all.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh core holds no map of hostnames, subdomains or domains.** A route contribution carries a
|
||||
**label** (the subdomain) chosen by the module's operator. A node contributes its **public domain**
|
||||
as node-level configuration. The mesh composes `<label>.<public-domain>`, grants exactly that name,
|
||||
and never interprets what it means. One operator's forge at `git.example.tld` and another's at
|
||||
`code.other.example` are the same module with two facts supplied around it.
|
||||
|
||||
**When the proxy is granted a name, the mesh publishes that name → the node that serves it into
|
||||
internal resolution, mesh-wide** — the same mechanism, and the same "given by the mesh, not chosen
|
||||
by a module," that already writes `<node>.internal` into every declared container
|
||||
([connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md)). The mesh propagates the names it was
|
||||
told to serve. It still knows nothing about what any of them mean.
|
||||
|
||||
**An internal authority certifies those names by the same path a public one would.** The proxy is
|
||||
pointed at whichever issuer the mesh names — a public ACME authority, or an internal one — and
|
||||
trusts that issuer's root; nothing else about issuance changes. The internal authority validates by
|
||||
reaching the routed name, which the clause above has just made resolvable inside the mesh. So the
|
||||
three are one decision: **compose the name, propagate it, certify it** — each is meaningless without
|
||||
the one before it.
|
||||
|
||||
> **The mechanism changed — 2026-09-30, by [ADR 0148](0148-the-meshs-names-are-resolved-not-copied-into-containers.md).**
|
||||
> A routed name still reaches every asker in the mesh, which is what this record decided and it stands.
|
||||
> It no longer reaches them by being written into each declared container: copying the roster in made the
|
||||
> roster part of every container's identity, so one name moving replaced every container in the mesh
|
||||
> ([issue 151](../04-ISSUES/151-a-new-name-recreates-every-container-in-the-mesh/00-report.md)). A
|
||||
> container resolves through its machine's resolver instead. The consequence below — that an internal
|
||||
> issuer's challenge needs the routed name resolvable inside the mesh — holds unchanged, by the means the
|
||||
> machine itself already uses.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Lab-versus-production is one node-level `public-domain` setting**, not an override on every
|
||||
routed module. The same catalogue runs against any domain.
|
||||
- **The manifest layer needs composition it does not yet have.** Today a route name is stored as a
|
||||
literal, with no interpolation of a node's domain into a module's label. Until that exists, the
|
||||
composed name is produced by a per-node settings override — a stopgap that reproduces option 1's
|
||||
per-module cost and is explicitly *not* the design.
|
||||
- **An internal issuer depends on route-name resolution.** Its challenge validates against the
|
||||
routed name; without that name in internal resolution, issuance for it cannot complete inside the
|
||||
mesh. The lab found this as a live failure, not a theory.
|
||||
- **Nothing about the public path changes.** A publicly reachable node issuing a public name from a
|
||||
public authority is untouched; this widens the same shape to names and meshes that are not public.
|
||||
|
||||
**How each is checked** — an unenforced rule is indistinguishable from a wrong one:
|
||||
|
||||
- **Name-agnostic:** the same catalogue resolves against two different public domains by changing
|
||||
one node setting and nothing else; and no module manifest contains a full public domain. A
|
||||
manifest that pins an FQDN is the smell the check looks for.
|
||||
- **Resolution:** a request to a routed name, made from inside the mesh, reaches the workload that
|
||||
serves it — and, the sharper check, issuance for that name against the internal authority
|
||||
completes, which it cannot unless the validator resolved and reached the target.
|
||||
- **Internal authority:** a TLS handshake to a routed name verifies against the internal root and
|
||||
nothing else, the same shape §5 already uses for internal node-to-node names.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0007 — connectivity](0007-connectivity.md), which made a route a grant and named exposure,
|
||||
resolution and certificates as one context.
|
||||
- [ADR 0009 — modules and the graph](0009-modules-and-the-graph.md), the provide/require/contribute
|
||||
vocabulary a route and an authority both use.
|
||||
- [Connectivity design §2, §3, §5](../03-DESIGN/01-to-be/08-connectivity.md), amended alongside this
|
||||
record.
|
||||
@@ -1,129 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-10
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 67. Genesis is a pivot: a temporary control plane installs the registry that makes it permanent
|
||||
|
||||
## Context
|
||||
|
||||
**The mesh builds its own modules into its own registry, and there is no public registry for them
|
||||
— that is the point, not an omission.** A module is cloned from the forge, built, and published
|
||||
where the mesh can move, replace and back it up. Nothing about that arrangement wants a copy of the
|
||||
mesh's code hosted by somebody else.
|
||||
|
||||
**Every image must be pinned by digest** ([ADR 0006](0006-the-substrate-and-the-control-plane.md)),
|
||||
and the reasoning is exactly right: a bundle is applied where no mesh exists to check anything
|
||||
against anything, so what it names must be exact.
|
||||
|
||||
Those two sentences are individually correct and together they close a door. The digest a pin means
|
||||
is a *manifest* digest, and a manifest digest is **assigned by a registry when something is pushed
|
||||
to it**. The control plane's image is built from source and pushed nowhere, so it has no such
|
||||
digest, so it cannot be named — and a registry cannot be installed without a control plane to
|
||||
install it. That is not a pin. It is a dependency the pinning rule created by accident.
|
||||
|
||||
**It went unnoticed because the lab hid it.** The lab raised a disposable registry, stocked it from
|
||||
a workstation, and rewrote every image reference to point at it — so the lab bootstrapped along a
|
||||
path no real machine has. A first node in the lab always worked, and a first node anywhere else had
|
||||
no path at all. Every bootstrap fault found this year was found late for the same reason: **the
|
||||
install procedure existed only as a test fixture**, and a fixture is free to invent what it needs.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Publish the mesh's own images to a public registry.** Rejected on the premise: there is no
|
||||
public registry for the mesh's modules and there is not meant to be. It would also make raising a
|
||||
mesh depend on somebody continuing to host its code, which is the dependency the whole arrangement
|
||||
exists to remove.
|
||||
|
||||
**2. Build the control plane from source on the first machine.** Rejected. A bare machine would
|
||||
need a toolchain and a working tree — and worse, the source lives in a forge **that runs on the
|
||||
mesh**. A total rebuild would then need the mesh it is rebuilding. Acceptable for adding a node to
|
||||
a healthy mesh; useless for the case that matters.
|
||||
|
||||
**3. Keep a disposable registry as an install step.** Rejected. It exists in no production, and
|
||||
concealing this problem is precisely what it has been doing.
|
||||
|
||||
**4. Pivot through a temporary control plane.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**An image may be named by the digest of its own configuration.** A bare `sha256:…` names an image
|
||||
the machine already holds — content-addressed, immutable, unforgeable, and requiring nothing to
|
||||
have served it. It satisfies what the pinning rule asks for; the rule simply never contemplated an
|
||||
image that no registry had ever seen. It is legal exactly where nothing could have served one.
|
||||
|
||||
**Genesis is a pivot**, in this order:
|
||||
|
||||
1. the installer **carries the control-plane image** and loads it onto the machine
|
||||
2. a **temporary** control plane is raised from it, named by that image's own digest
|
||||
3. the **registry module is installed** — its image is upstream and it is never built, which
|
||||
[`04-ISSUES/029`](../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md)
|
||||
already settled: a module that provides the artifact store cannot be delivered through it
|
||||
4. the control-plane image is **pushed into the mesh's own registry**, which assigns it a manifest
|
||||
digest — the first one it has ever had
|
||||
5. the control plane is **reinstalled as an ordinary module** pinned to that digest
|
||||
|
||||
**The host performs the replacement, not the control plane.** Tier 0 outlives tier 2: the control
|
||||
plane composes a declaration naming the registry-pinned image, and the host applies it and recreates
|
||||
the container. Nothing is asked to replace itself while running, and the control plane is stateless
|
||||
— what it knows is in the store.
|
||||
|
||||
**The installer is tier 0, and a separate program from the host.** Bootstrapping is by hand and
|
||||
changes the machine, which is tier 0's definition. But the host states that it *connects to nothing
|
||||
and listens on nothing*, and that claim is what makes the one thing running forever on every
|
||||
machine auditable. An installer connects to plenty. Same tier, same delivery, different program.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The control plane stops being a special case.** It becomes an ordinary module with an ordinary
|
||||
image in the mesh's own registry — so the mesh can build and roll out **its own upgrades**, which
|
||||
is what a mesh that runs itself was always reaching for.
|
||||
- **The bundle's job shrinks** to raising a temporary control plane exactly once.
|
||||
- **The source builds the installer; it does not run it.** Cloning moves to a release machine, where
|
||||
a forge being available is an ordinary working assumption, and leaves the disaster-recovery path
|
||||
where it very much is not.
|
||||
- **The lab's disposable registry is deleted.** The lab bootstraps by running the same program a
|
||||
bare machine runs — the only arrangement in which the installer cannot quietly drift out of truth
|
||||
again.
|
||||
- **Two public images remain at genesis** — the store and the broker. An air-gapped install would
|
||||
embed those too, at a much larger artifact; that is a build variant, not a different design.
|
||||
- **A control-plane module manifest must exist**, and did not.
|
||||
- **The registry may require nothing.**
|
||||
[`04-ISSUES/029`](../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md)
|
||||
states that a module providing the artifact store may not *build* artifacts, because there is
|
||||
nowhere to put them until it runs. The pivot shows that is the narrow case of a wider rule: **it
|
||||
may not require anything the store is needed to deliver.** Found the hard way — an unrelated
|
||||
change gave the registry a public name and, with it, a route requirement. At genesis nothing
|
||||
provides a route, and nothing can, because the routing stack needs images and images need the
|
||||
store. The same cycle, re-entered through a door the existing wording did not cover.
|
||||
- **The handover is the sharp edge.** For one moment the bundle and the module both describe the
|
||||
same container, and the host tracks what it owns. If a safe handover is not expressible with what
|
||||
exists, the install **stops before it** and says what is missing. A machine left without a control
|
||||
plane cannot be fixed remotely, so a partial install that halts cleanly is the better outcome.
|
||||
|
||||
**How each is checked** — an unenforced rule is indistinguishable from a wrong one:
|
||||
|
||||
- **Naming by its own digest:** a machine that can reach no registry at all raises a control plane.
|
||||
- **The pivot completed:** after installing, the running control plane's image is pinned by a digest
|
||||
**the mesh's own registry assigned** — not by an image id. If it is still the image id, the pivot
|
||||
did not happen and the mesh cannot upgrade itself.
|
||||
- **No fiction left in the lab:** the scenario declares no registry machine, and the bed bootstraps
|
||||
through the installer rather than around it.
|
||||
- **The handover:** a machine whose control plane has been replaced still has one, and it answers.
|
||||
- **The registry requires nothing:** its manifest is resolvable on a mesh that has no other module
|
||||
in it. A requirement added to it later is caught where it is written, rather than by a genesis
|
||||
that cannot complete — which is how this one was found.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0006 — the substrate and the control plane](0006-the-substrate-and-the-control-plane.md),
|
||||
which pinned images by digest and named what a first node fetches.
|
||||
- [ADR 0005 — the node host](0005-the-node-host.md), which makes tier 0 the one thing installed by
|
||||
hand and the only thing that changes a machine — the property this keeps true by shipping the
|
||||
installer beside the host rather than inside it.
|
||||
- [`04-ISSUES/029`](../04-ISSUES/029-the-artifact-store-cannot-be-delivered-by-the-artifact-store/00-report.md),
|
||||
the same cycle one layer down, and the rule that a registry module is named and never built.
|
||||
@@ -1,119 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: superseded
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0016-the-lab.md
|
||||
superseded-by: 02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md
|
||||
---
|
||||
|
||||
# 68. The lab takes requests, one at a time, and runs each from its own copy
|
||||
|
||||
> **Superseded — 2026-09-30, by [ADR 0149](0149-the-live-mesh-is-the-test-bed.md).** Never built. The
|
||||
> live mesh became the test bed, because the faults that cost the most are faults of a mesh that
|
||||
> already exists — bound consumers, containers made against an older roster, an adopted machine — and a
|
||||
> bed is by construction a mesh that does not. The rule worth keeping from below is that a run reads a
|
||||
> copy that is not anybody's working tree.
|
||||
|
||||
## Context
|
||||
|
||||
**The lab is exclusive hardware, and today a person holds it.** Raising a scenario takes over
|
||||
addresses and names on the workstation for as long as it stands, and only one scenario can stand
|
||||
at a time. So a run is not merely slow — it occupies the machine and the person who started it,
|
||||
who then waits rather than works.
|
||||
|
||||
**Running it in the background against the working copy is worse than waiting.** The obvious fix
|
||||
is to start a run and carry on editing. But a run reads the working copy as it goes: binaries are
|
||||
rebuilt from it, manifests are read out of it, and the bed's own code is loaded from it. Edit
|
||||
while it runs and the result describes a state that never existed — a mixture of what was there
|
||||
when each file happened to be read. A green result obtained that way is not evidence, and a red
|
||||
one costs a day to disbelieve.
|
||||
|
||||
**Nothing today records what was asked for.** A run is a command line in somebody's terminal. What
|
||||
commit it exercised, what it was trying to find out, and what it answered all live in scrollback,
|
||||
which is why the same question gets re-run rather than looked up.
|
||||
|
||||
**Most of the parts already exist.** The lab writes a receipt of its last run. The mesh already
|
||||
carries messages between nodes and can notify a person. The machine already runs work on a
|
||||
schedule. What is missing is the thing in the middle.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Leave it as it is — a person drives the lab and waits.** Rejected. It is the loop
|
||||
[ADR 0010](0010-delivery.md) removed everywhere else, kept here by habit rather than by argument,
|
||||
and the cost compounds: because a run is expensive to start and blocks the person, fewer are run,
|
||||
so faults are found later and in larger batches.
|
||||
|
||||
**2. Run in the background against the working copy.** Rejected on the reasoning above. The
|
||||
failure is silent, which is the kind this repository exists to refuse.
|
||||
|
||||
**3. Put the lab behind the ordinary build pipeline.** Rejected for now. The pipeline builds
|
||||
artifacts and does not own a machine that can raise virtual machines; giving it one makes the
|
||||
pipeline's slowest job the lab's, and couples every push to hardware only one machine has. This
|
||||
may become right later; it is not the smallest thing that works.
|
||||
|
||||
**4. A queue in front of the lab, and an isolated copy behind it.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**The lab accepts requests rather than commands.** A request is recorded, queued, and answered.
|
||||
The person who made it is told when it is answered and does not wait.
|
||||
|
||||
**A request names a bed and a commit, and nothing else.** This is the load-bearing restriction. A
|
||||
request may say *run this bed, at this version of these repositories*. It may not say what to
|
||||
install, on which machine, or with which settings — because a request that could say those things
|
||||
would be a second way of installing a mesh, and the whole reason the installer exists is that the
|
||||
lab already was one ([ADR 0067](0067-genesis-is-a-pivot.md)). The bed decides what is installed;
|
||||
the request only decides which bed and which version.
|
||||
|
||||
**Requests are released one at a time.** The hardware admits one standing scenario, so the queue
|
||||
enforces what the hardware already requires, rather than leaving it to whoever remembers.
|
||||
|
||||
**Every run happens in a copy the lab owns.** The lab checks the requested commit out into its own
|
||||
path and builds and runs from there. A working copy is never read by a run. This is what makes the
|
||||
queue safe to use while work continues, and without it the rest of this record is not worth
|
||||
having.
|
||||
|
||||
**The lab is reached through tools, not only a command line.** A command line is available only
|
||||
to whoever is sitting at the machine, which is the constraint this record exists to remove. The
|
||||
lab answers three questions to anything that can reach the mesh — *what is standing now*, *what is
|
||||
queued or running*, and *what did this request answer* — and accepts a request and a cancellation.
|
||||
An agent can therefore start a run, stop attending to it, and come back; and somebody who did not
|
||||
start a run can still see it, which is the difference between a shared lab and a private one.
|
||||
|
||||
**The restriction holds at every door.** A tool submits a bed and a commit, exactly as a command
|
||||
line does. A tool that could name a module, a node or a setting would reintroduce the second
|
||||
installer through a different entrance, and the entrance is not what made it dangerous.
|
||||
|
||||
**Every run leaves a record that outlives the terminal**: what was asked, which commit, when it
|
||||
ran, what it answered, and where its output went. A question already answered is looked up rather
|
||||
than re-run.
|
||||
|
||||
## Consequences
|
||||
|
||||
Work continues while the lab runs, which is the point. A second session may edit freely, because
|
||||
nothing it edits is what the lab is reading.
|
||||
|
||||
A request is reproducible by construction: it names a commit, so the same request can be asked
|
||||
again and compared. Today two runs of "the same thing" are only as alike as the tree happened to be.
|
||||
|
||||
The lab gains a second copy of every repository it exercises, costing disk and needing to be kept
|
||||
from drifting into a place people edit by hand.
|
||||
|
||||
Anything that can reach the mesh can now see what the lab is doing, including an agent working on
|
||||
something else. That is the intended gain and also the obvious hazard: a thing that is easy to ask
|
||||
is easy to ask too often, and the hardware still admits one scenario at a time.
|
||||
|
||||
The queue becomes a thing that can fail — stuck, backed up, or lost — and a queue nobody watches
|
||||
is worse than no queue, because it absorbs requests silently.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A run never reads a working copy | The runner is given a path it owns and no other; a run started while a working copy is deliberately dirtied produces a result matching the commit, not the edits. |
|
||||
| One scenario stands at a time | A second request submitted while one runs is observed to wait, not to raise. |
|
||||
| A request cannot say what to install | The request format admits a bed and a commit only. A request naming a module, a node or a setting is refused, and the refusal is exercised. |
|
||||
| A request is answered | Every queued request reaches a terminal state with a record. A request that vanishes is a failure of the queue, not a quiet nothing. |
|
||||
| The lab can be asked from elsewhere | What is standing is asked from a session that did not raise it, and the answer matches the machine. A lab that only answers its own caller has not left the terminal. |
|
||||
@@ -1,89 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 69. A module is a repository and a path within it
|
||||
|
||||
## Context
|
||||
|
||||
**The builder clones one repository and reads `module.json` at its root.** The to-be design says
|
||||
so in as many words — *"one file at the root"* — and the code implements it: clone, read the root
|
||||
manifest, build what it declares.
|
||||
|
||||
**Nothing that exists is shaped that way.** The catalogue holds sixty-seven modules, each in its
|
||||
own directory, and has no manifest at its root. None of the five code repositories has one either.
|
||||
So today the builder cannot be asked to build any module that exists: pointed at the catalogue it
|
||||
finds no manifest, and pointed at a module's source it finds no manifest.
|
||||
|
||||
**The system being replaced already works the other way**, and has for years: a monorepo with one
|
||||
directory per piece of software, and the coordinator builds a module from a repository and a path
|
||||
inside it. The root-only assumption is not a simplification of that — it is a different model that
|
||||
was never reconciled with it.
|
||||
|
||||
**And it splits what a build needs into two places.** The control plane's manifest sits in the
|
||||
catalogue; the source it describes sits in the control plane's own repository. A build must read
|
||||
one tree, so under the root-only model neither location can be built from.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. One repository per module.** Rejected. Sixty-seven repositories for sixty-seven modules, most
|
||||
of which are a single manifest naming a public image, and every one needing its own creation,
|
||||
permissions and lifecycle. It also contradicts [ADR 0015](0015-applications-live-in-their-own-repository.md),
|
||||
which put *applications* in their own repositories precisely because modules do not need one.
|
||||
|
||||
**2. Keep manifests in the catalogue and source elsewhere, and have a build fetch both.** Rejected.
|
||||
A build would clone two trees whose versions can disagree, so "what commit is this module?" stops
|
||||
having one answer — and that question is the whole basis of knowing when to rebuild.
|
||||
|
||||
**3. A module is a repository and a path within it.** Chosen. It is what the current system does,
|
||||
what the catalogue already looks like, and it keeps a module's description beside the thing it
|
||||
describes.
|
||||
|
||||
## Decision
|
||||
|
||||
**A module is named by a repository and a path within it.** The path holds `module.json`, and
|
||||
everything that manifest declares is produced from that path. A module whose path is the root is
|
||||
the ordinary case of this, not a separate one.
|
||||
|
||||
**A module's manifest lives beside its source.** Where a module has code, its directory holds both,
|
||||
so one commit answers "what is this module, and what is it made of". Where a module has no source —
|
||||
a manifest naming a public image — the directory holds only the manifest, and there is nothing to
|
||||
build.
|
||||
|
||||
**This moves the core modules.** The control plane and the builder are built from the control
|
||||
plane's repository, so their manifests belong in that repository at their own paths, not in the
|
||||
catalogue. The catalogue keeps the modules whose source it holds, and the modules that are only a
|
||||
manifest.
|
||||
|
||||
**One commit, one module version.** Because a module is one path in one repository, the commit that
|
||||
built it identifies it exactly, and "the source has moved ahead of what the mesh holds" stays a
|
||||
question with a yes or no answer.
|
||||
|
||||
## Consequences
|
||||
|
||||
The builder gains a path alongside the repository and the ref. A build is `repository, path, ref`,
|
||||
and the manifest it returns is the module the mesh records.
|
||||
|
||||
The catalogue stops being the place every manifest lives, and becomes the place manifests live
|
||||
*when their module has no other home*. That is a smaller claim than it sounds: most of the
|
||||
sixty-seven stay exactly where they are.
|
||||
|
||||
Two repositories change shape — the control plane's gains manifests for the modules built from it.
|
||||
Nothing else moves.
|
||||
|
||||
A repository can hold modules that are built and modules that are not, and no rule distinguishes
|
||||
them beyond whether their manifest declares anything to build.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| A module is buildable from its repository and path | The builder is asked for a module by repository and path, and returns a manifest whose artifacts are pinned to digests the mesh's registry assigned. |
|
||||
| A manifest sits beside what it describes | A module declaring something to build, whose path holds no source to build it from, is refused at build time rather than producing an empty result. |
|
||||
| One commit identifies one module | Two builds of the same repository, path and commit produce the same digests. |
|
||||
| The core modules are built like any other | The control plane is rebuilt from its own repository and path, and the running mesh is upgraded to it — the same path an ordinary module takes. |
|
||||
@@ -1,107 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0067-genesis-is-a-pivot.md
|
||||
---
|
||||
|
||||
# 70. The catalogue owns the module graph, and genesis builds rather than carries
|
||||
|
||||
## Context
|
||||
|
||||
**The module graph has no owner.** What modules exist, what each requires and provides, what each
|
||||
claims, what each is made of — all of it lives inside the control plane because that is where it
|
||||
was first written, not because anything decided it belonged there.
|
||||
|
||||
**The control plane's own test says it does not belong there.**
|
||||
[`06-the-control-plane`](../03-DESIGN/01-to-be/06-the-controller.md) defines the tier as
|
||||
*everything that needs to know about more than one node*, and states the corollary plainly:
|
||||
anything a single machine could answer alone is not the control plane's. What a module is, and
|
||||
what it needs, requires no knowledge of any node whatsoever.
|
||||
|
||||
**And nothing can query it.** The graph is the thing that answers *what must be rebuilt when this
|
||||
changes*, *what would break if this were removed*, and *what can be installed here* — and today it
|
||||
is reachable only as control-plane internals. [ADR 0009](0009-modules-and-the-graph.md) already
|
||||
decided that a build edge is derived, that an artifact is stale when anything it was built against
|
||||
moved, and that the rebuild set is therefore computable. None of that has anywhere to live.
|
||||
|
||||
**Genesis currently carries an image, and cannot produce a builder at all.**
|
||||
[ADR 0067](0067-genesis-is-a-pivot.md) has the installer carry the control plane's image. That
|
||||
works, but it leaves the builder with no route onto a fresh mesh — it cannot be fetched from the
|
||||
public internet, because the mesh builds it, and the installer carries one image only. So a raised
|
||||
mesh cannot build anything, including the modules it is made of.
|
||||
|
||||
## Decision
|
||||
|
||||
**The catalogue is a core module, beside the control plane and the builder, and it owns the module
|
||||
graph.** Modules, what they require and provide, what they claim, their dependencies and their
|
||||
build edges, assignments and configuration — the catalogue holds them and serves tools over them:
|
||||
install a module, query what is available and what it needs, query the graph, query and update
|
||||
settings.
|
||||
|
||||
**It is one per mesh**, expressed the way the control plane already expresses it — a claim scoped
|
||||
to the mesh, not a new mechanism.
|
||||
|
||||
**The control plane consumes it.** Resolving what a module requires into an actual binding, and
|
||||
composing what a machine should be, both need to know what modules are. So the dependency runs
|
||||
from the control plane to the catalogue, which is the opposite of what the tiers suggest and is
|
||||
therefore written down here rather than left to be inferred.
|
||||
|
||||
**Genesis builds the core modules rather than carrying them.** The installer ships an *init
|
||||
builder* — the one thing carried — which is started, clones the source, and builds the control
|
||||
plane, the catalogue and the builder. The installer then raises a temporary control plane, which
|
||||
installs the catalogue, registers the permanent control plane and the builder, and assigns all
|
||||
three to the first machine. The temporary control plane stops; the installer verifies that the
|
||||
permanent one answers and can query the graph. The builder then sees a catalogue in its initial
|
||||
state and builds the core modules into it.
|
||||
|
||||
**So exactly one thing is carried, and it is a builder rather than a result.** That is the
|
||||
difference from [ADR 0067](0067-genesis-is-a-pivot.md), which carried the control plane's image:
|
||||
carrying a builder produces every core module on the machine, including the builder itself, so
|
||||
there is no component left without a route.
|
||||
|
||||
## Consequences
|
||||
|
||||
The catalogue joins the small set of things that cannot arrive through the ordinary path, because
|
||||
it cannot be installed by something that needs it in order to install anything. It arrives the same
|
||||
way everything else does under this record — built by the init builder before the mesh can install
|
||||
anything — so the set is answered by one mechanism rather than three special cases.
|
||||
|
||||
The rebuild fan-out gains a home. *What was this built against* and *what must rebuild now* are
|
||||
questions about the graph, and the graph now has an owner to hold the edges and answer them.
|
||||
|
||||
The catalogue holds state, so it owns a store in the substrate's database, the same way the control
|
||||
plane's contexts do. That is the mesh's own store and not the `postgres` module, which is a
|
||||
provider other modules consume.
|
||||
|
||||
A mesh without a catalogue cannot resolve anything, where previously it merely lacked an interface.
|
||||
That is the cost of ownership over surfacing, and it is deliberate: one owner beats two copies.
|
||||
|
||||
## Open, and to be settled before this is built
|
||||
|
||||
**Where the init builder clones from.** [ADR 0067](0067-genesis-is-a-pivot.md) rejected building
|
||||
from source at genesis partly because the source lives in a forge that runs on the mesh, so a
|
||||
total rebuild would need the mesh it is rebuilding. Carrying a builder answers the toolchain half
|
||||
of that objection and not this half. Genesis must therefore name a source that exists before the
|
||||
mesh does.
|
||||
|
||||
**What the init builder publishes into.** Building produces artifacts that must be pinned by a
|
||||
digest a registry assigned, and today the registry is installed after the machine has joined. If
|
||||
the core modules are built first, the registry has to exist first, so the substrate's order needs
|
||||
restating rather than assumed.
|
||||
|
||||
**Where the line falls between the catalogue and the control plane.** Claims and assignments need
|
||||
to know about every node, which by the control plane's own test is its work. Whether the catalogue
|
||||
holds them and asks, or the control plane holds them and the catalogue surfaces them, is not
|
||||
settled here.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The catalogue owns the graph | The control plane answers *what does this module require* by asking the catalogue, and a mesh whose catalogue is stopped cannot resolve — observed, not assumed. |
|
||||
| One per mesh | A second catalogue assigned anywhere in the mesh is refused by the claim, and the refusal is exercised. |
|
||||
| Genesis builds rather than carries | The installer carries exactly one artifact, and after installing, every core module is pinned to a digest the mesh's own registry assigned. |
|
||||
| The builder has a route | A mesh raised by the installer, with no hand-placed image, can build a module. |
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md
|
||||
---
|
||||
|
||||
# 71. Genesis clones from a mesh, and checks what it got
|
||||
|
||||
## Context
|
||||
|
||||
**[ADR 0070](0070-the-catalogue-owns-the-module-graph.md) has the init builder clone the source,
|
||||
and does not say from where.** [ADR 0067](0067-genesis-is-a-pivot.md) had already rejected building
|
||||
at genesis partly for that reason: the forge holding the source runs *on* the mesh, so a total
|
||||
rebuild would need the mesh it is rebuilding.
|
||||
|
||||
That objection is real but narrower than it reads. It only binds when the mesh being raised and the
|
||||
mesh holding the source are the same one, which is true exactly once.
|
||||
|
||||
## Decision
|
||||
|
||||
**Genesis clones from a mesh's forge, reached by name.** Any mesh that holds the source can serve
|
||||
it. The first mesh is not structurally special — it is simply the only one that existed when there
|
||||
was nothing else to clone from.
|
||||
|
||||
**If the mesh serving the source is lost, the name moves to another mesh that holds a copy.**
|
||||
Recovery is a name pointing somewhere else, not a backup being restored. This is what makes the
|
||||
source's survival a property of there being more than one mesh, rather than a property of somebody
|
||||
having remembered to take a copy. A mesh that has installed from that name holds the source
|
||||
afterwards, so every installation adds a place the name could point.
|
||||
|
||||
**Genesis names a commit and checks what it got.** It does not clone whatever a branch happens to
|
||||
point at. The forge a mesh installs from is the trust anchor for everything that mesh will ever
|
||||
run, and a branch is a moving target that somebody else controls.
|
||||
|
||||
*This is not hypothetical.* On 2026-09-11 the forge that would serve this role was running a
|
||||
cryptominer, and its git operations were being tampered with in flight — output injected into the
|
||||
protocol stream by a hook that fired on every fetch. Nothing was altered: the repositories were
|
||||
verified against local copies and found byte-identical. But a mesh installing from that name during
|
||||
those hours had no way to establish that for itself, and would have had none.
|
||||
|
||||
## Consequences
|
||||
|
||||
The init builder needs a name it can resolve and a commit it can verify, and nothing else. It does
|
||||
not need to know which mesh answers.
|
||||
|
||||
Whoever operates the mesh that name points at carries a responsibility to everyone installing from
|
||||
it, and should know that. It is not merely a convenience host.
|
||||
|
||||
A mesh that cannot reach any forge cannot be raised. That is a real limit and it is accepted: the
|
||||
alternative is carrying the whole source in the installer, which makes the installer a release
|
||||
artifact that goes stale rather than a program that fetches what it was told to.
|
||||
|
||||
## Open — what relationship a mesh keeps afterwards
|
||||
|
||||
**Not decided, and named here so it is not decided by accident** by whoever writes the init
|
||||
builder. Two shapes, and they are meaningfully different:
|
||||
|
||||
**A snapshot, and then independence.** A mesh installs once, mirrors the source into its own forge,
|
||||
and has no upstream afterwards. It is fully self-hosted, in the sense that nothing it needs lives
|
||||
anywhere else. Updates are then something an operator does deliberately, by pulling changes in —
|
||||
tooling for which is possible and is not a priority.
|
||||
|
||||
**A continuing upstream for core modules**, the way a distribution serves packages and a separate
|
||||
collection serves everything else. A mesh keeps looking at the origin for the modules that make a
|
||||
mesh a mesh, and holds its own for the rest.
|
||||
|
||||
The first is more obviously aligned with the rest of this design, which is arranged so nothing a
|
||||
mesh needs depends on somebody else continuing to host it. The second is more convenient and makes
|
||||
a security problem in one forge everybody's problem. Neither is chosen here.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| Genesis needs only a name and a commit | A mesh is raised with the name pointed at a different mesh than the last time, and the result is identical. |
|
||||
| What was cloned is what was asked for | Genesis is pointed at a commit and refuses a forge serving different content under it, rather than building what it received. |
|
||||
| Losing the serving mesh is survivable | The name is repointed at a mesh that installed from it earlier, and a raise succeeds. |
|
||||
@@ -1,86 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-12
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md
|
||||
---
|
||||
|
||||
# 72. Two graphs, and a build chain that orders itself
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0070](0070-the-catalogue-owns-the-module-graph.md) gave the catalogue the module graph and
|
||||
said the control plane consumes it — that resolving a requirement and composing what a machine
|
||||
should be *"both need to know what modules are, so the dependency runs from the control plane to
|
||||
the catalogue."*
|
||||
|
||||
**That paragraph is wrong, and this record corrects it.** It was written before the two graphs had
|
||||
been told apart, and it creates a dependency that does not need to exist: a catalogue that is down
|
||||
would leave the control plane unable to compose the declaration that would repair it.
|
||||
|
||||
## Decision
|
||||
|
||||
**There are two graphs, with different owners, and they meet only when something is installed.**
|
||||
|
||||
| Graph | Owner | What it links | Answers |
|
||||
|---|---|---|---|
|
||||
| the module graph | the **catalogue** | module-versions to each other | what was this built against · what must rebuild now · what does this need |
|
||||
| the runtime graph | the **control plane** | module-versions to nodes | what runs where · who consumes this provision · what breaks if this machine goes |
|
||||
|
||||
The catalogue does not know nodes exist. The control plane holds module-versions and nodes, along
|
||||
with capabilities and claims, because deciding whether a machine qualifies needs every node.
|
||||
|
||||
**So the control plane never asks the catalogue anything.** It holds what it needs to compose a
|
||||
declaration. A catalogue that is down stops new installs and stops the rebuild fan-out, and does not
|
||||
touch anything already running or the ability to repair it.
|
||||
|
||||
**Everything between them travels as events over the broker**, like all module-to-module
|
||||
communication. The builder finishes and announces that a module was built. The catalogue registers
|
||||
it, places it among what it depends on, and announces that a module was upgraded. The control plane
|
||||
reacts to *that*, not to build output — a semantic fact rather than an artifact.
|
||||
|
||||
**Nothing is lost if a receiver is down.** A consuming module's queue is durable with a dead-letter
|
||||
exchange, declared by the mesh rather than by the module, so an event waits for a consumer that is
|
||||
not there.
|
||||
|
||||
**Build order is not computed. It emerges from the chain.** The builder never consults the graph: it
|
||||
builds what it is asked for, one at a time. The catalogue asks for the next build after the previous
|
||||
registration, so *"do not start this until that is registered"* holds by construction rather than by
|
||||
a schedule somebody maintains.
|
||||
|
||||
**The catalogue's rule is a condition, not an order:** ask for a module to be rebuilt once everything
|
||||
it was built against is current. That handles a chain and a diamond with one rule, where an
|
||||
order-based approach needs to know the shape in advance.
|
||||
|
||||
**Two refusals belong to the catalogue.** A cycle, because the chain would never settle. And a
|
||||
rebuild whose artifacts are identical to what it replaced, which is not an upgrade and must not be
|
||||
announced as one — or a single change ripples outward forever through modules that did not change.
|
||||
|
||||
**Genesis does none of this.** The init builder has a fixed, short list — control plane, catalogue,
|
||||
builder — in a written order, because there is no catalogue yet to ask.
|
||||
|
||||
## Consequences
|
||||
|
||||
The builder stays simple, and independent of the catalogue. That is what makes genesis possible at
|
||||
all: the thing that builds the catalogue cannot require the catalogue.
|
||||
|
||||
The two sides can be briefly out of step — the catalogue may hold a module-version a moment before
|
||||
the control plane knows of it. Assigning in that instant fails, and should say why rather than
|
||||
report that no such module exists.
|
||||
|
||||
A module's declared events stop being documentation and become its permissions: an account is
|
||||
scoped from what a module emits and consumes, so a builder that announces what it built is granted
|
||||
what it needs by the ordinary mechanism rather than by a special case.
|
||||
|
||||
## Open
|
||||
|
||||
**Whether an upgrade is applied or merely noticed.** Today the system this replaces deploys
|
||||
automatically, and that is a defensible default for core modules on a mesh its operator runs. But
|
||||
the design as it stands does the opposite: it records that the source moved ahead, makes it visible,
|
||||
and waits to be told. This must become a setting with a chosen default rather than inherited
|
||||
behaviour — and the choice matters most on the day a bad commit reaches something that carries mail.
|
||||
|
||||
**Whether a module assigned to several machines upgrades on all of them at once.** Doing so makes
|
||||
one bad commit simultaneous everywhere. Doing one machine and pausing turns it into one casualty.
|
||||
@@ -1,94 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-13
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0070-the-catalogue-owns-the-module-graph.md
|
||||
---
|
||||
|
||||
# 73. The installer carries a builder, and the registry stays where it is
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0070](0070-the-catalogue-owns-the-module-graph.md) decided that genesis builds rather than
|
||||
carries, and [ADR 0071](0071-where-genesis-gets-its-source.md) settled where it clones from. Two
|
||||
questions were left open, and the design record names them as the one gap that stops a fresh mesh
|
||||
from being able to produce anything at all: **how the builder arrives**, and **what it publishes
|
||||
into**.
|
||||
|
||||
Today the installer carries the control plane's image inside itself. That works, and it is why
|
||||
genesis needs no registry: nothing is ever fetched, because the one image that matters is already
|
||||
present. The cost is that the mesh which results holds an artifact it did not make, cannot rebuild,
|
||||
and knows nothing about — no version, no source, no edges. That is the same shape as the fault
|
||||
[issue 044](../04-ISSUES/044-the-runtime-every-module-builds-on-cannot-be-built-by-the-mesh/00-report.md)
|
||||
recorded for the shared runtime, and fixing it there while shipping it here on every new mesh would
|
||||
be a strange place to stop.
|
||||
|
||||
## Decision
|
||||
|
||||
**The installer carries a builder, and nothing else.** One artifact, not a growing set. It clones
|
||||
the source at a named commit, checks what it got ([ADR 0071](0071-where-genesis-gets-its-source.md)),
|
||||
and produces the control plane from the same repository and path that any later rebuild of it would
|
||||
use. What raises the mesh is therefore the same thing that will maintain it, and there is no second
|
||||
mechanism kept in step with the first.
|
||||
|
||||
**The registry does not move, and the argument for moving it does not survive being made.**
|
||||
|
||||
It was put this way: a produced image has to be put somewhere before anything can fetch it, so the
|
||||
registry must now precede the control plane, and
|
||||
[ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md)'s answer to that question has to flip.
|
||||
|
||||
It does not, because the premise is false. **The thing that builds the image and the machine that
|
||||
runs it are the same machine.** A built image is already in that machine's container runtime, and
|
||||
the temporary control plane names it exactly as it names a carried one — by the digest of its own
|
||||
configuration, a local identity that requires nothing to have served it. Building changes where the
|
||||
bytes came from. It does not change where they are.
|
||||
|
||||
| | is it substrate? | must it precede the control plane? |
|
||||
|---|---|---|
|
||||
| the store | yes | yes — there is nowhere else to put the control plane's state |
|
||||
| the broker | yes | yes — the control plane reaches a machine only over it |
|
||||
| the image registry | yes — it cannot grant itself a repository | **still no** — the first machine neither fetches the control plane nor needs to, whether the image was carried in or made here |
|
||||
|
||||
So the registry stays where [ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md) put it:
|
||||
substrate by role, ordinary by delivery, installed by the temporary control plane as its first act.
|
||||
The bundle carries two services and a control plane, as it did. **What publishes into the registry
|
||||
is unchanged too** — the existing step that pushes the control plane's image into it, which is the
|
||||
moment that image first receives a digest assigned by something other than itself. It now pushes
|
||||
something this mesh built rather than something it was handed.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Genesis gains one step and changes no others.** A build happens before the image is loaded. The
|
||||
pivot described in [ADR 0067](0067-genesis-is-a-pivot.md) survives exactly as written, because the
|
||||
step it pivots on never cared where the image came from.
|
||||
|
||||
**A fresh mesh can produce from the moment it exists.** The builder is present before the control
|
||||
plane is, so the core modules, the catalogue and the builder's own module can be built in the
|
||||
ordinary way rather than waiting for somebody to carry them in. The paragraphs in
|
||||
[`17-raising-a-mesh`](../03-DESIGN/01-to-be/17-raising-a-mesh.md) that describe this were describing
|
||||
something that could not start; they can start now.
|
||||
|
||||
**Genesis needs more of the outside world.** Carrying an image needed nothing but the installer.
|
||||
Building one needs the source, and whatever the build itself reaches for. This is a real cost and
|
||||
is not waved away: it makes genesis fail in more ways, all of them at a step that says what it was
|
||||
doing. It is accepted because the alternative is a mesh that cannot rebuild its own control plane,
|
||||
which fails in exactly one way, silently, later, and for ever.
|
||||
|
||||
**A pre-built bundle remains possible and is not this.** Nothing here forbids delivering artifacts
|
||||
rather than building them; it fixes where they may come from. A bundle of pre-built core modules is
|
||||
an **export of a mesh that built them**, carrying what the catalogue knows about each alongside the
|
||||
artifact itself — so that loading one leaves the graph in the state building would have left it. A
|
||||
bundle that carries images without that is the thing this decision rejects, whoever ships it.
|
||||
|
||||
## What this does not decide
|
||||
|
||||
**Whether the builder's own module is carried or built.** It builds everything else; what installs
|
||||
*it* as an ordinary module afterwards, so that it too can be upgraded, is the same closed-list
|
||||
question [`12-a-module-repository`](../03-DESIGN/01-to-be/12-a-module-repository.md) already holds,
|
||||
and is unchanged by this.
|
||||
|
||||
**How a machine authenticates to a registry that asks it to.** Genesis raises its own and reaches it
|
||||
over the loopback, so this remains a joining problem
|
||||
([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)).
|
||||
@@ -1,172 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-15
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0039-what-the-sdk-holds-and-refuses.md
|
||||
---
|
||||
|
||||
# 74. The mesh defines a module protocol; an SDK is an implementation of it
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0039](0039-what-the-sdk-holds-and-refuses.md) says what the SDK holds: the tool-serving
|
||||
harness, the messaging and event framework, the contracts, and core primitives. It settles what
|
||||
belongs in *an* SDK. It does not say what happens when there is more than one.
|
||||
|
||||
There is already more than one. **The contracts are expressed twice** — as Go types in the control
|
||||
plane and the host, and as TypeScript types in the SDK — and nobody has felt it because both live
|
||||
in one repository and one head.
|
||||
|
||||
**A correction, made after inspecting the wire rather than the types** (2026-09-16). This record
|
||||
first claimed the two implementations already disagreed — `resource` vs `Provision`, `consumer`
|
||||
meaning the module in one and the node in the other, headers declared on one side and emitted by
|
||||
neither. **On inspection the live wire agrees**, and the claim was wrong:
|
||||
|
||||
- The grant types that disagreed (`Grant`, `Interface`, `Credential` in the SDK's `contracts`)
|
||||
were **dead** — exported and imported by nothing. The live provisioning wire is the contributions
|
||||
file, whose shape (`as`, `secret`, `node`, `at`, `values`) is the same on both sides. Those dead
|
||||
types have been removed.
|
||||
- The envelope agrees too: Go emits all five required headers, and `x-causation-id`/`x-schema` are
|
||||
**optional** — the SDK sets them when a handler has a causation or a schema, and a bare event
|
||||
carrying neither is correct, not a drift.
|
||||
|
||||
So the danger was never live disagreement. It was **dead types that contradicted the live wire**,
|
||||
which read as the contract and were not — and are exactly what led this record to assert a drift
|
||||
that inspection did not find. That is a sharper reason for the decision below, not a weaker one: a
|
||||
type is only as good as its being the wire, and the way to guarantee that is to specify the wire and
|
||||
check implementations against it, rather than to trust a hand-kept type to still describe it.
|
||||
|
||||
A failure of this kind does not announce itself. Two implementations that disagree about an
|
||||
envelope do not fail to compile — they ignore each other's messages, and a mesh where a module
|
||||
stops reacting looks exactly like a mesh where nothing happened.
|
||||
|
||||
## The question this settles
|
||||
|
||||
A module may be written in any language the mesh can build
|
||||
([`18-building-a-module`](../03-DESIGN/01-to-be/18-building-a-module.md)). Every language needs an
|
||||
SDK. What is an SDK *of*?
|
||||
|
||||
Two answers were available, and the obvious one is wrong.
|
||||
|
||||
**Shared types, generated.** Write the shapes once — a schema, an IDL — and generate Go, TypeScript,
|
||||
Rust. It is the familiar answer and it solves the smaller half of the problem. The shapes are not
|
||||
where the difficulty is.
|
||||
|
||||
**A specified wire, with a conformance suite.** The shapes are a consequence; what an SDK must get
|
||||
right is *behaviour*.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh defines a module protocol. An SDK is an implementation of that protocol in one
|
||||
language, and nothing more.**
|
||||
|
||||
That is the whole of what an SDK is. Not a library a language happens to have, not a convenience
|
||||
layer, not a place for helpers to accumulate — an implementation of a specified protocol, finished
|
||||
when it implements it and correct when it agrees with every other implementation.
|
||||
|
||||
### The protocol is split per capability
|
||||
|
||||
**A module does not use all of it, so an SDK need not implement all of it.** A module that only
|
||||
consumes events uses the event capability. One that serves tools uses the tool capability. A
|
||||
provider uses provisioning. Nothing about consuming an event requires knowing how a grant is
|
||||
answered.
|
||||
|
||||
So the protocol is a floor plus capabilities:
|
||||
|
||||
| part | what it covers | who needs it |
|
||||
|---|---|---|
|
||||
| **connection** — the floor | reading the sealed credential, pinning the certificate fingerprint, taking identity from the credential rather than the environment | everything |
|
||||
| **events** | the envelope and its headers, the durable per-consumer queue, binding, at-least-once with dedup on `x-event-id` | a module that emits or consumes |
|
||||
| **tools** | registration, the shared durable `serve.<key>` queue, request and reply | a module with a surface |
|
||||
| **provisioning** | a grant in, a credential out, and what each carries | a module that provides something |
|
||||
|
||||
**This is the same shape the host already has.** A host declares which resource kinds it can apply,
|
||||
and a partial host — one that can write files and run things but not manage users or containers —
|
||||
is a real thing rather than a broken one ([ADR 0005](0005-the-node-host.md)). An SDK that implements
|
||||
the floor and events is exactly as legitimate, and a module written against it is a module that
|
||||
does events.
|
||||
|
||||
**So a language arrives in pieces rather than all at once.** A Rust SDK implementing connection and
|
||||
events is useful the day it exists; tools and provisioning follow when something needs them. The
|
||||
alternative — a language is unsupported until it is entirely supported — is what makes adding one a
|
||||
project rather than a contribution.
|
||||
|
||||
**And what a language can be used for is then a fact the mesh can state**, rather than something an
|
||||
author discovers by writing a module that cannot be built: the toolchain list says which languages
|
||||
exist, and the conformance results say what each can do.
|
||||
|
||||
### What the specification covers
|
||||
|
||||
Per capability, what two implementations can disagree about:
|
||||
|
||||
- **the exchanges and queues** — which exchanges exist, that a consumer's queue is durable and
|
||||
named `<node>.<module>.events`, that a tool is served from a shared durable `serve.<key>`
|
||||
- **the envelope** — every header, which are required, what an unknown `x-` header means, and that
|
||||
ignoring one is correct rather than lax
|
||||
- **identity** — that a module's node and module name come from its sealed credential and not from
|
||||
its environment, so what it emits matches what the mesh authorised
|
||||
- **delivery** — at-least-once, and that dedup is on `x-event-id`, which only the emitter can make
|
||||
- **the credential** — the sealed document's fields, and that a connection pins a certificate
|
||||
fingerprint rather than trusting an authority
|
||||
- **provisioning** — a grant in, a credential out, and what each carries
|
||||
- **the vocabulary** — that `consumer` is one thing, named once
|
||||
|
||||
### Conformance is per capability
|
||||
|
||||
**A suite per part, and an SDK claims the parts it passes.** A monolithic pass/fail would make a
|
||||
partial implementation indistinguishable from a broken one, which is the distinction this is built
|
||||
on.
|
||||
|
||||
**And the suite is executable, not prose.** A specification nobody can run is a document two
|
||||
implementations drift from while both believe they conform. Conformance is a set of fixtures — an
|
||||
emitted event, a served tool call, a grant and its answer — that every SDK must produce and consume
|
||||
byte-for-byte.
|
||||
|
||||
**The existing two implementations are the first two to be made to pass it.** Not a future language:
|
||||
the drift above is present, and a suite that only new SDKs must satisfy would leave the disagreement
|
||||
that already exists in place while certifying everything added afterwards against it.
|
||||
|
||||
## Why not generated types
|
||||
|
||||
Generation makes the shapes agree and leaves everything that matters unspecified. Two SDKs
|
||||
generated from one schema can still name their queues differently, take identity from the
|
||||
environment, dedup on the wrong field, or omit a header the other requires — and every one of those
|
||||
is a mesh that runs and quietly does not work.
|
||||
|
||||
It also makes the contract into whatever the generator supports, which is a decision nobody made
|
||||
about a boundary everything else depends on.
|
||||
|
||||
**The shapes are worth generating once the wire is specified.** That is a convenience, and it comes
|
||||
second.
|
||||
|
||||
## Consequences
|
||||
|
||||
**A language is a commitment, and now a divisible one.** Adding one means implementing the protocol
|
||||
and passing the suites for the parts it claims. That is more work than transliterating types, and
|
||||
it is the work that was always there — the difference is that it can be finished, and finished in
|
||||
pieces, rather than believed.
|
||||
|
||||
**Versioning becomes possible.** `x-schema` exists for it and is never written. A specified envelope
|
||||
with a version on the body is what lets a mesh hold a module built against an older SDK, which is
|
||||
the ordinary state of any mesh that has been running for a while.
|
||||
|
||||
**The two current implementations agree on the live wire** — inspection showed it. What was wrong
|
||||
was a set of dead types beside the wire, now removed. The suite's job here is therefore prevention:
|
||||
to keep that agreement true as the wire changes, and to hold a new language's SDK to it, rather than
|
||||
to repair a break that exists today.
|
||||
|
||||
**This does not make the mesh polyglot by itself**, and should not be reported as though it does. It
|
||||
makes polyglot possible to do correctly. A Rust SDK is still a Rust SDK.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| One vocabulary | A word means one thing across implementations, checked by the fixtures using it. |
|
||||
| The wire is what is specified | Both existing SDKs run the conformance suite in their own test suites, and a change to one that breaks a fixture fails there rather than in a mesh. |
|
||||
| A new SDK is a passing SDK | A language is not listed as buildable for a capability until its SDK passes that capability's suite; the toolchain list and the conformance results name the same set. |
|
||||
| A partial SDK is a real thing | An SDK implementing the floor and one capability passes, is listed for that capability, and a module using another is refused with the reason — rather than failing at runtime in a language nobody said was finished. |
|
||||
| An unknown header is ignored | A fixture carries one, and every implementation accepts it. |
|
||||
| Identity comes from the credential | A fixture sets an environment that disagrees with the credential, and the emitted event carries the credential's. |
|
||||
@@ -1,129 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-15
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0014-no-npm-workspace.md
|
||||
---
|
||||
|
||||
# 75. An artifact store is a provision; a package registry is a different one
|
||||
|
||||
## Context
|
||||
|
||||
Two questions have been circling, and they turn out to be one question asked twice.
|
||||
|
||||
**"Should gitea be the mesh's registry?"** It serves OCI images and a dozen package ecosystems, it
|
||||
is already needed — genesis clones from one — and the mesh's own registry has neither
|
||||
authentication ([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md))
|
||||
nor a transport a runtime will accept over a network
|
||||
([issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)). Gitea
|
||||
has both.
|
||||
|
||||
**"Where does the SDK come from?"** [ADR 0014](0014-no-npm-workspace.md) already answers it — each
|
||||
module consumes its dependencies, the mesh's own shared library included, *from the private
|
||||
registry* — and nothing installs one, so today it comes from a git URL, which is
|
||||
[issue 053](../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md).
|
||||
|
||||
**The framing that dissolves both:** `artifact-store` is already a provision, and `registry`
|
||||
already provides it. So "should gitea be the registry" is not a question about replacing a
|
||||
component. It is a question about **a second provider of an existing provision** — which this mesh
|
||||
has a mechanism for, and uses for certificate authorities and VPNs already.
|
||||
|
||||
## Decision
|
||||
|
||||
**Two provisions, because they are two jobs.**
|
||||
|
||||
| provision | is | for |
|
||||
|---|---|---|
|
||||
| `artifact-store` | content-addressed blobs, pinned by digest, no versions, no ranges | what the **mesh** delivers to **machines** |
|
||||
| `package-registry` | an ecosystem's own registry — npm, cargo, PyPI, Go | what **code** resolves when it is compiled |
|
||||
|
||||
They are not the same store with different clients. One is addressed by digest and immutable by
|
||||
construction; the other is addressed by name and version, and resolves ranges. Conflating them is
|
||||
how a mesh that pins everything ends up rebuilding one commit into two different things.
|
||||
|
||||
**`registry` remains the provider genesis installs.** Not because it is better, but because of what
|
||||
it is: a directory and one container, no database, no control plane, installable at step 8 of an
|
||||
install where neither exists yet. Gitea needs a store and provisioning, which means a control plane,
|
||||
which means the pivot has already happened — and the pivot needs somewhere to publish to.
|
||||
|
||||
**Gitea also provides `artifact-store`, and a mesh may choose it.** Two providers of one provision
|
||||
is a thing the mesh understands: it refuses, names both, and choosing is assigning the one you want.
|
||||
|
||||
**And it does not claim `the-artifact-store`.** That claim is node-scoped, so a module holding it
|
||||
cannot share a machine with another that does — and a machine running gitea for git and packages
|
||||
*alongside* a registry serving artifacts is an ordinary arrangement, not a conflict. They are
|
||||
different ports doing different jobs.
|
||||
|
||||
The exclusivity that matters is mesh-wide and is already expressed: `provides` at mesh scope means
|
||||
two providers are two answers, and the resolver refuses until one is assigned. Forbidding
|
||||
co-residence adds nothing to that and forbids something reasonable. **Whether `registry` should
|
||||
still hold that claim is left open here** — it may be protecting something about the port or the
|
||||
data directory that is not written down, and removing a claim is not a thing to do from the outside
|
||||
of a manifest.
|
||||
A mesh that assigns gitea gets authentication and TLS for its artifacts — which is to say, **issues
|
||||
042 and 048 are answered by choosing a provider that already solved them**, rather than by
|
||||
reimplementing accounts and certificates in a registry that has none.
|
||||
|
||||
**Gitea provides `package-registry`.** That is ADR 0014's private registry, and it is one service
|
||||
rather than one per ecosystem. `verdaccio` may provide it too, for npm alone, and is then a choice
|
||||
somebody makes rather than the answer.
|
||||
|
||||
**The registry is not removed at the end of installing.** A mesh that never runs gitea still has an
|
||||
artifact store. Retiring it is a migration a mesh performs, not a step an installation ends with.
|
||||
|
||||
## Why not simply gitea, from the start
|
||||
|
||||
Because genesis would need a control plane before the thing that stores the control plane's image,
|
||||
and that is circular rather than merely awkward. It would also make one of the three things the
|
||||
build loop cannot produce for itself into a stateful application with a database — the pivot is the
|
||||
hardest part of this design already.
|
||||
|
||||
And it puts every artifact in the service that is also the trust anchor for everything the mesh will
|
||||
ever run ([ADR 0071](0071-where-genesis-gets-its-source.md)), which records that forge serving a
|
||||
cryptominer with tampered git operations. Two blast radii are better than one.
|
||||
|
||||
## Moving from one provider to the other is a designed act
|
||||
|
||||
**Not a removal.** Every image a machine runs is pinned to a digest at a named store, the control
|
||||
plane's own included. Changing the provider means:
|
||||
|
||||
1. gitea installed, reachable, and holding an account the builder may publish with
|
||||
2. every artifact mirrored
|
||||
3. every declaration re-pinned, the control plane's **last**, because it is what performs the others
|
||||
4. **every machine verified to have converged and to be able to pull from the new store**
|
||||
5. only then the old provider unassigned, and its volume kept ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md))
|
||||
|
||||
Step 4 is the one that is easy to skip and the only thing between this and a mesh that cannot
|
||||
restart its own control plane. A machine that reboots mid-migration pulls from a store that no
|
||||
longer exists, and a local image cache hides that until exactly the moment it matters.
|
||||
|
||||
## Consequences
|
||||
|
||||
**The bootstrap is unchanged**, which is the point of keeping the small provider.
|
||||
|
||||
**042 and 048 gain a second answer.** They can be fixed in the registry, or dissolved by choosing a
|
||||
provider that already has accounts and TLS. The second is less work and more service.
|
||||
|
||||
**ADR 0014 becomes satisfiable.** There is a provision for the private registry, something that
|
||||
provides it, and a module may depend on it — so the SDK can be published and consumed rather than
|
||||
cloned, and issue 053 has somewhere to go.
|
||||
|
||||
**A mesh can be minimal or complete, and both are legitimate.** One with the small registry and no
|
||||
gitea builds and runs modules and cannot serve packages. That is a real configuration, not a broken
|
||||
one — the same way a partial host is real.
|
||||
|
||||
**And the bootstrap still has no package registry.** The first build of the shared base happens
|
||||
before anything has installed one. That is the same pivot as everything else and it is **not solved
|
||||
here**: it is named, so the next person does not discover it.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| Genesis needs no database | The installer raises a mesh of one on a machine with nothing, and the artifact store it installs has no store of its own. |
|
||||
| Two providers are a choice, not a conflict | A mesh holding both is asked to resolve `artifact-store` and refuses, naming both, until one is assigned. |
|
||||
| Providers may share a machine | A node is assigned both gitea and a registry, and both run — only one of them answers `artifact-store`. |
|
||||
| The two stores are not interchangeable | A module depending on `package-registry` is not satisfied by `artifact-store`, and the refusal says why. |
|
||||
| A migration is verified before it is finished | The old provider cannot be unassigned while any machine's declaration still names it. |
|
||||
@@ -1,87 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-16
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0075-two-stores-and-which-provides-what.md
|
||||
---
|
||||
|
||||
# 76. The SDK is a published package, and the toolchain resolves it by version
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0014](0014-no-npm-workspace.md) decided a module consumes its dependencies — the mesh's own
|
||||
shared library included — from the private registry. [ADR 0075](0075-two-stores-and-which-provides-what.md)
|
||||
decided the private registry is a `package-registry` provision, and that gitea provides it. What
|
||||
neither settled, and what [issue 053](../04-ISSUES/053-the-sdk-is-pinned-twice-and-the-two-disagree/00-report.md)
|
||||
left open, is the one build where the rule cannot simply be obeyed: **the first one.**
|
||||
|
||||
The TypeScript toolchain image is built *from* the SDK — it carries the SDK so that every module
|
||||
compiled inside it resolves the shared library without each build fetching it. So the thing that
|
||||
compiles TypeScript and the thing that contains the SDK were the same object, and that object
|
||||
cannot be what builds the SDK. Stated as a question — "how does the SDK reach the registry before
|
||||
the toolchain exists, when the toolchain is what builds it?" — it reads as a paradox.
|
||||
|
||||
It is not one. The paradox exists only because the toolchain *bakes a git-cloned copy* of the SDK.
|
||||
The SDK itself is plain TypeScript: it needs `node` and `tsc` and nothing the mesh makes. A public
|
||||
base image can build it. The circularity is a property of the workaround, not of the SDK.
|
||||
|
||||
## Decision
|
||||
|
||||
**The SDK is an ordinary published package in the mesh's `package-registry`, consumed by version.**
|
||||
The git URL in the toolchain's manifest and the sibling-path lock beside it — the two halves of
|
||||
issue 053 — are both removed. A build resolves the SDK the way it resolves any dependency, with a
|
||||
lock that agrees with its manifest, so `npm ci` is the command and reproducibility is by
|
||||
construction rather than by the machine the build ran on.
|
||||
|
||||
**The SDK is built with a public base image, not with the mesh's toolchain.** It is *not* one of
|
||||
the components the loop cannot build — the control plane, the registry, the builder, the catalogue
|
||||
([`12-a-module-repository`](../03-DESIGN/01-to-be/12-a-module-repository.md)), which arrive by
|
||||
carrying an init builder because they are the loop's own machinery. The SDK is machinery for
|
||||
nothing; it is an ordinary dependency the loop builds and publishes like any other. The only
|
||||
constraint is narrow: it cannot be compiled *in the mesh toolchain*, because that toolchain is built
|
||||
from it. So it is compiled on a public base image instead — which needs nothing the mesh makes — and
|
||||
published before the toolchain that consumes it. It is not carried, because building it does not
|
||||
wait on a mesh existing first.
|
||||
|
||||
**The toolchain base stays, thinned.** mesh-tools remains the image bundles are compiled in and the
|
||||
one place the SDK is resolved — but it `npm ci`s the SDK by version from the registry instead of
|
||||
baking a copy cloned from a git URL. Bundles keep borrowing its resolved dependencies; what changes
|
||||
is that the version they borrow is named and honest. This was the shape chosen over dropping the
|
||||
shared base entirely and having every bundle resolve the SDK itself: one resolution point, one
|
||||
place to be right about the version.
|
||||
|
||||
**Genesis orders the publish before the first compile.** The package-registry provider is a public
|
||||
image (gitea), so it comes up needing no toolchain; the SDK is published into it; only then is the
|
||||
toolchain built, so the first `npm ci` has a registry to read from. Nothing in that chain is
|
||||
circular, because the only thing that needed the toolchain — baking the SDK — is gone.
|
||||
|
||||
## Consequences
|
||||
|
||||
Each language's toolchain repeats the shape: its own SDK, built from that language's public base
|
||||
image, published to the same registry, resolved by version with that ecosystem's lockfile-honest
|
||||
install (`npm ci`, `cargo` against a vendored or registry source, `pip` against a pinned set). The
|
||||
warning in issue 053 — that whatever the TypeScript repository does the others will copy — is
|
||||
answered by making the copied thing the correct one.
|
||||
|
||||
A change to the SDK is publish-then-consume, exactly as [ADR 0014](0014-no-npm-workspace.md) already
|
||||
priced it: publish the new SDK version, then bump the toolchain (and any module pinning it directly)
|
||||
to consume it. There is no shortcut that resolves an unpublished SDK, which is the property that was
|
||||
missing.
|
||||
|
||||
mesh-tools is no longer an SDK carrier in the sense that mattered — it does not contain a copy
|
||||
whose provenance is a branch head somebody force-pushes. It contains a version.
|
||||
|
||||
A mesh with no package-registry cannot build TypeScript. This is accepted and is not new: it is the
|
||||
same shape as a mesh that cannot reach a forge being unable to be raised
|
||||
([ADR 0071](0071-where-genesis-gets-its-source.md)). Installing brings the registry up first.
|
||||
|
||||
## How this is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The SDK a build compiles against is named, not cloned from a branch | The toolchain manifest pins `@novox/mesh-sdk` to a version, and the build runs `npm ci`, which refuses a lock that disagrees with the manifest. Issue 053's two checks become this one. |
|
||||
| The SDK builds without the mesh's own toolchain | The SDK's build recipe names a public base image. A recipe that named the mesh toolchain would reintroduce the cycle and is refused in review. |
|
||||
| The registry is up before the first compile | The genesis bed asserts the package-registry answers, and the SDK is published, before the base build runs. A base build that ran first would fail its `npm ci` with no registry, which is the positive control. |
|
||||
| A second language repeats the shape, not a new one | When a second SDK is added, its recipe is compared to this one: public base, publish by version, lockfile-honest install. |
|
||||
@@ -1,61 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-16
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 77. The parts are named controller, foundation, node — not control plane, substrate, master
|
||||
|
||||
## Context
|
||||
|
||||
The words drifted. In conversation and in code the same thing was called *control plane*,
|
||||
*controller*, *master*, and *hub*; the store-and-broker pair was called *substrate* and
|
||||
*foundation*; a machine was a *node*, a *worker-node*, a *peer*, a *slave*. A mesh named
|
||||
differently by two people is a mesh they describe differently, and the drift was worst on the
|
||||
parts talked about most.
|
||||
|
||||
Three of the terms carried wrong ideas. *Control plane* is borrowed from networking's
|
||||
control-plane/data-plane split and means nothing here. *Master/slave* and *hub/peer* imply a
|
||||
subordinate — but no node is: a node applies its own declaration and keeps running when the
|
||||
control-node dies, so it is as much its own machine as any other. *Substrate* is a biology
|
||||
metaphor that landed for no one.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep the inherited words.** Rejected: they are the source of the drift, and two of them
|
||||
(control plane, substrate) are metaphors that teach the wrong shape to anyone reading them cold.
|
||||
2. **master / slave, or hub / peer, for the nodes.** Rejected: both name a hierarchy the mesh does
|
||||
not have. The control-node owns no other node; lose it and the rest keep running what they were
|
||||
last told.
|
||||
3. **controller / foundation / node + control-node.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
The component that decides what each node should be, holds the mesh's records, and tells nodes is
|
||||
the **controller** — the module `mesh-controller`, which claims the mesh-scoped `the-controller`
|
||||
seat. The store and broker raised at genesis are the **foundation**. Machines are **nodes**;
|
||||
there are 0..n of them, and exactly one — the one running the controller — is the **control-node**.
|
||||
|
||||
Retired: *control plane*, *substrate*, *master/slave*, *hub/peer*, *worker-node*.
|
||||
[`00-META/glossary.md`](../00-META/glossary.md) is the authority, and a new name for an existing
|
||||
thing lands there in the change that introduces it in code.
|
||||
|
||||
## Consequences
|
||||
|
||||
`mesh-control` became `mesh-controller` across the module, container, image, binary, `cmd/` dir and
|
||||
the git repository; `substrate` became `foundation` in the embedded base bundles, the default
|
||||
template and the example lock; the seat `the-control-plane` became `the-controller`. The 03-DESIGN
|
||||
prose and 00-META follow the new words.
|
||||
|
||||
What got harder: the records under `02-DECISIONS/` are immutable, so they keep the words they were
|
||||
written with — this record included, whose own title names what it retires. A term retired here
|
||||
still appears there, and the glossary is how to read it. The git repository on the forge was
|
||||
renamed `mesh-control` → `mesh-controller`.
|
||||
|
||||
## References
|
||||
|
||||
- [`00-META/glossary.md`](../00-META/glossary.md) — one name per thing, and the words retired.
|
||||
- The rename shipped across all six code repositories and hq (main).
|
||||
@@ -1,63 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-16
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0033-the-substrate-is-a-store-and-a-broker.md
|
||||
---
|
||||
|
||||
# 78. The store and the broker are ordinary modules
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0033](0033-the-substrate-is-a-store-and-a-broker.md) settled that the foundation is a store
|
||||
and a broker, raised at genesis; [ADR 0006](0006-the-substrate-and-the-control-plane.md) settled
|
||||
that the controller cannot grant itself either, because it consumes them and is not running yet to
|
||||
ask. Both were raised as bundle resources — plumbing, with no record in the mesh's module graph.
|
||||
|
||||
That left two costs, named in [issue 051](../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md).
|
||||
The foundation's own store and broker could not be upgraded — nothing owned them as modules. And a
|
||||
mesh that wanted a database or a queue for its modules installed the `postgres`/`lavinmq` modules,
|
||||
each of which raised a **second** server: a mesh ran two postgres and two brokers.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave them as bundle-only plumbing.** Rejected: they cannot be upgraded, and the second
|
||||
server stays. The floor keeps a permanent specialty in it.
|
||||
2. **The control-plane pivot verbatim — raise a temporary one, install the module, retire the
|
||||
temporary** ([ADR 0067](0067-genesis-is-a-pivot.md)). Rejected for a *stateful* server: it means
|
||||
a handover with real downtime, tearing down the store the controller is mid-read of.
|
||||
3. **Adopt in place.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
The foundation's store and broker are **adopted in place** as the ordinary `postgres` and `lavinmq`
|
||||
modules. Genesis still raises them first (nothing else can — ADR 0006), then each module declares a
|
||||
container with the **same name, image and spec** the foundation raised, so the applier — which keys
|
||||
on the container name and compares a spec digest — reconciles it rather than raising a second. The
|
||||
credentials are the foundation's, made at genesis and carried in through `secret accept`, because
|
||||
the mesh cannot invent a credential that already made the databases. The servers bind mesh-wide so
|
||||
a consumer on any node can reach the one shared server.
|
||||
|
||||
A mesh runs **one postgres and one lavinmq**, and each is upgradeable through a stated window: the
|
||||
store's is a connection-pool reconnect; the broker's is the harder case of recreating the bus the
|
||||
push travels over, so the mesh reconnects to the one that returns.
|
||||
|
||||
## Consequences
|
||||
|
||||
The twelve-module floor has no specialty left in it — the store and broker are moments in a
|
||||
module's life, not a separate kind of thing. Two follow-ups are tracked:
|
||||
[issue 054](../04-ISSUES/054-the-adopted-store-and-broker-are-open-before-the-filter/00-report.md)
|
||||
(the adopted servers bind `0.0.0.0` before the packet filter is installed) and
|
||||
[issue 055](../04-ISSUES/055-the-adopted-store-and-broker-may-be-reachable-on-one-node-only/00-report.md)
|
||||
(whether a consumer on another node reaches them over the overlay).
|
||||
|
||||
What got harder: a foundation upgrade recreates the very server the controller reads from, or the
|
||||
bus the instruction to upgrade travels over — a window that a stateless module upgrade does not have.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 051](../04-ISSUES/051-the-mesh-cannot-update-what-it-depends-on/00-report.md) — the gap this closes.
|
||||
- [`03-DESIGN/01-to-be/07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md) — the amended design.
|
||||
- Shipped across mesh-host, mesh-catalog and mesh-lab (main); proven 22/22 in the one-node lab.
|
||||
@@ -1,63 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0078-the-store-and-broker-are-modules.md
|
||||
---
|
||||
|
||||
# 79. The foundation seats are named after their servers
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0078](0078-the-store-and-broker-are-modules.md) settled that the store and broker are the
|
||||
ordinary `postgres` and `lavinmq` modules, adopted in place on the control-node, and that a mesh
|
||||
runs **one postgres and one lavinmq**. But that singularity held only by convention: genesis
|
||||
assigns them to the control-node alone. [Issue 056](../04-ISSUES/056-an-adopted-module-assigned-to-a-second-node-raises-a-second-server/00-report.md)
|
||||
recorded the gap — a second `assign` to another node finds no container of that name there and
|
||||
raises a *second* server, holding none of the first's data, and nothing refuses it.
|
||||
|
||||
The mesh already has the mechanism for "there is one of me": a mesh-scoped exclusive **seat**.
|
||||
[ADR 0077](0077-the-controller-and-the-foundation.md) gave the controller one, which it named
|
||||
`the-controller`. Nothing gave the store and broker theirs.
|
||||
|
||||
## Decision
|
||||
|
||||
Each foundation module claims a mesh-scoped seat named **after the server it guards**:
|
||||
`mesh-controller`, `mesh-store`, `mesh-broker`. So the `postgres` module claims `mesh-store` and
|
||||
the `lavinmq` module claims `mesh-broker`; the resolver refuses a second holder mesh-wide, the same
|
||||
way it keeps one hub and one controller. A second `assign` is now a refusal at resolution — *one
|
||||
per mesh* — not a silent second server.
|
||||
|
||||
The controller's seat, which [ADR 0077](0077-the-controller-and-the-foundation.md) named
|
||||
`the-controller`, is renamed `mesh-controller` under this same rule, so all three foundation seats
|
||||
follow one convention: the seat is the server. (The module `mesh-controller` and its seat now share
|
||||
a name, which is the point — there is one of that server, and the seat says so.)
|
||||
|
||||
This does not tie a foundation module to the control-node — a mesh-scoped seat forbids a *second*
|
||||
holder, not a wrong single one. Adoption still requires the container to already be running where
|
||||
the module lands, which genesis arranges; the seat closes the "two servers" gap, and the
|
||||
control-node convention remains what puts the one holder in the right place.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- The resolver's `checkClaims` refuses two holders of a mesh-scoped seat
|
||||
(`mesh-controller/internal/catalogue/resolve.go`).
|
||||
- `TestAFoundationModuleCannotBeRaisedOnASecondNode` asserts each foundation module's second
|
||||
assignment is refused with *one per mesh*.
|
||||
- The `postgres`, `lavinmq` and `mesh-controller` manifests declare the seat.
|
||||
|
||||
## Consequences
|
||||
|
||||
"One store, one broker, one controller" is now a property the mesh enforces rather than a
|
||||
convention it hopes for. The silent operational edge that remains — a cross-node consumer
|
||||
provisioned only when the provider is pushed again — is separate, and tracked as
|
||||
[issue 057](../04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md).
|
||||
|
||||
## References
|
||||
|
||||
- [issue 056](../04-ISSUES/056-an-adopted-module-assigned-to-a-second-node-raises-a-second-server/00-report.md) — the gap this closes.
|
||||
- [ADR 0077](0077-the-controller-and-the-foundation.md) — named the controller's seat, here renamed.
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — asserted one postgres and one lavinmq, here enforced.
|
||||
- [`00-META/glossary.md`](../00-META/glossary.md) — seat and claim.
|
||||
@@ -1,79 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0019-how-this-repository-works.md
|
||||
---
|
||||
|
||||
# 80. The development cycle is checked, not trusted
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0019](0019-how-this-repository-works.md) made this repository the source of truth, and the
|
||||
process overview drew the flow work must follow: an idea or a symptom, a decision, a to-be design,
|
||||
a build in a code repository, an as-is update on shipping. The playbooks describe every step, and
|
||||
frontmatter carries every status.
|
||||
|
||||
But the flow itself was enforced by nothing. A design could appear citing no decision; a design
|
||||
could sit `in-progress` naming no code; an issue could be `fixed` by nobody knows what. Each is
|
||||
indistinguishable from correct work until somebody reads carefully — and the whole point of the
|
||||
playbooks is that nobody should have to hold this repository in their head. A session that starts
|
||||
cold (or an agent after a context clear) must be able to *find* the chain by following frontmatter
|
||||
pointers, which only works if the pointers are reliably there.
|
||||
|
||||
## Decision
|
||||
|
||||
The development cycle is enforced mechanically, to the extent frontmatter can carry it:
|
||||
|
||||
- **No design without a decision** — every to-be design names at least one record in `decisions:`.
|
||||
- **No development without a design that says where** — an `in-progress` or `implemented` design
|
||||
names its owning code in `code:`.
|
||||
- **No owner-less diagnosis, no fix-less fix** — an issue marked `located` or `fixed` names
|
||||
`located-in:`; one marked `fixed` or `resolved` says `fixed-by:` (prose counts — "nothing, the
|
||||
capability existed" is an answer).
|
||||
- **No silent graduation** — a `graduated` research overview says what it `became:`, and the
|
||||
targets exist.
|
||||
- **No two records answering to one number** — added 2026-09-30; see the insight below.
|
||||
|
||||
> **Progressive insight — 2026-09-30.** The list above named four things `cycle.py` enforces, and
|
||||
> now names five. Nothing enforced that two issue records hold different numbers: two machines
|
||||
> filing issues within one hour both read `main`, both took "the next free number", and collided
|
||||
> twice — the second collision reaching `main` with `records.py`, `cycle.py` and `index.py` all
|
||||
> reporting success (04-ISSUES/155). A number is how every other record cites one, so two records
|
||||
> answering to it is a citation that resolves to whichever folder the reader opened. `cycle.py`
|
||||
> refuses it now. The decision here stands exactly as written: this is one more thing frontmatter
|
||||
> and file names can carry, found by its absence rather than by reasoning.
|
||||
|
||||
> **Progressive insight — 2026-10-10.** The third rule above, no owner-less diagnosis and no fix-less
|
||||
> fix, and the replay rule `cycle.py` took on under ADR 0237, moved out of `cycle.py` the day slice 3 of
|
||||
> [ADR 0275](0275-the-mesh-numbers-and-tracks-its-issues-in-a-module-of-its-own-and-hq-keeps-only-the-narrative-of-a-design-or-governance-issue.md)
|
||||
> landed: an issue's status lives in the issue tracker, whose `move` refuses at the write what the
|
||||
> check refused at merge — `located` without an owner, `resolved` without a fix, a core issue without
|
||||
> its replay, `wontfix` without a reason. Enforced mechanically still, one step earlier, in the one
|
||||
> place the status is. `cycle.py` keeps the fifth rule, no two records answering to one number, and
|
||||
> refuses a status key in a report opened after the import. The decision stands: the cycle is checked,
|
||||
> not trusted; which program checks each rule is a fact that moved.
|
||||
|
||||
> **Progressive insight — 2026-10-10, later.** `cycle.py` refuses a status key in any report, not only one
|
||||
> opened after the import: the keys the older reports carried were removed once the import was checked
|
||||
> against every folder (ADR 0275's last note).
|
||||
|
||||
[`00-META/checks/cycle.py`](../00-META/checks/cycle.py) refuses violations, beside `records.py`
|
||||
and `index.py`; all three run before any merge here. What frontmatter cannot see — that code work
|
||||
actually started from a handoff — remains held by playbooks 04 and 07: a feature branch exists
|
||||
because a design or an issue sent it, and a merge is a human checkpoint.
|
||||
|
||||
## Consequences
|
||||
|
||||
A `/clear` costs little: [`AGENTS.md`](../AGENTS.md) now carries the cycle and a where-to-look
|
||||
table, and the chain a fresh session needs is guaranteed present in frontmatter rather than
|
||||
reconstructed from memory. The checks are the floor, not the ceiling — they verify pointers exist,
|
||||
not that their content is true; reading remains the job.
|
||||
|
||||
## References
|
||||
|
||||
- [`00-META/process/00-overview.md`](../00-META/process/00-overview.md) — the flow, and its new
|
||||
"The cycle is checked" section.
|
||||
- [ADR 0019](0019-how-this-repository-works.md) — the repository this disciplines.
|
||||
@@ -1,51 +0,0 @@
|
||||
---
|
||||
topic: how we work
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0080-the-development-cycle-is-checked.md
|
||||
---
|
||||
|
||||
# 81. A decision nothing cites is not yet in the chain
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0080](0080-the-development-cycle-is-checked.md) made the development cycle checked — but its
|
||||
checks covered designs, issues and research, not the decisions themselves. Measuring showed why
|
||||
that matters: 19 of 70 records were cited by nothing — no design doc's `decisions:`, no research
|
||||
`became:`, no issue, no other record. Among them sat load-bearing decisions (the credential flow,
|
||||
the module-runtime cluster), and the cost had already been paid once in practice: a stale premise
|
||||
about an orphaned decision survived in working memory precisely because no pointer led to the
|
||||
record that had settled it.
|
||||
|
||||
## Decision
|
||||
|
||||
Every **accepted** decision must be reachable from the cycle: cited by a design doc's frontmatter
|
||||
(`decisions:` — a governing citation, not a prose mention), a research overview, an issue report,
|
||||
a `00-META` document, or another record's `extends`/`supersedes` chain.
|
||||
[`cycle.py`](../00-META/checks/cycle.py) refuses orphans. Proposed records are exempt — a record
|
||||
under consideration has no home yet — and superseded records are reachable through their
|
||||
supersession chain by construction.
|
||||
|
||||
The 19 orphans were given true homes in the same change: the module-runtime cluster
|
||||
(0044–0049, 0053–0055) into the connectivity, controller, protocol, writing and model-access
|
||||
designs; the build decisions (0072, 0076) into the building design; 0021 into the playbook that
|
||||
implements it; the process records were already reachable once `00-META` counted as a source.
|
||||
|
||||
Taken together with 0080, the practice has a name the industry will recognise:
|
||||
**spec-driven development, with provenance** — a decision is the *why*, the design doc is the
|
||||
spec, `code:` names the implementation, and the lab beds are the conformance tests. What the
|
||||
common form leaves implicit, the cycle makes checked: the spec itself must trace to a decision,
|
||||
and the decision must be findable from the work it governs.
|
||||
|
||||
## Consequences
|
||||
|
||||
Following pointers now reaches every accepted decision, so a cleared session (or a person) can
|
||||
trust the frontmatter graph as the whole map. The check is reachability, not truth: a citation
|
||||
placed wrongly still lies, and reading remains the job.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0080](0080-the-development-cycle-is-checked.md) — the cycle this completes.
|
||||
- [`00-META/process/00-overview.md`](../00-META/process/00-overview.md) — the flow.
|
||||
@@ -1,92 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-17
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0072-two-graphs-and-the-build-chain.md
|
||||
---
|
||||
|
||||
# 82. The registry is reached by name, and the overlay is its security
|
||||
|
||||
## Context
|
||||
|
||||
Delivery ends at a node pulling an image, and for every node but the one that built it, that step
|
||||
has never worked. Three facts conspired, each recorded separately:
|
||||
|
||||
- Artifact references are written under `127.0.0.1:5000` — a deliberate parking (the catalogue
|
||||
says so in the commit that reverted the mesh-reachable name: *"the runtime refused it: 'http:
|
||||
server gave HTTP response to HTTPS client' … the binding expression returns then"*), so a joined
|
||||
node is told to pull from its own loopback
|
||||
([issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)).
|
||||
- A container runtime treats any non-loopback registry as HTTPS, and the mesh's registry serves
|
||||
plain HTTP; nothing the mesh writes tells any runtime otherwise — the one place that file
|
||||
existed was the lab's, which is why this never failed in a bed (issue 048).
|
||||
- Nothing gives a node an account for the registry
|
||||
([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)) —
|
||||
and nothing has ever said whether one is required.
|
||||
|
||||
The tempting fix is TLS from the mesh's own authority. The machinery even exists — the mesh CA
|
||||
issues leaves for `<node>.internal` names, and a `certificate:` manifest field delivers them. But
|
||||
the registry is reached **over the overlay**, and the overlay is WireGuard: every byte is already
|
||||
encrypted and already authenticated to a peer the mesh admitted. The mesh's own code has carried
|
||||
this position for weeks: *"the registry is reached over the mesh's own private network … a second
|
||||
layer inside it would be certificates to issue and rotate for no property the first does not
|
||||
have."* TLS inside the tunnel would also re-order genesis (a certificate needs the network, the
|
||||
registry precedes it) and put CA handling into three clients (the runtime, the archive fetcher,
|
||||
the builder) — cost with no new property.
|
||||
|
||||
## Decision
|
||||
|
||||
**The overlay is the registry's transport security, and the mesh writes the trust it means.**
|
||||
|
||||
1. **References name the registry by its mesh name.** The builder publishes to
|
||||
`<provider>.internal:<port>` (the binding's `at` and `port`) — the parked one-line change
|
||||
lands. Genesis still publishes to loopback on the first machine, before any overlay exists;
|
||||
references minted at genesis stay loopback and are valid where they matter — on that machine.
|
||||
A rebuild re-pins to the mesh name.
|
||||
2. **Every node on the private network is told the mesh's registry speaks plain HTTP.** The
|
||||
controller injects, into every such node's declaration, a merged `/etc/docker/daemon.json`
|
||||
naming `<provider>.internal:<port>` under `insecure-registries`, and a service resource that
|
||||
restarts the runtime when that file changes — the `/etc/hosts` pattern for the content, the
|
||||
nftables pattern for the reload. No module author is involved; being on the network is what
|
||||
grants the trust, because being on the network is what the trust *is*.
|
||||
|
||||
> **The mechanism changed — 2026-10-05, by [ADR 0222](0222-a-module-is-told-where-a-mesh-seats-holder-is-reached-and-the-controller-writes-no-file-a-seats-holder-owns.md).**
|
||||
> What stands: being on the network grants the trust, the overlay is the transport security, no
|
||||
> module author chooses it, and the trust is written into the runtime's file rather than over it
|
||||
> and reloaded rather than restarted (ADR 0102). What moved: the controller no longer injects the
|
||||
> file or the service. The container runtime's own module writes `insecure-registries`, told where
|
||||
> this machine reaches the store by `${seat:mesh-artifact-store:reach}`, because the controller
|
||||
> writes no file a seat's holder owns ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)).
|
||||
3. **No accounts (issue 042), recorded as the position it always was.** Reading and pushing
|
||||
require presence on the overlay and nothing else. The boundary is enforced, not assumed: the
|
||||
registry's `listens` is `from: mesh`, the firewall derives from it, and the overlay admits only
|
||||
peers holding mesh-issued keys. An operator's *external* registry credential is an ordinary
|
||||
operator-supplied secret (`secret accept`), owned by whichever module names that registry.
|
||||
Per-node accounts return as a decision, not a patch, if the boundary assumption ever changes —
|
||||
the shape would be the existing provision flow.
|
||||
|
||||
## How this is checked
|
||||
|
||||
- The no-fake lab bed: a joined node pulls a mesh-built image by the registry's `.internal` name
|
||||
over the overlay — an address outside every range the lab's own runtime configuration trusts,
|
||||
so the mesh-written trust is what makes it work or nothing does.
|
||||
- The firewall half is generated from `listens` and visible in the node's ruleset (`from: mesh`).
|
||||
- The plaintext-inside-tunnel position holds exactly as long as the registry is unreachable off
|
||||
the overlay; the `listens` stanza and the derived ruleset are the check.
|
||||
|
||||
## Consequences
|
||||
|
||||
A runtime restart when the trust first lands on a node — at joining, before workloads, where it
|
||||
is free. The registry's HTTP is exposed to whatever stands on the overlay, which is the stated
|
||||
boundary; a mesh that wants defence in depth inside its own tunnel reopens this record rather
|
||||
than bolting certificates on quietly. Issues 042 and 048 close on this record.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md),
|
||||
[issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md).
|
||||
- [ADR 0072](0072-two-graphs-and-the-build-chain.md) — the build chain this completes.
|
||||
- [ADR 0029](0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md) — the overlay as a
|
||||
boundary.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-18
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
---
|
||||
|
||||
# 83. One push leaves the mesh consistent
|
||||
|
||||
## Context
|
||||
|
||||
A provision is minted while composing the *consumer's* node; the provider's grant list is a pure
|
||||
read of secrets already issued from it. So assigning a cross-node consumer and pushing its node
|
||||
produced a consumer that retried forever against a provider that had never heard of it, until the
|
||||
provider's node was pushed a second time — an action with no signal to take, documented nowhere,
|
||||
and invisible whenever consumer and provider share a machine
|
||||
([issue 057](../04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md)).
|
||||
|
||||
Two remedies were on the table: **cascade** — a push also delivers to the machines its compose
|
||||
changed — or **report** — a push says "now push the provider" and leaves the act to the operator.
|
||||
|
||||
## Decision
|
||||
|
||||
A push finishes what it starts: after composing and sending the named node, the controller flushes
|
||||
every *other* machine that is now behind — whose declaration differs from what it was last sent —
|
||||
by name, in the push's own output, converging over a bounded number of rounds (a flushed send may
|
||||
itself mint).
|
||||
|
||||
Behind is measured against what a machine was last *sent*, not against a before/after snapshot of
|
||||
this push. The mint that makes a provider behind happens when the consumer is assigned or its
|
||||
account issued — before `push` runs at all — so by push time the provider already differs from
|
||||
what it holds, with no in-command delta to detect. The only durable signal is "what it should be"
|
||||
versus "what it last received", which is the same comparison `push --behind` already makes.
|
||||
|
||||
A machine behind for an unrelated reason is flushed by this too, and that is correct rather than a
|
||||
cost: a named push that knew a machine was behind and left it so would be the very silence this
|
||||
decision removes. The narrower reading — flush only what this push provably changed — was
|
||||
rejected because it cannot see a mint that a prior command performed, which is precisely the 057
|
||||
case.
|
||||
|
||||
Reporting alone was rejected because it converts a derived fact the controller already holds into
|
||||
an operator obligation, and an obligation enforced by nothing is issue 057 restated. The
|
||||
declaration is computed from the whole mesh; delivering a mesh that is knowingly inconsistent and
|
||||
merely saying so would make "push succeeded" mean less than it says.
|
||||
|
||||
> **The mechanism changed — 2026-10-05, by [ADR 0221](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md).**
|
||||
> A named push still flushes every other machine that is behind, compared against what each was last
|
||||
> sent. A machine whose modules would move to a build their upgrade policy records, or that an open plan
|
||||
> has not sent it yet, is no longer flushed: the push names it and leaves it for `push <node>`. "Behind
|
||||
> for an unrelated reason" no longer covers a held upgrade
|
||||
> ([issue 259](../04-ISSUES/259-a-named-push-sent-every-machine/00-report.md)).
|
||||
|
||||
## Consequences
|
||||
|
||||
- One push is sufficient for a cross-node consumer: the provider's grants arrive from the same
|
||||
act that minted the provision. The undocumented rule "push the provider node too" ceases to
|
||||
exist rather than becoming documentation.
|
||||
- A named push delivers to every machine that is behind, not only the one named — each named in
|
||||
the output, never silent. `push --behind` remains the way to reconcile the mesh without naming
|
||||
a node; a named push now carries the same guarantee for the machines its work touched and any
|
||||
others already waiting.
|
||||
- How this is checked: the built-store-cross-node bed registers a cross-node consumer, pushes
|
||||
only the consumer's node, and asserts the provider minted its vhost — the workaround push is
|
||||
removed, so a regression fails the bed.
|
||||
@@ -1,119 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-20
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0027-a-provision-names-what-the-consumer-is-coupled-to.md
|
||||
---
|
||||
|
||||
# 84. Which provider serves a consumer, when the mesh runs more than one
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) settled what a provision
|
||||
is *named* for — the thing the consumer's code is coupled to, so `postgres-database` and
|
||||
`mssql-database` are different provisions and a wrong match is refused at resolution. It said one
|
||||
thing more, in passing, and left it: *"Two providers of `postgres-database` — a container on this
|
||||
node and a managed instance elsewhere — are interchangeable and should both match."* Naming was
|
||||
decided; **which of several providers serves a given consumer was not.**
|
||||
|
||||
The mesh assumes there is only one to choose. Provisions are mesh-scoped: the adopted store is a
|
||||
single `mesh-store` a consumer on any node reaches, and `scope: "mesh"` is written into the
|
||||
provision definitions. **That assumption is false on day one, and was always meant to be.**
|
||||
Node-specific services delivered to the mesh is the plan, not an edge case:
|
||||
|
||||
- Both control-capable nodes already run their **own** general-purpose relational store (the same
|
||||
engine, one instance each), serving that node's own applications.
|
||||
- Each runs its **own** SQL server, its **own** cache, its **own** object store. One node alone
|
||||
runs six separate relational-store instances, each raised by the module that needed it.
|
||||
- The one provision that is currently single — the identity provider, one instance on one node —
|
||||
already authenticates applications whose home is a **different** node.
|
||||
|
||||
So several providers of one provision name genuinely coexist, and they are **not** interchangeable
|
||||
the way 0027's aside supposed. They differ by node, by the data they hold, and by locality. A
|
||||
consumer bound to the wrong one reads the wrong database, or takes a cross-node hop it did not
|
||||
need, or cannot be moved without silently rebinding. The model has no field in which to say which
|
||||
one. This is 0027's own fault — *a match that resolves and is wrong* — one level up: 0027 refused
|
||||
the wrong **dialect**; nothing refuses, or even asks about, the wrong **instance**.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep `scope: "mesh"` — one provider per provision, mesh-wide.** Rejected: it is false on day
|
||||
one, and making it true would force every node's applications onto one node's server — the
|
||||
exact opposite of node-specific services delivered to the mesh, and a single point of failure
|
||||
the topology was built to avoid.
|
||||
2. **Resolve to any provider of the name (0027's "both match").** Rejected: when providers hold
|
||||
different data and live on different nodes they are not interchangeable, and picking one
|
||||
arbitrarily is a wrong-instance match — the confidently-wrong answer 0027 exists to prevent,
|
||||
restated at the level of the instance rather than the dialect.
|
||||
3. **Always require the consumer to name the provider explicitly.** Rejected: needless ceremony in
|
||||
the common case, where the consumer wants the provider on its own node; and a field every
|
||||
manifest must carry is a field an author forgets, which then matches everything again — the
|
||||
failure 0027 warned about for qualifiers.
|
||||
4. **A provision is node-scoped; the consumer selects the provider, defaulting to co-location.**
|
||||
Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A provision is served by a provider identified by its node, and the consumer selects which one.**
|
||||
A provider is a (node, module) pair, not a mesh-wide singleton. A consumer's binding resolves to a
|
||||
specific provider, and the selection is part of the assignment
|
||||
([ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md)), not the
|
||||
manifest.
|
||||
|
||||
**The default is co-location.** A consumer that names no provider is served by the provider of
|
||||
that provision on **its own node**. This is the common case and needs nothing said. A mesh with
|
||||
one provider of a kind is just the case where co-location and "the only one" coincide — expressed
|
||||
as *there happens to be one*, not as a scope.
|
||||
|
||||
**A consumer coupled to a provider's data names it.** Where two consumers must share one database,
|
||||
or a consumer must reach a provider on another node, the assignment names that provider — because
|
||||
that coupling is exactly what may not be guessed, and naming it is what makes a later move safe.
|
||||
|
||||
**A module need not consume a provider at all.** It may carry its **own** instance inside its own
|
||||
composition — on its own module network, publishing no host port, **not** declared as a provision —
|
||||
when a genuine engine fork or a pinned server version makes the shared provider unusable. Such an
|
||||
instance is invisible to resolution and can be bound by nothing else. The rule is *share by
|
||||
default; embed only when a fork or a version forces it* — most of the per-module stores that exist
|
||||
today are vanilla engines on stale pins that a consolidation onto the node's provider would absorb.
|
||||
|
||||
## Consequences
|
||||
|
||||
The mesh can carry its real topology **deliberately** rather than by the accident of which
|
||||
provider happened to be the single one. A consumer's data-coupling becomes a stated fact, which is
|
||||
what lets a provider be moved without a consumer silently following the wrong one — provided a
|
||||
provider keeps its identity across a relocation, which is a follow-up this record opens rather than
|
||||
closes. Rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)) addresses a
|
||||
specific provider's holders, not "the provision's".
|
||||
|
||||
What got harder: an assignment now may carry a provider selection, and a wrong one is a new way to
|
||||
misconfigure. It is mitigated the way 0027 mitigated its own: the co-location default removes the
|
||||
choice in the common case, and genuine ambiguity — several providers, none named, none co-located —
|
||||
is refused with the candidates named, never resolved by picking.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0027](0027-a-provision-names-what-the-consumer-is-coupled-to.md) — the naming this extends;
|
||||
its "both match" aside is the gap closed here.
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is exactly such a
|
||||
provider, and already serves consumers on another node.
|
||||
- [ADR 0046](0046-a-module-configuration-is-its-assignments-not-its-manifest.md) — where the
|
||||
selection lives.
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the adopted store, whose mesh-wide
|
||||
binding is the single-provider assumption this record replaces.
|
||||
- [issue 067](../04-ISSUES/067-a-provision-cannot-name-which-provider-serves-it/00-report.md) — the
|
||||
gap, and the day-one evidence.
|
||||
- [`03-DESIGN/01-to-be/23-choosing-a-provider.md`](../03-DESIGN/01-to-be/23-choosing-a-provider.md)
|
||||
— the design.
|
||||
|
||||
> **Widened, 2026-10-01 (issue #258).** The pin named a node, on the reasoning that "the same
|
||||
> module on two machines is two answers, and which machine is the whole question". Half right:
|
||||
> two modules on one machine can both answer a provision — `public-acme` and `step-ca` both offer
|
||||
> `acme-ca` on novox — and then which *module* is the whole question, and a node alone cannot ask
|
||||
> it. A provider is a (node, module) pair ([design 23](../03-DESIGN/01-to-be/23-choosing-a-provider.md)),
|
||||
> and a pin now names the pair: `pin <node> <provision> <from-node> <module>`. The resolver
|
||||
> refuses a node that answers twice instead of taking the last one listed, and refuses two
|
||||
> providers beside the consumer instead of settling them by a map walk — the same stance design 23
|
||||
> takes: ambiguity is refused, never resolved by picking. Records made before are completed by
|
||||
> migration where the node they name answers once. mesh-controller: `feat/pin-names-the-provider`.
|
||||
@@ -1,153 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-20
|
||||
amended: 2026-09-20
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0031-the-control-plane-authenticates-nobody.md
|
||||
---
|
||||
|
||||
# 85. A secret is a provision, and the vault is the module that provides it
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0031](0031-the-control-plane-authenticates-nobody.md) decided that identity *runs on the
|
||||
mesh, not of it* — a module other modules require, rather than a privileged part of the
|
||||
controller. [ADR 0078](0078-the-store-and-broker-are-modules.md) did the same for the store and
|
||||
the broker: the twelve-module floor has no specialty left in it. **Secrets are the exception that
|
||||
survived.** No module owns a secret.
|
||||
|
||||
Secret handling is smeared across three built-in parts of the runtime:
|
||||
|
||||
- the **controller mints** one credential per consumer↔provider pair and seals it to both node
|
||||
keys ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md));
|
||||
- the **mesh generates local secrets** — *"generated secrets are the mesh's, never authored"*
|
||||
(as-is, `06-configuration-and-secrets.md`) — for a value a single module needs for its own use;
|
||||
- the **synchroniser injects** both into a node's generated files.
|
||||
|
||||
Nothing is the owner of "a secret" the way the store module is the owner of "a database", and the
|
||||
cost is recorded rather than hypothetical. The as-is design names the weakness in its own words:
|
||||
*"Rotation is not a mesh operation… there is no mechanism that rotates one and informs everything
|
||||
holding it."* A `rotate` command has since been built and proven, but it reaches **only** the
|
||||
provisioned pairs; a secret a module generates for its own fully-local use — the password of a
|
||||
version-pinned embedded store ([ADR 0084](0084-which-provider-serves-a-consumer.md)), an internal
|
||||
token — is minted by the mesh and then has no operation that can remake it. Three species of
|
||||
secret, and only the first has an owner:
|
||||
|
||||
| species | minted by | rotates? |
|
||||
|---|---|---|
|
||||
| a provisioned credential (a database login) | controller, sealed to nodes (0048) | yes — `rotate`, per pair |
|
||||
| a module's own local secret | the mesh, as a generated value | **no owner, no rotation** |
|
||||
| an operator-delivered secret (an external key) | a person, sealed in (`secret accept`) | no rotation, no audit |
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave it a property of the controller.** Rejected: it is the smear above — no owner, local
|
||||
secrets that cannot be rotated, operator secrets that cannot be audited — and it is exactly the
|
||||
specialty 0078 removed for the store and broker, kept here for no reason anyone recorded.
|
||||
2. **A dedicated vault built into the foundation, not a module.** Rejected: it reintroduces a
|
||||
privileged built-in, the thing 0031 and 0078 went out of their way to remove, and a mesh that
|
||||
wants none would still carry it.
|
||||
3. **Fold all minting, the controller's provisioning credentials included, into the vault.**
|
||||
Rejected: the controller must mint in order to **deliver** any provision — the vault's own
|
||||
credential among them — so making the vault mint the credential of its own delivery is the
|
||||
store/broker chicken-and-egg for no gain. The provisioned-pair credential already has an owner
|
||||
and a rotation ([13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)); this record
|
||||
does not disturb it.
|
||||
4. **A secret is a provision; the vault is an ordinary module that provides it.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**Secret-holding is a module, parallel to identity.** A module that needs a secret **for its own
|
||||
use** — a local service's password, an internal token, an external key it was handed — requires a
|
||||
`secret` provision from a vault provider, exactly as it requires a database from the store. The
|
||||
vault generates the value (or holds one it was given), and because the credential belongs to the
|
||||
consumer↔vault pair it **rotates, backs up and is audited through the same per-pair machinery**
|
||||
[13](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md) already defines. That is what
|
||||
gives the second and third species the rotation and audit they lack. A module's own secret stops
|
||||
being a generated value that nothing owns and becomes an ordinary provision with a provider.
|
||||
|
||||
**The controller's minting of provisioning-pair credentials is unchanged** ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)):
|
||||
it is how every provision, the vault's own included, is delivered. The vault does not mint the
|
||||
mesh's delivery credentials; it provides secrets to modules, and is itself provisioned the ordinary
|
||||
way.
|
||||
|
||||
**The vault is node-scoped like every provider** ([ADR 0084](0084-which-provider-serves-a-consumer.md)):
|
||||
each node its own, selected the same way, so a module's own secret is held by the vault on the
|
||||
module's node. **A mesh that wants no vault runs none** — a module requiring no secret needs
|
||||
nothing, which is the same test 0031 applied to identity.
|
||||
|
||||
## Consequences
|
||||
|
||||
The gap that opened this — a generated local secret with no rotation — **closes without new
|
||||
machinery**: rotating such a secret is the vault's provisioner remaking a pair credential, the
|
||||
operation 13 already specifies. Backup, audit and break-glass gain an owner — the vault module —
|
||||
and become things a design specifies rather than absences. The as-is sentence *"rotation is not a
|
||||
mesh operation"* is already false for provisioned pairs and, once the vault ships, for local
|
||||
secrets too; the as-is document is updated when it does, not before.
|
||||
|
||||
What got harder: a break-glass path — recovering a secret when the sealed delivery path is
|
||||
unavailable — must not reintroduce a key that one place holds, which is the property
|
||||
[ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) was built to preserve; how
|
||||
the vault offers recovery without it is left to the design as an open question. And a module that
|
||||
today bakes a password into its own composition must instead require it from the vault — a
|
||||
migration taken module by module, not a flag day.
|
||||
|
||||
## Amendment — 2026-09-20, before anything shipped
|
||||
|
||||
Recorded on the record itself rather than as a supersession, by its decider, on the day it was
|
||||
accepted and before any code was merged against the sentences that change. The original text above
|
||||
is left as written; this section says what it got wrong and what stands instead.
|
||||
|
||||
**What it got wrong.** The decision treated the vault as a provider like the store — optional, and
|
||||
one per node — and left the mesh's own root secrets outside it: the store's superuser, the broker's
|
||||
administrator, the controller's contexts, sealed to a node key and nothing else. Those are the
|
||||
secrets with no rotation and no recovery, and they are the ones a vault exists for. At genesis they
|
||||
are not even secret: the foundation raises its store and broker with fixed, well-known credentials
|
||||
and carries those into the mesh. Leaving that floor in place gave module secrets an owner and the
|
||||
root secrets none.
|
||||
|
||||
**What stands instead.**
|
||||
|
||||
- **The vault is a foundation module.** It is installed at genesis as part of the foundation
|
||||
([ADR 0078](0078-the-store-and-broker-are-modules.md) is the precedent: a foundation piece is
|
||||
still an ordinary module), not assigned later by a mesh that happens to want one. *"A mesh that
|
||||
wants no vault runs none"* is withdrawn. A mesh has root secrets, so a mesh has a vault.
|
||||
- **One per mesh, on the control-node.** *"The vault is node-scoped like every provider"* is
|
||||
withdrawn. Node scoping ([ADR 0084](0084-which-provider-serves-a-consumer.md)) exists because a
|
||||
store holds data a consumer is coupled to; the vault holds nothing a consumer is coupled to, and a
|
||||
second one would be a second place to lose. Consumers on other nodes reach it as they reach
|
||||
identity.
|
||||
- **The vault holds the mesh's root secrets under an operator-held key.** The controller mints and
|
||||
delivers exactly as before; in addition, every secret a module holds for itself is sealed a second
|
||||
time, to an **operator sealing key** whose private half never enters the mesh. The vault keeps
|
||||
those operator-sealed copies on its own disk, outside the store, and can hand them out — they are
|
||||
ciphertext to everything but the operator. This is the break-glass path the original text left
|
||||
open, and it does **not** reintroduce a key one place holds: the mesh holds blobs it cannot open,
|
||||
and the operator holds a key with nothing to open until given a blob. Recovery needs both.
|
||||
- **Genesis mints real root secrets and seals them to the operator key first**, so the fixed
|
||||
credentials the foundation is raised with are replaced before the mesh is handed over.
|
||||
|
||||
**Unchanged.** A module's own secret is a `secret` provision the controller mints and the vault
|
||||
records; the provisioned-pair path of [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)
|
||||
is untouched; the vault stores no plaintext, ever.
|
||||
|
||||
**What it costs.** An operator key is a thing a person must keep, and a mesh whose operator key is
|
||||
lost has root secrets that can be rotated but not recovered — the same standing as today, stated.
|
||||
Sealing every own secret twice is a column and a call. Genesis grows a step. A module's
|
||||
vault-provided secret (a pair credential) is not yet sealed to the operator key; that is the next
|
||||
increment, not this one.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0031](0031-the-control-plane-authenticates-nobody.md) — identity is a module; this is the
|
||||
same move for secrets.
|
||||
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md) — the provisioning-credential
|
||||
path, left unchanged.
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md) — the de-specialisation this completes.
|
||||
- [ADR 0084](0084-which-provider-serves-a-consumer.md) — the node-scoping the vault obeys.
|
||||
- [issue 068](../04-ISSUES/068-secrets-have-no-owning-module/00-report.md) — the gap.
|
||||
- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md) —
|
||||
the design. The source mesh's `secret_locate` / `secret_backup` / `secret_verify` /
|
||||
`secret_breakglass` subsystem is the prior art it draws on.
|
||||
@@ -1,75 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
---
|
||||
|
||||
# 86. A secret reaches a process as a file, and an exception is declared
|
||||
|
||||
## Context
|
||||
|
||||
The mesh seals a secret to the machine that uses it and discards the plaintext; the host unseals it
|
||||
into a file at 0600 ([ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md)). Then
|
||||
the module hands it to its container through an env-file, and the runtime puts it where anything
|
||||
on the machine that can talk to the runtime can read it: `docker inspect` prints it, and
|
||||
`/proc/<pid>/environ` holds it for the life of the process
|
||||
([issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md)).
|
||||
|
||||
Two facts made this a decision rather than a fix. **It is a supported shape**: placing
|
||||
`${secret:…}` in a file a container reads as its environment is what playbook 06 shows, and 36
|
||||
containers in 25 catalogue modules do it, the controller's own among them. And **nothing says which
|
||||
secrets are exposed**: a reader cannot tell from a manifest whether a credential is protected from
|
||||
`inspect` or not, because the manifest looks the same either way. The vault
|
||||
([ADR 0085](0085-a-secret-is-a-provision.md)) rests on the seal this undoes.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave it: the environment is where configuration goes.** Rejected — it gives the sealed value
|
||||
back to a routine operation, and the care taken to seal it is then theatre.
|
||||
2. **Refuse every secret in an environment.** Rejected — some software reads its configuration
|
||||
from the environment and nothing else, and a rule the catalogue cannot obey is a rule that gets
|
||||
switched off.
|
||||
3. **A secret reaches a process as a file; an environment exception is declared, with a reason,
|
||||
and refused otherwise.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A secret reaches a process as a file.** A module mounts the file the host wrote and points the
|
||||
program at it; the mesh's own programs accept a `_FILE` twin for every variable that carries a
|
||||
credential, the way the controller's store connections already did.
|
||||
|
||||
**A container that reads a secret from its environment says so.** The manifest key
|
||||
`secrets-in-environment` on the container carries the reason. It is catalogue-level — the host
|
||||
never sees it — and it exists so a reader can tell from the manifest which secrets are exposed
|
||||
that way and why.
|
||||
|
||||
**Everything else is refused.** A `${secret:…}` placeholder inside a container's `env` is never
|
||||
filled and is refused outright. A file carrying a secret that a container names in `env-file` is
|
||||
refused unless the container declares the exception.
|
||||
|
||||
## Consequences
|
||||
|
||||
The property *a sealed credential is readable only where it is used* becomes a manifest-level
|
||||
fact: true where no exception is declared, stated where one is. The controller reads all six of
|
||||
its credentials from files; the 35 other containers are marked with their reason and convert one
|
||||
by one where their software accepts a path, which is per-module work under
|
||||
[issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md).
|
||||
|
||||
What got harder: a module author meets one more refusal, and the reason they write is only as
|
||||
honest as they are. What is not decided: the reach of the exposure on a node — which identities can
|
||||
talk to the runtime — which decides whether a declared exception is a hardening item or something
|
||||
sharper.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The catalogue engine refuses at composition, before anything reaches a machine, and its tests
|
||||
refuse both shapes and prove the reason is stripped from what is sent.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 041](../04-ISSUES/041-a-sealed-credential-ends-up-in-the-process-environment/00-report.md)
|
||||
- [ADR 0048](0048-a-provider-creates-the-credential-the-mesh-minted.md), [ADR 0085](0085-a-secret-is-a-provision.md)
|
||||
- [`03-DESIGN/01-to-be/13-credentials-and-their-rotation.md`](../03-DESIGN/01-to-be/13-credentials-and-their-rotation.md)
|
||||
@@ -1,63 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
---
|
||||
|
||||
# 87. A seeded file is created once, and what grows in it is not the mesh's
|
||||
|
||||
## Context
|
||||
|
||||
A declaration is complete for what the host owns, and the host reconciles what is declared
|
||||
([ADR 0010](0010-delivery.md)): a file with this content, held to it. That is the only thing a
|
||||
manifest could say about a file, and it is the wrong thing for a file a module needs to **exist
|
||||
before first start** and something else then legitimately writes into — an access list a
|
||||
provisioner appends consumers to and the program persists back, a bootstrap configuration a
|
||||
program rewrites. Every reconcile restored the seed behind the running program, erased what had
|
||||
grown in it, and reported success
|
||||
([issue 035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)).
|
||||
|
||||
A run-once step ([ADR 0052](0052-a-step-that-runs-once-before-a-container.md)) can write a seed
|
||||
only if absent, and that closed the instance for a broker whose seed is a program's job. It left
|
||||
the general case: a plain file the mesh writes and never overwrites.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Two owners never share a file: the provisioner owns it, and first-start ordering is solved
|
||||
another way.** Rejected as the only answer — some software refuses to start without the file,
|
||||
and a module that must ship a program merely to write an empty file has been made to write a
|
||||
program to say one word.
|
||||
2. **A create-once semantic on a file.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A file resource may say `create-once`. The host writes it when it is absent and, when it is
|
||||
present, leaves it entirely alone — content, mode and owner — and reports it as **kept**, not
|
||||
corrected. What is in the file then is somebody else's work the mesh asked for. The mesh removes
|
||||
nothing it did not create ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md)); it now
|
||||
also does not overwrite what it created once and handed over.
|
||||
|
||||
On the security question ADR 0010 asks of every new resource behaviour: this **narrows** what a
|
||||
declaration can do to a machine. A create-once file gives a compromised control plane one fewer
|
||||
way to change a machine repeatedly — it can seed, once, and never again.
|
||||
|
||||
## Consequences
|
||||
|
||||
A module says which of its files are seeds, and the difference is visible in the manifest rather
|
||||
than in whether the file happened to be revisited. A later change to a seed's declared content
|
||||
does not reach a machine that already has the file; that is the meaning of a seed, and a module
|
||||
that needs the new content ships it as a run-once step that migrates the existing file.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The host's apply tests: a seed is created, grown into by hand, reconciled, and the growth survives
|
||||
with the outcome `kept`. The vault bed declares one on a real node, grows it, pushes again, and
|
||||
reads it back.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 035](../04-ISSUES/035-reconciling-a-seed-file-wipes-what-grew-in-it/00-report.md)
|
||||
- [ADR 0010](0010-delivery.md), [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), [ADR 0052](0052-a-step-that-runs-once-before-a-container.md)
|
||||
@@ -1,65 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0078-the-store-and-broker-are-modules.md
|
||||
---
|
||||
|
||||
# 88. The foundation filters before anything listens
|
||||
|
||||
> **Superseded in part — 2026-09-22, by [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md).**
|
||||
> On a node raised adopted, the base ruleset is not loaded: its duty, the store never reachable from outside,
|
||||
> passes to a table of the mesh's that only refuses. On every other node the base ruleset, loaded before the
|
||||
> store and broker are raised, stands as decided here.
|
||||
|
||||
## Context
|
||||
|
||||
Adopting the store and broker as ordinary modules
|
||||
([ADR 0078](0078-the-store-and-broker-are-modules.md)) needs them reachable by consumers across
|
||||
the mesh, so genesis raises them bound to every interface. The packet filter that decides who may
|
||||
reach them is a module too, installed a dozen steps later. Between the two, a control-node facing
|
||||
the network has its store and its bus open to anyone who can reach the machine, for the length
|
||||
of the install ([issue 054](../04-ISSUES/054-the-adopted-store-and-broker-are-open-before-the-filter/00-report.md)).
|
||||
The design's rule — what a port is reachable from is decided by the filter — is enforced by
|
||||
nothing for that window.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Accept the window**: the machine is mid-bootstrap and the exposure matches what the
|
||||
pre-adoption modules had in steady state. Rejected — the whole point of deriving the filter
|
||||
was to stop accepting that.
|
||||
2. **Bind narrowly at genesis and widen once the filter exists.** Rejected — a bind change is a
|
||||
recreate of the mesh's store during install, and adoption in place needs the same spec.
|
||||
3. **The foundation carries a filter of its own, applied before the store.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
The foundation bundle installs the packet filter and loads a base ruleset **before the store and
|
||||
broker are raised**: drop by default; keep loopback, replies, ping and ssh; keep the mesh's own
|
||||
ports a node must reach before it is on the private network — the bus it enrols over and the
|
||||
registry it pulls from; and let the container runtime's own networks through the forward chain so
|
||||
containers keep working. It is written into the **same table** the filter module later derives, so
|
||||
that module replaces it wholesale the moment it can compute one from what the mesh knows, and
|
||||
nothing of the base survives to contradict it.
|
||||
|
||||
## Consequences
|
||||
|
||||
From its first resource a machine being made into a mesh refuses what it will refuse when
|
||||
finished; the window closes. What got harder: the base ruleset is static and names two ports the
|
||||
mesh's derived one also names — a change to which ports the foundation needs is now made in two
|
||||
places, and the bundle's own test says which.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The installer's bundle test asserts the filter and its load precede the store and broker and that
|
||||
the rules name ssh, the bus and the registry and not the store's or broker's client ports. The
|
||||
genesis bed probes the machine from outside throughout the install: the store's port is never
|
||||
reachable, while the bus becomes reachable.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 054](../04-ISSUES/054-the-adopted-store-and-broker-are-open-before-the-filter/00-report.md), issue 047
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md)
|
||||
- [`03-DESIGN/01-to-be/07-the-foundation.md`](../03-DESIGN/01-to-be/07-the-foundation.md)
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
topic: checking it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0016-the-lab.md
|
||||
---
|
||||
|
||||
# 89. A bed reads the catalogue it proves
|
||||
|
||||
## Context
|
||||
|
||||
A lab bed installs a catalogue module and asserts what the mesh does with it; "proven in the
|
||||
lab" is the standard a module must meet before it ships. The beds built the manifests they
|
||||
install inline — a literal copied from the catalogue when each bed was written, and never since.
|
||||
Six modules were converted to file-delivered secrets ([ADR 0086](0086-a-secret-reaches-a-process-as-a-file.md))
|
||||
and not one bed ran the converted shape; each ran its copy, and the copies still delivered
|
||||
secrets the way the catalogue engine now refuses
|
||||
([issue 073](../04-ISSUES/073-beds-carry-copies-of-catalogue-manifests/00-report.md)). A run's
|
||||
receipt named the commits of the host, the controller and the lab, and said nothing about the
|
||||
catalogue, so a catalogue change and a proven catalogue change were indistinguishable.
|
||||
|
||||
The end-to-end design already has the rule this breaks — *the run rebuilds what it tests; an
|
||||
artifact rebuilt from memory is one rebuilt sometimes* — for binaries and images. A manifest is
|
||||
an artifact too.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep the copies and check them against the catalogue** — a test that diffs each literal
|
||||
against the module's manifest, ignoring what the lab must rewrite. Rejected: it keeps two
|
||||
sources of truth and adds a third thing that can drift, the list of what to ignore.
|
||||
2. **Read the catalogue, rewriting only what the lab must.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A bed that installs a catalogue module reads that module's manifest from the catalogue checkout
|
||||
the run was pointed at. It may rewrite what the lab must and nothing else: a build artifact
|
||||
becomes the image the machine holds, an image is pinned to what the machine holds, a host port
|
||||
is remapped where one machine carries colliding modules, and an address may point at a stand-in
|
||||
the bed raises in place of an upstream. Everything else is the catalogue's, verbatim.
|
||||
|
||||
A bed that needs less than the catalogue declares — no upstream server, a secret in the
|
||||
environment, a requirement edge removed — is not testing that module. It is a mesh test, and it
|
||||
carries a name of its own (see [issue 074](../04-ISSUES/074-a-mesh-test-wears-a-catalogue-modules-name/00-report.md)).
|
||||
|
||||
The run's receipt names the catalogue's commit alongside the other repositories', so a receipt
|
||||
taken before a manifest changed says so.
|
||||
|
||||
## Consequences
|
||||
|
||||
A catalogue change is proven by the beds that install the module, or it is not proven, and the
|
||||
receipt says which. What got harder: a bed can no longer trim a module to the shape it finds
|
||||
convenient; it meets the module's declared requirements or gives its fixture another name.
|
||||
The beds that still carry a copy are declared, each with its reason, and the declared list
|
||||
only shrinks.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A unit test in the lab refuses an inline manifest literal that names a catalogue module unless
|
||||
the bed is declared, with its reason, in the test's own list; a declaration for a bed that no
|
||||
longer carries the copy is refused too, so the list cannot outlive the debt. The receipt test
|
||||
asserts the catalogue is claimed whenever the run is pointed at one.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 073](../04-ISSUES/073-beds-carry-copies-of-catalogue-manifests/00-report.md), [issue 074](../04-ISSUES/074-a-mesh-test-wears-a-catalogue-modules-name/00-report.md)
|
||||
- [ADR 0016](0016-the-lab.md), [ADR 0086](0086-a-secret-reaches-a-process-as-a-file.md)
|
||||
- [`03-DESIGN/01-to-be/01-end-to-end-testing.md`](../03-DESIGN/01-to-be/01-end-to-end-testing.md)
|
||||
@@ -1,65 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
---
|
||||
|
||||
# 90. A failure that repeats is said to be stuck
|
||||
|
||||
## Context
|
||||
|
||||
Delivery is a comparison, not a one-shot ([ADR 0010](0010-delivery.md)): a node re-applies the
|
||||
declaration it holds on a steady interval and reports each time. That is right for a failure that
|
||||
goes away by itself — the overlay not up yet, a registry briefly unreachable — and it makes a
|
||||
failure that will never go away look exactly the same. A resource nothing can ever apply is
|
||||
attempted, fails, is reported, and is attempted again every few minutes, indefinitely; the mesh
|
||||
keeps one report per machine, replaced, so each attempt arrives as "failed" at a fresh time.
|
||||
Nothing distinguished "failed once, will succeed when its dependency arrives" from "failed
|
||||
identically for ever", and nothing escalated the second
|
||||
([issue 065](../04-ISSUES/065-a-permanently-failing-resource-is-retried-for-ever-with-no-escalation/00-report.md)).
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The host gives up** after some number of attempts. Rejected: the host does not know whether
|
||||
a failure is permanent — that a registry has not answered three times is not evidence it never
|
||||
will — and a host that stops trying is a node that must be pushed to again by hand.
|
||||
2. **A duration** since the failure was first seen. Rejected as the signal: a laptop shut for a
|
||||
week has had one attempt, and a week is not evidence of anything.
|
||||
3. **The controller counts identical reports**, and says when there are enough of them. Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
The mesh keeps, beside each machine's last report, when the current failure was first reported
|
||||
and how many reports in a row have said it — the same outcome, the same refusal, the same failed
|
||||
resources by id. Not by the host's words: an error carrying a duration or a counter would read as
|
||||
new on every report, and the resource looping on it is exactly what this is for. A report that
|
||||
says something different starts the count again; a clean apply clears it. **Three identical reports in a row make a machine stuck**: `status` says
|
||||
so beside the failure, with the count and the time it began, and the machine-readable status
|
||||
carries the same three facts. The host keeps retrying; being stuck is a statement about the
|
||||
mesh's knowledge, not an instruction to the machine.
|
||||
|
||||
The controller counts rather than the host, because only it sees every node: one stuck machine
|
||||
and a mesh-wide fault are different situations, and the host cannot tell them apart.
|
||||
|
||||
## Consequences
|
||||
|
||||
A resource that will never apply is visible from `status` after three reconcile intervals, to
|
||||
anyone who looks, without being asked for. What got harder: nothing on the machine changes — a
|
||||
gating failure that stops what follows still stops it, and this only makes the wait visible.
|
||||
Whether a stuck machine should also be raised as an event, and what a gating failure should do,
|
||||
stay open in the issue's own questions.
|
||||
|
||||
## How it is checked
|
||||
|
||||
An inventory test records the same failure three times and asserts the count and the unchanged
|
||||
start; the same resource failing in other words, and asserts the count went on; a different
|
||||
failure, and asserts it restarted; a clean apply, and asserts both cleared. The status command's own test asserts a stuck machine is said to be one.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 065](../04-ISSUES/065-a-permanently-failing-resource-is-retried-for-ever-with-no-escalation/00-report.md)
|
||||
- [ADR 0010](0010-delivery.md)
|
||||
- [`03-DESIGN/01-to-be/10-delivery.md`](../03-DESIGN/01-to-be/10-delivery.md)
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0051-shared-data-is-the-operators.md
|
||||
---
|
||||
|
||||
# 91. A mount is declared, and there are three things it can be
|
||||
|
||||
## Context
|
||||
|
||||
A container's bind mount whose source does not exist is created by the container runtime, as
|
||||
root, with whatever mode it picks. So `owner` and `mode` — which exist so a module can say who
|
||||
its data belongs to — never reach the directories that hold data, and the rule that keeps a
|
||||
directory when a module goes away ([ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md))
|
||||
does not cover them, because the mesh has never heard of them
|
||||
([issue 026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md)). Fourteen
|
||||
such mounts were declared by hand; a check that every mount is declared was then written and
|
||||
withdrawn, because it refused the builder: the builder mounts the container runtime's socket,
|
||||
which is not its data, already exists, and belongs to the machine. Declaring it as the module's
|
||||
own directory would be a lie the host would act on.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Declare the socket as a directory anyway.** Rejected: the host would create, own and
|
||||
protect a path that is the machine's.
|
||||
2. **A new manifest field** naming machine paths a module may mount. Rejected: the manifest
|
||||
already says the module needs the container runtime, and a second field would say the same
|
||||
thing in paths.
|
||||
3. **Three declarations, one for each kind of path a mount can be.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A container may not mount a path the module never declared, and a path is declared in one of
|
||||
three ways, which are the three things a path can be:
|
||||
|
||||
- **the module's own** — a directory or file resource, or where a secret, a grant or a
|
||||
contribution lands. Created and owned by the mesh for this module, kept when the module goes;
|
||||
- **the operator's** — an `accesses` entry ([ADR 0051](0051-shared-data-is-the-operators.md)):
|
||||
pre-existing, shared, granted for use, never owned;
|
||||
- **the machine's** — a facility a declared capability grants. `container-runtime` grants its
|
||||
socket. The path exists, the machine owns it, and the capability is the declaration.
|
||||
|
||||
A mount under a declared directory is declared. The check runs where the manifest is parsed,
|
||||
and names the path and the three remedies.
|
||||
|
||||
## Consequences
|
||||
|
||||
Every directory that holds a module's data is one the mesh created with the module's owner and
|
||||
mode, and one ADR 0030 protects. What got harder: a manifest borrowed from a compose file no
|
||||
longer passes on the strength of its volume lines; each must say what kind of path it mounts.
|
||||
The table of what a capability grants is small and in the catalogue's parser; a new capability
|
||||
that grants a path adds a row.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Manifest tests refuse an undeclared mount, accept one under a declared directory, accept one
|
||||
the module accesses, and accept the runtime's socket with the capability and refuse it without.
|
||||
A test parses every manifest in the catalogue beside the checkout and fails on any that breaks
|
||||
the rule, so the catalogue cannot drift back.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 026](../04-ISSUES/026-the-data-directories-are-not-declared/00-report.md)
|
||||
- [ADR 0030](0030-data-outlives-the-mesh-that-declared-it.md), [ADR 0051](0051-shared-data-is-the-operators.md)
|
||||
- [`03-DESIGN/01-to-be/18-building-a-module.md`](../03-DESIGN/01-to-be/18-building-a-module.md)
|
||||
@@ -1,61 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
---
|
||||
|
||||
# 92. An operator delivers a pair credential, and the mesh never replaces it
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0085](0085-a-secret-is-a-provision.md) names three species of secret and gives the vault
|
||||
two of them: a module's own secret, which the mesh mints, and an operator-delivered secret — a
|
||||
credential for something outside the mesh, which only a person can supply. Under 0085 a secret
|
||||
from the vault is a pair credential between the consumer and the vault. The controller's one
|
||||
command that takes a value from a person wrote only a module's own secret, sealed to one node.
|
||||
Nothing could put a value into a pair, so the third species had no entry
|
||||
([issue 070](../04-ISSUES/070-an-operator-cannot-deliver-a-pair-credential/00-report.md)), and
|
||||
an operator's credential could be held only as an own secret — un-audited, un-rotatable, the
|
||||
gap 0085 opened to close.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A new verb on the pair.** Rejected: `secret accept` already means "a value a person
|
||||
supplied, sealed on the way in, plaintext discarded"; a second verb would mean the same.
|
||||
2. **`secret accept` grows a provider end.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
`secret accept <node> <module> <name> --provider <node>` seals the supplied value to the
|
||||
consumer's node, to the provider's node, and to the operator's key when the mesh has one, and
|
||||
records the pair as `accepted`. Every pair credential now says where it came from: `made` or
|
||||
`accepted`.
|
||||
|
||||
An accepted pair is never replaced by a made one. When a sealing key at either end changes, the
|
||||
mesh cannot re-seal a value it does not hold, so the read is refused and names the remedy —
|
||||
accept it again. `rotate` refuses an accepted pair for the same reason: the mesh cannot make its
|
||||
replacement, and deleting it would have the next read mint one, delivered and reported as
|
||||
applied while failing to authenticate somewhere else entirely. Rotating an accepted credential
|
||||
is accepting a new value.
|
||||
|
||||
## Consequences
|
||||
|
||||
A credential for something outside the mesh lives in the vault's ledger with the others, sealed
|
||||
to both ends and recoverable by the operator. What got harder: a mesh whose node keys change
|
||||
cannot heal an accepted pair by itself; a person is asked. That is the honest shape — the value
|
||||
was never the mesh's to make.
|
||||
|
||||
## How it is checked
|
||||
|
||||
An inventory test accepts a value into a pair, reads it back twice unchanged with origin
|
||||
`accepted`, asserts `rotate` refuses it naming the remedy while a made pair still rotates;
|
||||
another changes a node's key and asserts the read is refused, then accepts again and reads.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 070](../04-ISSUES/070-an-operator-cannot-deliver-a-pair-credential/00-report.md), [issue 069](../04-ISSUES/069-one-secret-provision-yields-one-value/00-report.md)
|
||||
- [ADR 0085](0085-a-secret-is-a-provision.md)
|
||||
- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md)
|
||||
@@ -1,53 +0,0 @@
|
||||
---
|
||||
topic: checking it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0089-a-bed-reads-the-catalogue-it-proves.md
|
||||
---
|
||||
|
||||
# 93. A fixture that runs a module's runtime carries the module's name
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0089](0089-a-bed-reads-the-catalogue-it-proves.md) said a bed that needs less than a module
|
||||
declares is a mesh test, and carries a name of its own. Twelve beds were to be renamed on that
|
||||
basis ([issue 074](../04-ISSUES/074-a-mesh-test-wears-a-catalogue-modules-name/00-report.md)).
|
||||
Reading how a module's tools are reached showed why they cannot be: the account the mesh issues
|
||||
a module is scoped to `serve.<module>.*` from the manifest's name, and the runtime binds its tool
|
||||
queues under the name baked into its image. A fixture named otherwise but running the real
|
||||
runtime would be refused its own queues. The name is not a label; it is the tool namespace and
|
||||
the broker scope.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Rename anyway, and rebuild each runtime under the fixture's name.** Rejected: a runtime built
|
||||
under a false name proves nothing about the module and costs a build per bed.
|
||||
2. **A fixture that runs a module's runtime carries the module's name, and therefore reads the
|
||||
catalogue.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A bed that runs a module's runtime installs the catalogue's manifest for that module and what it
|
||||
requires — the vault for a `secret`, the route module for a route — and proves the mechanism
|
||||
against the real module. A name of its own is for a fixture that runs no real runtime: a
|
||||
declaration-only stub, a bare upstream image.
|
||||
|
||||
## Consequences
|
||||
|
||||
The mechanism beds become module beds with a mechanism inside them, which is more than they
|
||||
were. What got harder: a bed that wanted a cut-down redis now raises the vault beside it; one that
|
||||
wanted a sidecar without its server raises the server. Each conversion is a lab run, and the beds
|
||||
still carrying a copy are declared with this reason until converted.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The lab's inline-copy check refuses undeclared copies as before; the declared list's reason for
|
||||
these beds names this record. Two beds converted with the vault beside them ran green.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 074](../04-ISSUES/074-a-mesh-test-wears-a-catalogue-modules-name/00-report.md)
|
||||
- [ADR 0089](0089-a-bed-reads-the-catalogue-it-proves.md), [ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)
|
||||
- [`03-DESIGN/01-to-be/01-end-to-end-testing.md`](../03-DESIGN/01-to-be/01-end-to-end-testing.md)
|
||||
@@ -1,63 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
---
|
||||
|
||||
# 94. A module may hold several secrets from one provider, each a pair of its own
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0085](0085-a-secret-is-a-provision.md) makes a module's own secret a provision: the module
|
||||
requires `secret` from the vault and reads the pair credential minted for that consumer↔vault
|
||||
pair. A pair has one credential, a module requires a provision once, and so a module received
|
||||
one value. Read against the catalogue, nine modules hold two or more secrets besides their
|
||||
broker account; seven of them hold genuinely independent values with independent lifetimes — a
|
||||
root certificate, its key and that key's password; an admin password beside an API token. None
|
||||
derives from another, so "one value, derivation the module's business" answers nothing
|
||||
([issue 069](../04-ISSUES/069-one-secret-provision-yields-one-value/00-report.md)).
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **One value per module; the module derives the rest.** Rejected: the values are independent.
|
||||
2. **Require the provision several times.** Rejected: `requires` is a list of names, and a
|
||||
requirement is matched by name everywhere.
|
||||
3. **The `secrets` map names several files under local names, and each local name is a pair
|
||||
credential of its own.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A module's `secrets` entry for a requirement may be a path, as before, or an object of local
|
||||
names to paths. Each local name is its own pair credential, keyed on it beside the provision,
|
||||
the consumer node, the consumer module and the provider; its own file on the consumer, referred
|
||||
to as `${secret:<local name>}`; its own holder at the provider, named the consumer's identity
|
||||
with the local name after it; and rotated apart from the others. A local name may not be one of
|
||||
the module's own secrets or something it requires, so what a placeholder means is never
|
||||
ambiguous. The plain shape is unchanged, and every credential that exists is the one it was.
|
||||
|
||||
The holder's suffix is not a login any backend checks — a secret is not a login — so the
|
||||
identity limit that binds a database role or an access key does not apply to it.
|
||||
|
||||
## Consequences
|
||||
|
||||
The ten modules that could not move onto the vault can. What got harder: `rotate secret` for a
|
||||
consumer rotates every local name it holds from that provider; rotating one of several is a
|
||||
finer command than the mesh has, and waits for a case that needs it.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Manifest tests read both shapes, write them back, and refuse a colliding or unusable local
|
||||
name. A resolver test asserts two local names are two needs, two files with two credentials,
|
||||
and two holders at the provider. An inventory test asserts two local names are two rows, that
|
||||
rotating one leaves the other, and that the provider is told both. The vault bed installs a
|
||||
consumer that keeps two secrets and asserts two values delivered, two holders in the vault's
|
||||
ledger, and both rotated by one command.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 069](../04-ISSUES/069-one-secret-provision-yields-one-value/00-report.md)
|
||||
- [ADR 0085](0085-a-secret-is-a-provision.md), [ADR 0049](0049-a-consumers-identity-fits-the-tightest-backend.md)
|
||||
- [`03-DESIGN/01-to-be/24-the-secrets-vault.md`](../03-DESIGN/01-to-be/24-the-secrets-vault.md)
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md
|
||||
---
|
||||
|
||||
# 95. The control plane is the way to ask a module
|
||||
|
||||
## Context
|
||||
|
||||
A module serves tools over the broker under an account scoped to what it emits, consumes and
|
||||
serves ([ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)). A
|
||||
tool call is a request and a reply: the caller creates a reply queue and publishes to the serving
|
||||
module's request key, and no module's scope grants either — nor should it, since a module that
|
||||
only publishes events has no business declaring queues. So a module could serve tools and nothing
|
||||
in the mesh could call them
|
||||
([issue 049](../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)):
|
||||
not an operator at a terminal, not an agent acting for one.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Calling is a grant**: a module declares it may be asked, and a consumer is issued an
|
||||
account that may create a reply queue and publish to that module's request key. Deferred: a
|
||||
module-to-module call is the only caller that resembles what the mesh mints today, and none
|
||||
asks for one yet.
|
||||
2. **The control plane is the way in.** Adopted. It holds a connection that may already, so a
|
||||
person or an agent asks through it, and every question passes one process — which is where
|
||||
an audit of who asked what belongs.
|
||||
|
||||
## Decision
|
||||
|
||||
`mesh-controller ask <module> <tool> [json]` publishes the request on the tool exchange under
|
||||
`<module>.<tool>`, with a private reply queue bound under its own name, waits for the answer
|
||||
whose correlation matches, and prints it as the module gave it. A tool that answered with an
|
||||
error has answered: the answer is printed and the exit status says so. A module that never
|
||||
answers is said to have not answered, with where to look.
|
||||
|
||||
A module declares nothing about being asked: serving a tool is being askable through the control
|
||||
plane. A module-to-module call, if one is wanted, is a grant like any other and a later decision.
|
||||
|
||||
## Consequences
|
||||
|
||||
Anything with the control plane in reach can ask any module anything it serves. What got
|
||||
harder: nothing outside the control plane can, and the control plane's connection is one more
|
||||
thing on the path of every question — a cost accepted for the audit it buys.
|
||||
|
||||
> **The mechanism changed — 2026-09-30, by ADR 0152.** What stands: `ask` on the control plane, and
|
||||
> that every call passes an account whose permission list says what it may ask. What moved: "nothing
|
||||
> outside the control plane can" stopped being true when a person's account gained a publish grant per
|
||||
> tool (design 25 §7, 2026-09-28), and [ADR 0152](0152-the-operators-surface-is-a-module-the-console.md)
|
||||
> takes the first option above for a module as well — a manifest declares `invokes`, and the bus grants
|
||||
> exactly that publish side. The audit the second option bought is the bus's permission list, which
|
||||
> derives both.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A tools-only bed asks a served tool through the control plane and asserts an answer arrived —
|
||||
an error, since the lab has no upstream and no token, which is an answer where a timeout would
|
||||
not be.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 049](../04-ISSUES/049-a-module-can-serve-tools-and-nothing-can-call-them/00-report.md)
|
||||
- [ADR 0047](0047-a-module-runs-its-code-as-its-own-process-with-its-own-account.md)
|
||||
- [`03-DESIGN/01-to-be/19-the-module-protocol.md`](../03-DESIGN/01-to-be/19-the-module-protocol.md)
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
||||
---
|
||||
|
||||
# 96. An upstream image is copied between registries, never through a machine's image store
|
||||
|
||||
## Context
|
||||
|
||||
A module may declare that an artifact is an image published elsewhere, to be copied into the
|
||||
mesh's own registry so machines fetch it by a digest this mesh assigned rather than by a name
|
||||
somebody else controls. The builder pulled it into the build machine's image store and pushed it
|
||||
under the mesh's name, and the push was refused: a published image is an index over several
|
||||
architectures, the runtime's store keeps the index, and pushing one platform out of it fails
|
||||
however the platform is asked for
|
||||
([issue 046](../04-ISSUES/046-an-upstream-image-cannot-be-mirrored-into-the-mesh/00-report.md)).
|
||||
Every variant of pull-then-push was tried and failed the same way.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Resolve the index to one platform and push that.** Tried, reverted: it did not make the
|
||||
push work, and a workaround for a store's behaviour is a thing nobody removes later.
|
||||
2. **Tooling that copies between registries**, installed on the build machine. Rejected: one
|
||||
more thing the builder's image carries, for a protocol the builder already speaks for blobs.
|
||||
3. **Copy over the registry API**, in the builder. Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
The builder copies an upstream image between registries and never through a machine's image
|
||||
store: it reads the index and every manifest it names, moves each blob by digest into the mesh's
|
||||
registry — skipping what is already there, since blobs are content-named — puts the manifests
|
||||
and then the index under the module's repository, and pins the index's digest. Public images are
|
||||
read with the anonymous bearer token the registry hands out on challenge, which is how the
|
||||
public hub and the others the catalogue names serve them. The mesh mirrors the whole index, so
|
||||
what a machine fetches is the image for its own architecture; that every machine on one mesh is
|
||||
the same architecture is an assumption this mesh makes and had not written down until now.
|
||||
|
||||
Genesis has no registry to copy into and keeps the pull: the image stays in the first machine's
|
||||
store, named by its own id, as every artifact does before there is anywhere to publish.
|
||||
|
||||
## Consequences
|
||||
|
||||
An upstream artifact builds. What got harder: the builder now holds a registry client of its
|
||||
own, some two hundred lines, where a runtime command used to do; and a private upstream that
|
||||
demands a credential is refused, since the copy is anonymous by design.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A test raises a fake upstream registry serving an index over two platforms behind a bearer
|
||||
challenge, and a fake mesh registry that records what arrives: every blob of both platforms
|
||||
arrives once, two manifests and the index are put under their digests, the reference returned
|
||||
pins the index under the module's repository, and a second copy uploads nothing. A reference
|
||||
test reads names the way a runtime does. *Proven against the real thing the same day:* the genesis
|
||||
bed built the tool runtime through the mesh's builder with its node base copied out of the public
|
||||
hub into the mesh's registry by this code — after one finding the fake could not give: the builder
|
||||
ran on the default bridge, where loopback is not the machine, and now runs on the host network.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 046](../04-ISSUES/046-an-upstream-image-cannot-be-mirrored-into-the-mesh/00-report.md)
|
||||
- [ADR 0006](0006-the-substrate-and-the-control-plane.md)
|
||||
- [`03-DESIGN/01-to-be/18-building-a-module.md`](../03-DESIGN/01-to-be/18-building-a-module.md)
|
||||
@@ -1,76 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0096-an-upstream-image-is-copied-between-registries.md
|
||||
---
|
||||
|
||||
# 97. A vendor image is a declared build input, and a recipe fetches nothing undeclared
|
||||
|
||||
> **Superseded in part — 2026-10-08, by [ADR 0257](0257-a-copied-base-is-recorded-where-it-is-copied-kept-by-the-builds-that-stood-on-it-and-held-once-per-image.md).**
|
||||
> A declared image is no longer copied "under the module's repository": it is copied into one repository per
|
||||
> upstream image, whichever modules stand on it. Everything else decided here stands.
|
||||
|
||||
## Context
|
||||
|
||||
Three modules could not be built by the mesh's builder because their recipes reached for what
|
||||
no manifest named: a public package, or a binary copied out of a public image
|
||||
([issue 064](../04-ISSUES/064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md)).
|
||||
A module already names the bases it stands on — another module's artifact, by name — and the
|
||||
builder answers with what the mesh holds; a vendor's image had no such declaration, so a recipe
|
||||
named it directly and the build worked when the public registry answered, which is sometimes.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Let the build environment reach public registries.** Rejected: a build that fetches from
|
||||
somebody else's registry on its own is one the mesh cannot rebuild the same way twice.
|
||||
2. **A vendor image is a base like any other**, declared under `build.on` with the argument the
|
||||
recipe reads it from, pinned by digest, copied into the mesh's registry before the build
|
||||
([ADR 0096](0096-an-upstream-image-is-copied-between-registries.md)). Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A build's `on` entry is either a module's artifact or an image published elsewhere, pinned by
|
||||
digest, read from one build argument. Before the build the image is copied into the mesh's
|
||||
registry under the module's repository and the recipe is handed the copy; genesis, with no
|
||||
registry, pulls it into the first machine's store. A recipe whose `COPY --from` names a registry
|
||||
image the manifest did not declare is refused before the build, naming the image and the remedy;
|
||||
its own stages, declared arguments and `scratch` are not fetches. An unpinned vendor image is
|
||||
refused: a tag is what somebody else can move.
|
||||
|
||||
> **The mechanism changed — 2026-10-08, by [ADR 0257](0257-a-copied-base-is-recorded-where-it-is-copied-kept-by-the-builds-that-stood-on-it-and-held-once-per-image.md).**
|
||||
> A declared image is still copied into the mesh's registry before the build, and the recipe is still handed
|
||||
> the copy. What moved is where the copy goes. It no longer goes "under the module's repository"
|
||||
> (`<module>/on-<argument>`). It goes into one repository per upstream image, `upstream/<host>/<path>`,
|
||||
> whichever modules stand on it. Each copy is also recorded where it is made, so the store's sweep can
|
||||
> let it go.
|
||||
|
||||
A recipe whose `FROM` names an undeclared base was at first said, not refused: the mesh's own
|
||||
images — the control plane's, the tool runtime's, the route proxy's — started from a public base
|
||||
and declared none, and refusing those refuses genesis. *Amended the same day:* those three declare
|
||||
their bases now, and an undeclared `FROM` is refused like an undeclared copy. The builder's own
|
||||
image and the examples are built by `make`, not by the mesh, and take arguments with defaults.
|
||||
|
||||
The package half of the issue is not decided here: the mesh's package registry already proxies
|
||||
the public one, and the failure the report saw has to be run again to be placed.
|
||||
|
||||
## Consequences
|
||||
|
||||
A module's build inputs are all in its manifest, and every one of them is something the mesh
|
||||
holds a copy of. What got harder: a recipe that used to name a base image on its first line now
|
||||
names an argument, and the manifest names the image.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A builder test declares a pinned vendor image, asserts it is copied under the module's
|
||||
repository and handed to the recipe as the argument, and asserts an unpinned one is refused. A
|
||||
recipe test asserts an undeclared `FROM`, an undeclared `COPY --from` and an undeclared argument
|
||||
are named, and that stages, declared arguments and `scratch` are not.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 064](../04-ISSUES/064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md)
|
||||
- [ADR 0096](0096-an-upstream-image-is-copied-between-registries.md)
|
||||
- [`03-DESIGN/01-to-be/18-building-a-module.md`](../03-DESIGN/01-to-be/18-building-a-module.md)
|
||||
@@ -1,61 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0085-a-secret-is-a-provision.md
|
||||
---
|
||||
|
||||
# 98. A fact a provider makes at first start is fetched from it, not carried in its manifest
|
||||
|
||||
## Context
|
||||
|
||||
The catalogue's certificate authority declared its root certificate, its root key and that key's
|
||||
password as its own secrets, and told the container to initialise from them. The mesh mints an
|
||||
own secret nobody delivers as random bytes, and random bytes are not a certificate: issued, the
|
||||
authority could not start; only an operator hand-making its root could raise it
|
||||
([issue 076](../04-ISSUES/076-a-served-fact-made-at-first-start-cannot-be-served/00-report.md)).
|
||||
The authority can make its own root at first start. What it could not do then was tell the mesh
|
||||
what that root is: a consumer was given `${bound:acme-ca:root}` from the provider's `serves`,
|
||||
which is written in the manifest before anything runs.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A secret the module makes**, with the mesh taking custody once the file exists. Rejected
|
||||
for now: a node would have to send a value up to the mesh, which no channel does today, and
|
||||
a root key is the one thing the mesh has no reason to hold.
|
||||
2. **A served fact the provider contributes at run time.** Rejected for now: the same new
|
||||
channel, for a fact that is not secret at all.
|
||||
3. **The consumer fetches it from the provider**, over the mesh network, through a gate before
|
||||
the thing that needs it starts. Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
A provider's `serves` names where a fact made at first start can be fetched — the authority
|
||||
serves its root at a path beside its ACME directory — and a consumer fetches it in a `run-once`
|
||||
step declared before the resource that needs it, from the provider's bound address. The mesh
|
||||
network is where the fetch happens, which is what makes fetching without a prior trust
|
||||
acceptable: it is the network the mesh itself authenticates. The mesh mints only what it can
|
||||
make: the authority's password. The root key stays where it was made.
|
||||
|
||||
## Consequences
|
||||
|
||||
The catalogue's authority starts, and the proxy that requires it trusts what it fetched. What
|
||||
got harder: a consumer of such a fact carries one more resource, the gate that fetches it, and
|
||||
a fact that changes after first start is refetched only when the declaration changes.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The route-forwarding bed installs the authority, the proxy and a consumer from the catalogue and
|
||||
asserts a routed name is served through the proxy. The proxy refuses to start on a bundle that is
|
||||
not a certificate, so the name being served proves the gate fetched one; the gate itself refuses
|
||||
a body that is not a certificate. That the proxy obtains a certificate from this authority through
|
||||
that root is the certificate bed's proof, against the same authority with the same proxy. The
|
||||
catalogue-wide manifest test parses both manifests.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 076](../04-ISSUES/076-a-served-fact-made-at-first-start-cannot-be-served/00-report.md)
|
||||
- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md), [ADR 0085](0085-a-secret-is-a-provision.md)
|
||||
- [`03-DESIGN/01-to-be/08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md)
|
||||
@@ -1,94 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-21
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0052-a-step-that-runs-once-before-a-container.md
|
||||
---
|
||||
|
||||
# 99. A step that runs once names what it reads, and runs again when it changed
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md) has a consumer
|
||||
fetch a fact its provider made at first start through a run-once step: the route proxy fetches
|
||||
the certificate authority's root before it starts. A run-once step runs once per declaration
|
||||
([ADR 0052](0052-a-step-that-runs-once-before-a-container.md)): its marker is the digest of its
|
||||
own declaration, and a re-apply that finds the marker does nothing.
|
||||
|
||||
The provider can move. When the authority is assigned to another node it makes a new root there,
|
||||
and the mesh rewrites the consumer's binding file with the new address — but the step's own
|
||||
declaration has not changed, so the step does not run again, the proxy keeps the old root, and it
|
||||
refuses every certificate the new authority issues
|
||||
([issue 077](../04-ISSUES/077-a-fact-fetched-at-first-start-is-fetched-once/00-report.md)). A
|
||||
restart trigger was the natural remedy and was refused on a run-once step, on the ground that a
|
||||
step does not stay running to be restarted.
|
||||
|
||||
Two things the host already does point at the answer. What a container reads is part of what it
|
||||
is: a container's digest includes the digest of every resource it names under `restart-on`, so a
|
||||
rewritten file it reads is a changed container
|
||||
([issue 045](../04-ISSUES/045-a-container-keeps-the-values-it-started-with/00-report.md)).
|
||||
And a run-once step's marker *is* its digest. Nothing new is needed for the step to run again when
|
||||
what it reads changed; only the refusal stands in the way.
|
||||
|
||||
## Decision
|
||||
|
||||
**A run-once step may name what it reads under `restart-on`. For a step the word means *run
|
||||
again*: when a named resource changed in this apply, the step's digest has moved, its marker no
|
||||
longer matches, and it runs again — gating what follows, as it did the first time.** Nothing
|
||||
about the marker changes; the refusal of the pair is lifted, in the control plane and on the host.
|
||||
|
||||
**The container that consumes what a step made names the step.** A step that ran counts as a
|
||||
change, so a service that names it under `restart-on` is recreated after it, holding what the step
|
||||
fetched. Without this the step fetches a new root and the service keeps serving with the old one.
|
||||
|
||||
The route proxy's gate names the binding file it reads; the proxy's server names the gate and the
|
||||
binding. When the authority moves, the binding is rewritten, the gate fetches the new root, and the
|
||||
server is recreated with it — in one apply.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A provider epoch in the binding — the mesh raises a number when a provider is re-issued or
|
||||
moved, and the consumer's file carries it.** Rejected: the binding already changes when the
|
||||
provider moves (its address does), and a re-issue does not change what the authority serves —
|
||||
its state persists. An epoch would be a second signal for a change the file already shows.
|
||||
2. **The step runs before every start of the service, with no marker.** Rejected: every reconcile
|
||||
would run it, and a step that runs on every apply reads as a change on every apply, so the
|
||||
service naming it would be recreated every few minutes.
|
||||
3. **The proxy fetches the root itself, at start.** Rejected as the general answer: it fixes the
|
||||
proxy and leaves the next consumer of a fact made at first start to fix itself. The step is the
|
||||
general shape ([ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)).
|
||||
4. **Lift the refusal and read `restart-on` as *again* on a step.** Adopted: it is what the
|
||||
digest already does, and it needs no new word.
|
||||
|
||||
## Consequences
|
||||
|
||||
A fact fetched at first start follows its provider when the provider moves. What is not covered:
|
||||
a provider whose state is wiped behind the mesh's back, on the same node, makes a new fact that
|
||||
nothing the mesh knows reflects. That is not a change the mesh can see, and it is not claimed.
|
||||
|
||||
The one contradiction the refusal named is real and is now a documented reading: on a running
|
||||
container `restart-on` means recreate, on a step it means run again. Both are "this must reflect
|
||||
what it reads". This record is about a run-once *container*, the step ADR 0052 defined. A
|
||||
run-once process is a different shape whose marker does not carry what it reads; it is not
|
||||
covered here.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- mesh-host: a unit test declares a run-once step naming a file, records its marker against the
|
||||
file's old content, applies with the new content and asserts the step ran; applies again with
|
||||
nothing changed and asserts it did not. A second test declares a container naming a run-once
|
||||
step and asserts the container is recreated after the step ran, with the step as the stated
|
||||
reason.
|
||||
- mesh-controller: the manifest parser accepts a run-once step with `restart-on`; the
|
||||
catalogue-wide manifest test parses the route proxy's manifest, whose gate and server name what
|
||||
they read.
|
||||
- The route-forwarding bed still passes with the host that accepts the pair. No bed moves the
|
||||
authority: the mechanism is proven by the unit tests, the declaration by the manifest test.
|
||||
|
||||
## References
|
||||
|
||||
- [issue 077](../04-ISSUES/077-a-fact-fetched-at-first-start-is-fetched-once/00-report.md)
|
||||
- [ADR 0052](0052-a-step-that-runs-once-before-a-container.md), [ADR 0053](0053-a-step-that-runs-on-a-schedule.md), [ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)
|
||||
- [`03-DESIGN/01-to-be/08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md), [`03-DESIGN/01-to-be/20-writing-a-module.md`](../03-DESIGN/01-to-be/20-writing-a-module.md)
|
||||
@@ -1,228 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-22
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0078-the-store-and-broker-are-modules.md
|
||||
supersedes-in-part:
|
||||
- 02-DECISIONS/0088-the-foundation-filters-before-anything-listens.md
|
||||
---
|
||||
|
||||
# 100. A node in use is adopted before it is converged
|
||||
|
||||
> **Superseded in part — 2026-09-23, by [ADR 0105](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md).**
|
||||
> The rule that the private network's range must not overlap a tunnel the predecessor still runs now applies
|
||||
> only where the found tunnel is left running beside the mesh's. On the hub, where the mesh adopts the found
|
||||
> tunnel in place, there is one tunnel and one range. Everything else decided here stands.
|
||||
|
||||
## Context
|
||||
|
||||
The mesh replaces a predecessor mesh that is running, on the same machines, with the services
|
||||
people use. The control-node is the machine that already carries the predecessor's broker and
|
||||
build pipeline. The operator proposed the migration's shape: stop the predecessor's control on a
|
||||
machine, bring the mesh up there in an adoption mode that keeps the machine's configuration in
|
||||
force, migrate its modules one at a time, move to the next machine, and flip adoption off when every
|
||||
machine is done.
|
||||
|
||||
Measured on the control-node ([research 012,
|
||||
*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)):
|
||||
60 predecessor containers; the predecessor's control in user-level daemons separate from its
|
||||
services; its firewall active, with 54 incoming and 52 forwarding rules allowing each served port
|
||||
explicitly; 12 files its configuration sync writes, 3 of them system files.
|
||||
|
||||
Four things in the mesh as it stands break that shape:
|
||||
|
||||
1. **The base ruleset closes the machine.** Genesis loads a drop-by-default table
|
||||
([ADR 0088](0088-the-foundation-filters-before-anything-listens.md)). Every base chain at a hook
|
||||
runs in priority order; an accept ends only its own chain and a drop in any is final — whether
|
||||
the other firewall's chains are nftables or legacy iptables. So the predecessor's allowed ports
|
||||
would close at genesis.
|
||||
2. **The host replaces what it finds.** A declared file is written whatever is at its path, except
|
||||
a file declared create-once. A declared container replaces a running one of the same name. So
|
||||
assigning a module the predecessor also runs replaces the predecessor's service and its files at
|
||||
once.
|
||||
3. **Some foundation ports are held.** The foundation's store and bus ports are free — the
|
||||
predecessor publishes its own elsewhere — but the registry's port and the broker's management
|
||||
port are held, and on a control-node that is the private network's hub, so is the private
|
||||
network's port. The foundation's ports are fixed in the installer's bundle and the catalogue's
|
||||
manifests, so a collision surfaces as a container that fails to bind, and a port changed at
|
||||
genesis would be changed back when the foundation is adopted as modules.
|
||||
4. **Published container ports are forwarded, not received.** The foundation publishes its ports
|
||||
on every interface; a predecessor firewall that filters only incoming traffic never sees them.
|
||||
The base ruleset is what keeps the store unreachable from outside, and it is the thing that
|
||||
cannot be loaded.
|
||||
|
||||
The mesh already adopts in one place: the foundation's store and broker are taken over in place as
|
||||
modules, keyed on the container that is already running
|
||||
([ADR 0078](0078-the-store-and-broker-are-modules.md)). The research this record draws on already
|
||||
settled the conflict rule for adoption: *on conflict, what is on the machine stays*.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Cut each machine over in one go** — stop the predecessor's store, broker, registry and proxy,
|
||||
raise the foundation in their place. Rejected: every predecessor service on the machine is down
|
||||
until it has migrated, the predecessor's other machines lose their broker, and the rollback is
|
||||
restarting the predecessor — a recovery, not a step.
|
||||
2. **Put the control-node on a separate machine.** Rejected: the control-node is decided.
|
||||
3. **Converge on joining, as today.** Rejected: the base ruleset closes the machine at genesis, and
|
||||
the predecessor's services and files are replaced as soon as any module naming them is assigned.
|
||||
4. **Only make the foundation's ports configurable.** Rejected as insufficient: it answers the bind
|
||||
collisions and neither the firewall nor the files.
|
||||
5. **Treat assigning a module as migrating it.** Considered and rejected on review: it makes the
|
||||
rule that keeps found files never fire — the host only ever sees files of assigned modules — and
|
||||
it makes an assignment on an adopted node an outage rather than a preparation.
|
||||
6. **Adoption as a mode per node; each module taken explicitly; the node converged by an explicit
|
||||
flip.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A node is adopted or converged, and the controller records which.** The operator says so: at
|
||||
genesis for the control-node, and in the enrolment token for the others. The controller is
|
||||
authoritative, and every declaration it sends says whether the node is adopted and which modules
|
||||
have been taken on it. A node stays adopted until the operator converges it. An adopted node is
|
||||
said to be adopted wherever the mesh reports a node's state.
|
||||
|
||||
**A converged genesis refuses a machine in use.** A machine is in use when a container is running
|
||||
on it, or a port is listening on an address other than loopback that is not ssh's. Raised without
|
||||
saying adopted on such a machine, genesis refuses and names every container and listener it
|
||||
counted — a forgotten flag must not close a working machine.
|
||||
|
||||
**Before a node is adopted, its predecessor's control is stopped by the operator** — the daemons
|
||||
that write its configuration. Its services keep running on what they have.
|
||||
|
||||
**Found means present with no record.** A file at a declared path, or a container at a declared
|
||||
name, that the host's store has no record of writing is *found*. A file the host wrote in an
|
||||
earlier life of the node is not found; its record says so.
|
||||
|
||||
**On an adopted node, what is found is kept until its module is taken.** The host keeps a found
|
||||
file and a found container as they are, records the file's original content before anything else
|
||||
happens to it, and reports each as held. Assigning a module on an adopted node prepares it: what
|
||||
the module declares that is not found is created; what is found is held. **Taking a module** on a
|
||||
node is its cutover — the operator's act, done when that module's data has moved — and from then
|
||||
on the module's resources converge on that node like any other. What is held is never removed,
|
||||
even when its module is unassigned, and a held file or container that changes while held — a file
|
||||
rewritten, a container stopped or replaced — is reported as changed by something else, not reverted
|
||||
or restarted: that is how a predecessor still writing is caught.
|
||||
|
||||
**The firewall found on the machine stays in force.** The mesh loads no table on an adopted node
|
||||
that drops by default or holds an accept — neither genesis's base ruleset nor the filter module's
|
||||
derived one. What the mesh needs reachable is declared as **openings**: a resource that says a port
|
||||
is reachable, from where, on the incoming path or the forwarded path — a published container port is
|
||||
forwarded. The controller derives them from the same inputs as the filter, each from where the
|
||||
filter would admit it: the `listens` of the modules assigned there, the private network's hub port
|
||||
and the bus and the registry from anywhere, the store's port and the broker's management port from
|
||||
the private network. The host
|
||||
converges an opening through the found firewall in that firewall's own terms, marks it as the
|
||||
mesh's, and removes only what it marked; it re-checks each opening on every reconcile, so a reload
|
||||
or a reboot of the found firewall does not lose it for longer than one reconcile. An opening is a
|
||||
state, not a command, which is what lets it travel over the link. The host reports which firewall
|
||||
it found. A machine with no firewall needs no openings; a machine with a kind no host speaks is
|
||||
refused adoption.
|
||||
|
||||
**The mesh guards its own ports itself, in a table of its own that only refuses.** It passes
|
||||
everything by default and holds nothing but refusals, and the two ports it refuses are the
|
||||
foundation's own, checked free at genesis, so it cannot close anything the machine serves; it is
|
||||
the mesh's, so the found firewall reloading does not touch it. It refuses the store's port and the
|
||||
broker's management port except from the private network and from the machine itself — its
|
||||
loopback and the container runtime's own networks, known by the interface a packet arrives on and
|
||||
never by its source address alone — at the prerouting hook, ahead of the runtime's
|
||||
destination translation, so it matches the port the packet was sent to, for both address
|
||||
families. The bus and the registry stay reachable from anywhere, as a node
|
||||
enrols over the bus and pulls from the registry before it has a private-network address
|
||||
([ADR 0088](0088-the-foundation-filters-before-anything-listens.md)); so does the private network's
|
||||
hub port. The store is unreachable from outside whatever the found firewall does, and on a machine
|
||||
with none.
|
||||
|
||||
**The foundation's ports are the node's.** Every port the foundation binds is an input to genesis,
|
||||
checked free before anything is raised, refused with the name of what holds it. The ports given
|
||||
become that node's settings for the foundation's modules — the catalogue's numbers are only their
|
||||
defaults — and every place that uses them reads them from there: the modules' containers, the
|
||||
filter, the base ruleset, the private network's endpoint, and the addresses consumers are given.
|
||||
The private network's address range must not overlap a tunnel the predecessor still runs; genesis
|
||||
checks that too.
|
||||
|
||||
**Converging a node is one act, previewed.** It refuses while an assigned module still holds a
|
||||
found container: each service is taken on its own, when its data has moved, never by the flip. The
|
||||
preview lists what is reachable on the machine now — every listening socket and every published
|
||||
container port — and for each whether an assigned module declares it or it will close, and every
|
||||
module the flip will take, with the held files each will replace. The flip then takes those modules, loads the
|
||||
mesh's derived filter in place of its refusal-only table, and retires the found firewall by
|
||||
disabling it, never by flushing: the container runtime's rules and the found firewall's own
|
||||
configuration stay on disk. Returning a converged node to adopted unloads the derived filter,
|
||||
restores the refusal-only table, enables the found firewall again and converges the openings
|
||||
through it once more; what was taken stays taken. A
|
||||
node converges when its migration is done; the mesh is migrated when every node has converged.
|
||||
|
||||
**The order is the operator's:** the control-node first, adopted, its modules assigned and taken
|
||||
one at a time; then each other machine, adopted, migrated, converged in turn. The predecessor's
|
||||
pipeline runs on the control-node, so its updates stop for every machine while the migration runs;
|
||||
that is accepted.
|
||||
|
||||
## Consequences
|
||||
|
||||
Each step says what it changes before it changes it. Adopting a node changes nothing that serves;
|
||||
assigning a module adds what is not there; taking a module replaces one service; the flip replaces
|
||||
the firewall, after naming every port it will close. Two steps are not undone by the mesh: taking a
|
||||
module replaces the predecessor's container, and the kept original of a file is recorded but not
|
||||
yet restored by any act of the mesh
|
||||
([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) leaves where it lives
|
||||
open).
|
||||
|
||||
What got harder:
|
||||
|
||||
- **The mesh must speak a firewall it did not install**, on both the incoming and the forwarded
|
||||
path. One kind is found on the machines measured; another is refused until a host speaks it. The
|
||||
mesh's own guard does not depend on it: that table is the mesh's.
|
||||
- **The host gains a guard it did not have** — keep what you found — and its report must say
|
||||
which files and containers it holds, or an adopted node reads as converged.
|
||||
- **The declaration gains a node's mode and its taken modules**, and a resource, the opening.
|
||||
- **Genesis grows inputs, and they outlive genesis.** The foundation's ports stop being constants;
|
||||
every reader of them reads the node's settings.
|
||||
- **A node can sit adopted indefinitely.** Nothing forces the flip; the mesh's status says which
|
||||
nodes are adopted, so one left behind is visible.
|
||||
- **This narrows [ADR 0088](0088-the-foundation-filters-before-anything-listens.md) for adopted
|
||||
nodes**: the base ruleset is not loaded on a node raised adopted, and its duty — the store never
|
||||
reachable from outside — passes to a table of the mesh's that only refuses.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A lab bed prepares a machine the way the predecessor leaves one: its firewall allowing a served
|
||||
port and denying the rest, a service container listening on that port under a name a catalogue
|
||||
module also uses, a file at a path that module declares, a stand-in for the predecessor's control
|
||||
that would rewrite that file, and a container holding the registry's port. Then:
|
||||
|
||||
- **Genesis converged** on it refuses and names every container and listener it counted.
|
||||
- **Genesis adopted, with the registry's port held**, refuses and names the holder; with another
|
||||
port given, the foundation comes up — and adopting the foundation as modules leaves it on that
|
||||
port.
|
||||
- **Nothing that serves changed**: the service is reachable from a second machine, the file is byte
|
||||
for byte what it was, and the found firewall's rules differ only by rules marked as the mesh's.
|
||||
- **The store is unreachable from outside** — probed from a machine off the private network, and
|
||||
again after the found firewall is reloaded — and reachable over it and from a container on the
|
||||
node itself; the bus is reachable from a machine that has not yet enrolled.
|
||||
- **The mesh works through the found firewall, and keeps working after it is reloaded and after the
|
||||
machine reboots**: the second machine enrols, and the openings are there again.
|
||||
- **A predecessor still writing is caught**: with the stand-in left running, the held file's change
|
||||
is reported and not reverted.
|
||||
- **Assigning prepares, taking cuts over**: the module assigned holds the found container and file;
|
||||
taken, it replaces them and its port is opened.
|
||||
- **Converging previews, then changes**: it refuses while the service's module holds its found
|
||||
container; once that module is taken, the preview names the service's port and a published port no
|
||||
firewall rule mentions, and the modules it will take; after the flip the mesh's derived filter is
|
||||
loaded, the found firewall is disabled with its configuration still on disk, the declared port is
|
||||
open and the undeclared one closed. Returned to adopted, the found firewall is enabled again and
|
||||
the derived filter is gone.
|
||||
|
||||
Unit tests hold the host to keeping a found file and container on an adopted node, converging them
|
||||
once taken, never removing what it holds, and reporting a held file or container that changed;
|
||||
genesis to refusing a held port and a converged raise on a machine in use; the controller to
|
||||
carrying the mode and the taken modules in every declaration, deriving the openings, and refusing
|
||||
a flip while a found container is held.
|
||||
|
||||
## References
|
||||
|
||||
- [research 012 — the minimum viable node, and adopting what is already there](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md),
|
||||
and its document [*migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)
|
||||
- [ADR 0005](0005-the-node-host.md), [ADR 0011](0011-managed-files-are-generated-never-edited.md),
|
||||
[ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0088](0088-the-foundation-filters-before-anything-listens.md)
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-22
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
---
|
||||
|
||||
# 101. A machine's own resolver does not make it in use
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) has a converged genesis
|
||||
refuse a machine in use. It defines *in use* as a container running, or a port listening on an
|
||||
address other than loopback that is not ssh's. That definition was written before anything was
|
||||
measured.
|
||||
|
||||
Measured on a freshly installed lab machine, running nothing but its operating system:
|
||||
|
||||
| Listening | Held by |
|
||||
|---|---|
|
||||
| TCP and UDP on every address, the link-local name resolution port | the system's name resolver |
|
||||
| UDP on every address, the multicast name resolution port | the system's name resolver |
|
||||
| UDP on a link-local address, the address-configuration client port | the system's network manager |
|
||||
| TCP and UDP on loopback | the resolver's stub and the container runtime |
|
||||
|
||||
By ADR 0100's words, the resolver's TCP listener on every address makes **every** freshly
|
||||
installed machine a machine in use. A converged genesis would refuse them all, and `--adopted`
|
||||
would become the only way to raise anything. The refusal exists to catch a forgotten flag on a
|
||||
working machine. Refusing an empty one defeats it, and teaches operators to pass the flag by
|
||||
habit.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep the words.** Rejected: every fresh machine is refused.
|
||||
2. **Count TCP only, ignore UDP.** Rejected: the resolver listens on TCP too, and a machine that
|
||||
serves over UDP alone, a resolver or a tunnel, is in use.
|
||||
3. **Ignore listeners held by the operating system's own network daemons**, a short named list,
|
||||
on both protocols. Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A listener held by one of the operating system's own network daemons does not make a machine
|
||||
in use.** The daemons are the ones the measurement found: the name resolver and the network
|
||||
manager, named in the installer's code beside that measurement. Everything else in ADR 0100's definition stands: a running container, or any other
|
||||
listener on an address other than loopback that is not ssh's, makes the machine in use, and
|
||||
genesis still names every one it counted.
|
||||
|
||||
A daemon is added to the list only with a measurement of a fresh machine that holds it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A converged genesis on a fresh machine goes ahead, as it did before ADR 0100.
|
||||
- A machine whose resolver is also serving other machines is not counted as in use by its
|
||||
resolver alone. It is one of the daemons that serves nobody on a fresh machine, and the one
|
||||
kind of service this lets through.
|
||||
- The list is code, not configuration, so it changes by review.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A unit test in the installer feeds the listeners captured from the fresh machine, as the
|
||||
listening-socket tool printed them, and asserts the machine is not in use. The existing tests
|
||||
still assert that a serving machine is in use and that every container and listener is named.
|
||||
The adoption lab bed raises a converged genesis on a fresh machine and asserts it is not refused.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)
|
||||
- [research 012, *migrating a node that is in use*](../01-RESEARCH/012-the-minimum-viable-node/migrating-a-node-in-use.md)
|
||||
@@ -1,102 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-22
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
---
|
||||
|
||||
# 102. The mesh writes into a shared file, never over it
|
||||
|
||||
## Context
|
||||
|
||||
[Issue 084](../04-ISSUES/084-taking-networking-on-an-adopted-node-restarts-every-container/00-report.md)
|
||||
found that the networking module declares the container runtime's configuration file to state
|
||||
one fact in it: the mesh's registry is trusted over the private network. Reading the code
|
||||
closer showed it worse than reported. The controller merges an operator's settings into the
|
||||
module's own content, but the host writes the result **whole**. Whatever the machine had in that
|
||||
file is replaced, including where the runtime keeps its data. On a machine in use, that is every
|
||||
image and container gone from the runtime's view at its next start. The runtime's service is then
|
||||
restarted, which stops every container on the machine.
|
||||
|
||||
Measured on a lab machine: the runtime takes a new list of trusted registries on a reload, with
|
||||
no restart, and a running container with no restart policy keeps running through it. The
|
||||
runtime's log says it reloaded its configuration, and the registry reads as trusted afterwards.
|
||||
|
||||
Under [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md), the file is found
|
||||
on an adopted node and held until networking is taken. Holding it is safe, but it means an
|
||||
adopted machine cannot pull the mesh's images until then, and taking networking would restart
|
||||
the runtime.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Keep writing the whole file; take networking as a cutover of its own.** Rejected: it
|
||||
still replaces the machine's settings, and the cutover stops everything on the machine.
|
||||
2. **A drop-in the runtime reads beside its main file.** Rejected: the runtime has no such
|
||||
directory for its daemon settings.
|
||||
3. **Write into the file: set the mesh's keys, keep the rest; reload, don't restart.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A file the mesh shares with software it did not install is written into, never over.** A
|
||||
file resource may say it is written *into* a structured file. The host then reads what is there,
|
||||
sets only the keys the mesh declares, keeps every other key as it found it, and records what each
|
||||
of its keys held before. **A list is added to, never replaced**: where the machine already has a
|
||||
list under a key the mesh declares, the mesh's members are added to it and the host records
|
||||
exactly which members it added — setting the key would replace the operator's own list, the harm
|
||||
this record exists to prevent. Undeclared later, each key goes back to what it held and each
|
||||
added member is taken out again, and a file the mesh created is removed only if nothing but its
|
||||
own keys is left. A file written into replaces nothing, so on an adopted node it is never held:
|
||||
it is written whether or not its module has been taken.
|
||||
|
||||
**Whatever the host writes over without a record of it, it keeps first.** On any node, adopted
|
||||
or converged, before the host writes a file over one it has no record of making, it keeps the
|
||||
original once and says where; if it cannot keep it, it does not write. A file the mesh takes over
|
||||
is then never lost, whatever put it there.
|
||||
|
||||
**A service that re-reads its configuration on a reload is reloaded, not restarted.** A service
|
||||
resource may name what it must be *reloaded* on, beside what it must be restarted on. The
|
||||
container runtime is reloaded for the registry's trust.
|
||||
|
||||
**The networking module writes the runtime's trust into its file and reloads it.** An adopted
|
||||
node therefore trusts the mesh's registry as soon as it is on the private network, and taking
|
||||
networking no longer touches the runtime. The hosts file networking writes is still written whole
|
||||
and stays held until networking is taken; a converge preview names it among the files it replaces.
|
||||
|
||||
> **The mechanism changed — 2026-10-05, by [ADR 0222](0222-a-module-is-told-where-a-mesh-seats-holder-is-reached-and-the-controller-writes-no-file-a-seats-holder-owns.md).**
|
||||
> Writing into a shared file, adding to a list and reloading rather than restarting all stand. What
|
||||
> moved is the writer: the networking module no longer writes the runtime's trust or declares its
|
||||
> service. The container runtime's own module writes it into its own file and reloads its own
|
||||
> service ([issue 190](../04-ISSUES/190-the-runtimes-configuration-is-written-by-modules-that-are-not-the-runtime/00-report.md)).
|
||||
> The controller test named under *How it is checked* below, which held the networking module to
|
||||
> declaring the runtime's file, is replaced by one holding the networking module to declaring neither,
|
||||
> and by one holding the runtime's module to the trust.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The runtime's file on a machine in use keeps its data directory, its logging settings and
|
||||
everything else the predecessor set.
|
||||
- A host that does not know *into* or *reload-on* refuses a declaration carrying them, so hosts
|
||||
are upgraded before the controller that emits them — the same order ADR 0100 needs.
|
||||
- One shape more for every host: a file written into a structured document. Only JSON is spoken;
|
||||
another format is refused until written.
|
||||
- The hosts file remains a whole file. Writing a marked block into it is the same idea for a text
|
||||
file and is not decided here.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Unit tests hold the host to setting only the declared keys and keeping the rest, restoring each
|
||||
key and removing only a file it created when the resource is undeclared, adding to a list and
|
||||
removing only the members it added, keeping the original of a file it writes over without a
|
||||
record, refusing a file that is not a JSON object rather than overwriting it, never holding a file written into on an adopted
|
||||
node, and reloading rather than restarting a service whose reload-on resource changed. A test in
|
||||
the controller holds the networking module to declaring the runtime's file written into and the
|
||||
runtime reloaded. The adoption lab bed gives the machine a runtime file with a setting of its own
|
||||
and asserts it survives adoption with the registry trusted added, and that a container without a
|
||||
restart policy is still running afterwards.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 084](../04-ISSUES/084-taking-networking-on-an-adopted-node-restarts-every-container/00-report.md)
|
||||
- [ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)
|
||||
@@ -1,112 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-22
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
---
|
||||
|
||||
# 103. What an adopted node holds, and what its guard refuses
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) holds a found file or
|
||||
container until its module is taken. It also has the mesh guard the store's port and the
|
||||
broker's management port with a table that only refuses. Two independent reviews of the build
|
||||
found both rules drawn too narrowly and, for the guard, in the wrong place:
|
||||
|
||||
1. **Other kinds of resource reach what was found.** An untaken module's directory is re-owned
|
||||
and re-moded if a predecessor's directory is already at its path, and a database refuses to
|
||||
start on a data directory whose mode changed. A service of the same name as a predecessor's
|
||||
unit is started, stopped or re-enabled. An action run *in* a held container runs inside the
|
||||
predecessor's service. A container that is not found under its own name is created, and can
|
||||
mount the predecessor's data beside the predecessor's own container.
|
||||
2. **The guard's two ports are not the only ones a found firewall misses.** A published
|
||||
container port is forwarded, not received, and a firewall that filters only incoming traffic
|
||||
never sees it (ADR 0100's own Context). The broker's plaintext port is published on every
|
||||
interface and admitted by the filter from the private network only. So on an adopted node it
|
||||
is reachable from anywhere. The same holds for any published port a module restricts to the
|
||||
private network.
|
||||
3. **The guard refuses by port alone.** On a machine that routes for others, such as a
|
||||
predecessor's private-network hub, a packet for another machine's database port is refused as
|
||||
well. And a port the guard refuses for a module that is assigned but not taken may still be
|
||||
the predecessor's own, serving the predecessor's other machines.
|
||||
|
||||
## Decision
|
||||
|
||||
**Found covers every kind that can reach what the machine already has.** On an adopted node,
|
||||
for a module not yet taken:
|
||||
|
||||
- a **directory** present with no record is held: its mode and owner are left, and nothing in it
|
||||
is touched;
|
||||
- a **service** with no record is held when an administrator installed its unit — the service
|
||||
manager reads the unit from outside the packages' own directory — or when the machine uses it,
|
||||
running or started at boot. Its state and whether it starts at boot are then left. A reload
|
||||
named by the module still happens, since a reload stops nothing
|
||||
([ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)). A unit a package merely
|
||||
ships that nothing runs and nothing enables is not found: the private network's own tunnel unit
|
||||
is one, and holding it kept the private network from ever coming up;
|
||||
- an **archive** whose target is present with no record is not unpacked or re-owned; a
|
||||
**process** whose unit file is present with no record is not written over or restarted; a
|
||||
**user** that exists with no record keeps its shell and groups;
|
||||
- a **container** not found by its name is still held if it would mount a path or a volume that
|
||||
is present with no record, because it would share the predecessor's data;
|
||||
- an **action** or one-off step run *in* a held container is held until the module is taken.
|
||||
|
||||
What is held is reported as held and is never removed, as ADR 0100 decides for files and
|
||||
containers.
|
||||
|
||||
**The guard refuses what the found firewall may miss, for taken modules only.** Its ports are
|
||||
derived from each *taken* module: every machine port it publishes that the filter would admit
|
||||
from the private network only, and the ports its manifest names under `guards` — which is how
|
||||
the store's port and the broker's management port are included. Only TCP is guarded. A port of a module that is assigned but not taken is not guarded: it may still
|
||||
be the predecessor's. The guard matches only packets addressed to this machine. Traffic the
|
||||
machine routes for others is never its business.
|
||||
|
||||
**An opening already answered by a found rule is not added.** If the found firewall already
|
||||
admits what an opening says, the opening is reported as satisfied by the found rule, and the
|
||||
mesh adds nothing and later removes nothing. The found firewall treats two rules differing only
|
||||
in their action, log setting or comment as one rule, so adding the mesh's would take over the
|
||||
operator's. Where the found rule is such a twin but is not a plain allow — a deny, a limit, a
|
||||
logged allow — the opening is refused, naming that rule, and nothing is added.
|
||||
|
||||
**A machine raised adopted stays adopted if genesis is run again.** The installer reads the
|
||||
node's mode from what the machine records, not only from the operator's flag. A run without
|
||||
the flag on an adopted machine is refused.
|
||||
|
||||
**What of ADR 0100 this replaces.** ADR 0100's guard refused two ports, the foundation's own,
|
||||
checked free at genesis, and could therefore close nothing the machine served. That guard is
|
||||
replaced by the one above: a guarded port of a taken module may be one the predecessor served
|
||||
more widely, and taking the module narrows it to the private network. Everything else in ADR 0100
|
||||
stands, and its rules for files and containers now cover the other kinds listed here.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Taking a module can narrow a port the predecessor served to anyone: the guard then refuses it
|
||||
from outside the private network, as the module declares. Nothing says so yet
|
||||
([issue 086](../04-ISSUES/086-taking-a-module-narrows-a-port-without-saying-so/00-report.md)).
|
||||
- An untaken module on an adopted node can come up only beside what was found, never on top of
|
||||
it: whatever would share the predecessor's data waits for the cutover.
|
||||
- The guard grows with the modules taken, and a module's published private-network port is
|
||||
protected on an adopted node the way the derived filter protects it on a converged one.
|
||||
- Guarding only taken modules means a foundation port on a node joining adopted is guarded once
|
||||
its module is taken, not when it is assigned. Genesis takes the foundation's modules, so the
|
||||
control-node's store is guarded from the first push.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Unit tests hold the host to holding a found directory, a found service, a found archive, process
|
||||
and user, and a container that would mount found data; to not holding a unit nothing runs; to
|
||||
deferring an action run in a held container; to adding no opening a found rule answers, and to
|
||||
refusing one a conflicting found rule would absorb. They hold the controller to deriving the guard from taken modules'
|
||||
private-network published ports, and the guard's text to matching only this machine's
|
||||
addresses. The installer's tests hold a re-run without the flag on an adopted machine to a
|
||||
refusal. The adoption lab bed asserts that the broker's plaintext port is unreachable from
|
||||
outside the private network even with the found firewall admitting it, and that an operator's
|
||||
own rule for a port the mesh opens survives the opening being removed.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md),
|
||||
[ADR 0102](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-22
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
---
|
||||
|
||||
# 104. A provision may be answered by an adapter to the predecessor
|
||||
|
||||
## Context
|
||||
|
||||
[Issue 093](../04-ISSUES/093-the-successor-proxy-cannot-serve-what-the-predecessor-still-serves/00-report.md),
|
||||
found on the first adopted node hours after it was raised. Every module reachable by name requires
|
||||
a **route**. The mesh has one provider of it, its own proxy, which binds the two public ports on
|
||||
the machine's own network. The predecessor's proxy holds those ports and serves every public name
|
||||
there. So:
|
||||
|
||||
- a web module cannot be taken until the mesh's proxy runs;
|
||||
- the mesh's proxy cannot run until the predecessor's stops;
|
||||
- and when it stops, every name the predecessor served goes dark, because the mesh's proxy serves
|
||||
only what mesh modules have contributed.
|
||||
|
||||
Adoption ([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md)) keeps a *file* as it
|
||||
was and a *container* as it was. It has nothing for a service whose successor is **different
|
||||
software**: nothing shares a name to hold, so there is only replacement, of every name at once.
|
||||
|
||||
The certificates decide the rest. The predecessor's proxy holds them and renews them. A successor
|
||||
taking over first would have to obtain a certificate for every public name in the same window in
|
||||
which it took the ports — with nothing else migrated yet, and the only way back being to start the
|
||||
predecessor again.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The successor proxy serves routes it did not derive**, given as the operator's configuration,
|
||||
each disappearing as the module owning that name contributes its own. Rejected for the
|
||||
migration: it puts the riskiest step first, and it puts configuration the mesh does not own
|
||||
into the one component whose design is that the mesh derives its configuration.
|
||||
2. **Take the proxy first and accept the window.** Rejected: every public name at once, before
|
||||
anything else has moved, with certificates to obtain in the same window.
|
||||
3. **Make the requirement optional**, so a module can be taken while nothing provides route.
|
||||
Rejected: it is not optional. A module that says it must be reachable and is not is a module
|
||||
reporting success while serving nobody.
|
||||
4. **An adapter module provides the provision by writing into the predecessor's service.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A provision whose only provider owns a scarce machine-wide resource may be answered, during a
|
||||
migration, by an adapter module that writes into the predecessor's own configuration.**
|
||||
|
||||
For the route: a module that **provides `route`**, receives every contribution as the proxy does,
|
||||
and writes each one as a route file where the predecessor's proxy reads its configuration —
|
||||
pointing at the machine port the contributing module now publishes. The predecessor's proxy keeps
|
||||
serving every name it already serves, keeps its certificates and keeps renewing them; a name whose
|
||||
module has migrated is served by the same proxy, pointing at the mesh's container instead of the
|
||||
predecessor's.
|
||||
|
||||
**It is migration scaffolding and says so.** It is assigned only on an adopted node, it writes only
|
||||
files it can name as its own, and it is removed when the predecessor's proxy retires — at which
|
||||
point the mesh's own proxy takes the ports and already knows every route, because by then every
|
||||
name belongs to a module that contributes it.
|
||||
|
||||
**This does not make the mesh the predecessor's manager.** The adapter writes route files and
|
||||
nothing else, and only because the predecessor's control has been stopped by the operator, which
|
||||
ADR 0100 requires before a node is adopted.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The migration keeps its shape: one service at a time, each reversible on its own. The proxy is
|
||||
the **last** cutover again, not the first.
|
||||
- There is no certificate event until the end, and at the end there is one, for a proxy that
|
||||
already has every route from the mesh.
|
||||
- The mesh writes into a directory the predecessor owns. That is safe only while the predecessor's
|
||||
control is stopped, and it is why the adapter is refused on a converged node.
|
||||
- The adapter is throwaway code with a stated end. Something has to delete it; that something is
|
||||
the operator, and the flip is the moment.
|
||||
- A second provision with this shape — a resolver on 53, a mail relay on 25 — now has a pattern to
|
||||
follow rather than a new argument to have.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Unit tests hold the adapter to writing one route file per contribution, naming each file as its
|
||||
own, removing a file when its contribution goes, and leaving every file it did not write. The
|
||||
controller is held to resolving `route` from it exactly as from the proxy. The adoption lab bed
|
||||
gains a step: with the predecessor's proxy serving a name, a module taken behind the adapter is
|
||||
reachable under that name, and the predecessor's own names keep answering. Assigning the adapter
|
||||
to a converged node is refused.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 093](../04-ISSUES/093-the-successor-proxy-cannot-serve-what-the-predecessor-still-serves/00-report.md)
|
||||
- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md), [ADR 0007](0007-connectivity.md)
|
||||
@@ -1,148 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-23
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
supersedes-in-part:
|
||||
- 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
|
||||
---
|
||||
|
||||
# 105. The mesh adopts the predecessor's tunnel in place
|
||||
|
||||
## Context
|
||||
|
||||
The control-node is adopted and its store and broker have been moved onto the ports the
|
||||
predecessor served them on, so the predecessor's other machines keep reaching them. They reach
|
||||
them **over the predecessor's tunnel**: a WireGuard interface on the control-node with three
|
||||
peers, an address range, and a port the hosting provider already lets through. The mesh's own
|
||||
private network runs beside it on a second interface, a second range and a second port — one the
|
||||
provider does not let through, so no other machine can join the mesh
|
||||
([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md), the runbook's
|
||||
measurement of the upstream filter).
|
||||
|
||||
[ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) says the mesh's range *must not
|
||||
overlap a tunnel the predecessor still runs*. That was written for coexistence. It leaves the
|
||||
migration with two tunnels for as long as any predecessor machine exists, and the second one
|
||||
unreachable.
|
||||
|
||||
The mesh already adopts in place where the thing found is the thing it would have raised: the
|
||||
store and the broker were taken over as modules, keyed on the container that was running
|
||||
([ADR 0078](0078-the-store-and-broker-are-modules.md)). A WireGuard interface is the same shape: a
|
||||
private key, a listening port, a list of peers by public key, an address. The mesh's is not
|
||||
different in kind from the predecessor's; it is a second one.
|
||||
|
||||
The operator's instruction: take over the tunnel interface, same range — everything stays the same.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Two tunnels until the last predecessor machine is gone.** Rejected: the mesh's stays
|
||||
unreachable from outside, so no machine can join, so the last predecessor machine is never gone.
|
||||
2. **Move the mesh's tunnel onto the predecessor's port with the mesh's own key and range.** The
|
||||
peers' packets arrive and are dropped — WireGuard authenticates by key, and the mesh's key is not
|
||||
the one they know. Every other machine loses its tunnel until it enrols, and it enrols over a bus
|
||||
it reaches through that tunnel. Rejected.
|
||||
3. **Adopt the predecessor's tunnel in place: its private key, its peers, its range, its port.**
|
||||
Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**On an adopted node that is the hub, the private network takes over the tunnel it finds.** The
|
||||
mesh's interface is raised with the found interface's **private key**, on its **port**, with its
|
||||
**address and range**, and every **peer** the found interface had — public key, allowed address —
|
||||
carried into the mesh's peer list as a peer not yet enrolled. The found interface is stopped, never
|
||||
flushed; its configuration stays on disk, kept like any held file.
|
||||
|
||||
**Nothing a peer knows changes.** A predecessor machine keeps the same server key, the same
|
||||
endpoint, the same address and the same route; it cannot tell the tunnel changed hands. When that
|
||||
machine enrols, it keeps its address: the mesh assigns an enrolling node the address the tunnel
|
||||
already had for its key, and only a node with no such address is given a fresh one from the range.
|
||||
|
||||
**The mesh's own addresses are the range's.** The controller composes every node's private address
|
||||
from the tunnel it holds, so adopting the predecessor's range moves the mesh's addresses with it —
|
||||
the hub's, and every binding, hosts-file entry and endpoint derived from it. Those are readers of
|
||||
the setting; they follow it, per ADR 0100's rule for ports. A reader that does not follow is
|
||||
[issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md).
|
||||
|
||||
**This narrows ADR 0100.** Its rule that the range must not overlap a tunnel the predecessor still
|
||||
runs applies to a node that is *not* adopting the tunnel: where the found tunnel is left running
|
||||
beside the mesh's, the ranges must differ. Where it is adopted, there is one tunnel and one range.
|
||||
|
||||
**The guard's question answers itself.** [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
|
||||
admits the mesh's own ports only from the private network's interface. With one tunnel, the
|
||||
predecessor's peers arrive on it, and nothing needs to be admitted from an interface the mesh does
|
||||
not own.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The order of a migration changes.** The hub's tunnel can be taken as soon as the node is
|
||||
adopted, before any service — and should be, because it is what lets other machines join. The
|
||||
runbook's step order is amended.
|
||||
- **The hub takes the provider's open port for free.** The port the predecessor's tunnel used is
|
||||
by definition one the provider passes.
|
||||
- **A peer's identity precedes its enrolment.** The mesh holds public keys and addresses for
|
||||
machines it has no record of. They are peers of the tunnel, not nodes of the mesh, until they
|
||||
enrol; the registry must be able to say both.
|
||||
- **What got harder:** the private key of the found interface is read from the machine and becomes
|
||||
the mesh's — the one case where the mesh takes a credential it did not mint. It is sealed like
|
||||
any own secret from then on, and the found configuration file is kept, not copied further.
|
||||
- Two documents currently say the opposite: ADR 0100's non-overlap rule (narrowed above) and the
|
||||
runbook's port plan, which is amended with this record.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A lab bed prepares a hub the way the predecessor leaves one: a WireGuard interface with a key, a
|
||||
port, a range and two peers, each peer a second machine that reaches a service on the hub through
|
||||
the tunnel. Then:
|
||||
|
||||
- **Adopted, the tunnel changes hands and the peers notice nothing**: the found interface is down
|
||||
and its file is on disk; the mesh's interface is up with the found key, port and address; each
|
||||
peer's service call succeeds before, during and after, with no reconfiguration on the peer.
|
||||
- **A peer enrols and keeps its address**: the machine joins the mesh over the tunnel it already
|
||||
has, and its node address is the one the tunnel held for it.
|
||||
- **A new machine gets a fresh address from the same range**, and reaches both the hub and the
|
||||
enrolled peer.
|
||||
- **Nothing derived from the address is stale**: every binding, hosts entry and endpoint the
|
||||
controller composes says the adopted range, before and after a push.
|
||||
|
||||
Unit tests hold the controller to reading the hub's address and range from the adopted tunnel,
|
||||
assigning an enrolling node the address its key already had, and refusing to hand out an address
|
||||
the tunnel already holds; and the host to raising the mesh's interface with the found key and
|
||||
peers and stopping the found interface without flushing it.
|
||||
|
||||
## What review settled that this record did not
|
||||
|
||||
*Added 2026-09-24, from the review of the implementation. A decision record is not edited to change
|
||||
its meaning; this says what was decided under it.*
|
||||
|
||||
- **A spoke's view of its hub is not a peer the mesh carries.** A predecessor gives a spoke the
|
||||
whole subnet through the hub, so the spoke's found tunnel names one peer routed a range rather
|
||||
than an address. Only the hub's peers are ever carried; a spoke presenting its own is skipped,
|
||||
not refused — a machine enrols with what it found, and what it found is its route home.
|
||||
- **The range and the carried peers outlive the flip.** They follow from the hub having taken the
|
||||
tunnel over — its key being the tunnel's — and not from the node being adopted. A converged hub
|
||||
keeps the range it adopted and the addresses it is holding, and `AssignAddress` keeps excluding
|
||||
them.
|
||||
- **Converging the hub is not refused while a carried peer has not enrolled.** Proposed in review
|
||||
and rejected on the record's own terms: *a node converges when its migration is done*, and the
|
||||
other machines' migrations are not this node's. With the range and the peers surviving the flip
|
||||
there is nothing left for the refusal to protect, and it would have made one machine's converge
|
||||
wait on every other machine.
|
||||
- **A takeover whose placement disagrees with the tunnel is refused before it is composed** — an
|
||||
address or a port that is not the tunnel's would stop the found interface and raise the mesh's
|
||||
somewhere the peers are not, while reporting success.
|
||||
- **The host says three things, not two**: the found interface still up, the mesh's up in its place,
|
||||
or — the state worth naming — the found one down and the mesh's not up, which is the only one
|
||||
where the peers reach nothing.
|
||||
- **A hub that enrolled before this existed keeps its identity.** Re-enrolling would have remade
|
||||
every credential in the mesh, because the hub provides the store and the broker. Instead the
|
||||
machine takes the tunnel's key as its overlay key and says so in a message signed with the
|
||||
identity it already has, so a forged report cannot move a node's key.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md),
|
||||
[ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
|
||||
- [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
|
||||
- [research 012 — the minimum viable node](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md)
|
||||
@@ -1,107 +0,0 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-09-23
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0002-nodes-communicate-over-a-broker.md
|
||||
---
|
||||
|
||||
# 106. The bus is NATS
|
||||
|
||||
## Context
|
||||
|
||||
The mesh's bus is an AMQP broker. [Research 014](../01-RESEARCH/014-the-bus-on-nats/00-overview.md)
|
||||
measured what that costs and what it would take to change: AMQP is spoken in three places of the
|
||||
mesh's own code — the controller's link, the host's link, the tool runtime's client — and in none of
|
||||
the sdk or the modules, because [ADR 0039](0039-what-the-sdk-holds-and-refuses.md) kept the client
|
||||
out of the sdk for exactly this. The predecessor's world is AMQP and is being retired module by
|
||||
module; it cannot move and does not need to.
|
||||
|
||||
The operator's direction: NATS, native — the mesh built on it, not bridged to it — and the tools a
|
||||
person reaches from a workstation designed on it rather than built quickly on what exists.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Stay on AMQP.** Rejected by the operator.
|
||||
2. **NATS behind a bridge**, the mesh unchanged. Rejected: two buses, every guarantee crossing a
|
||||
seam, and the reason for changing — one native, simple, subject-addressed bus with request/reply
|
||||
and accounts built in — lost at the seam.
|
||||
3. **NATS native, built in the lab in parallel with the migration, cut over in one rehearsed
|
||||
rollout after the migration's core is done.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh's bus is NATS.** Every link the mesh has — control, node queues, builds, enrolment,
|
||||
reports, events, tool invocation — is carried on NATS subjects; durability, catch-up and
|
||||
hold-unacknowledged-and-retry ([ADR 0083](0083-one-push-leaves-the-mesh-consistent.md)) are
|
||||
JetStream's; a module's account is a NATS account with per-subject permissions derived from its
|
||||
`emits` and `consumes` ([ADR 0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md)),
|
||||
written as configuration the host declares and the server reloads — never through a management
|
||||
API. The broker module changes; the seat `mesh-broker` does not
|
||||
([ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md)).
|
||||
|
||||
**The sdk's broker contract does not change.** Modules are written against `request`, `handle`,
|
||||
`publish`, `subscribe`; the runtime implements them on NATS; no module speaks a protocol.
|
||||
|
||||
**The AMQP broker the mesh adopted becomes the predecessor's compatibility broker**, kept as a
|
||||
module with one purpose — the predecessor's clients — and retired with the last of them.
|
||||
|
||||
**It is built beside the migration and cut over after its core.** The bus is not changed under a
|
||||
half-migrated node. A lab bed proves the whole path — enrolment, the store window, upgrades, tool
|
||||
invocation — on NATS before any node's bus moves, and the move is one rollout: controller, every
|
||||
host, every runtime together.
|
||||
|
||||
**What a person reaches the mesh's tools with is designed on NATS**, as part of the same design: a
|
||||
person's account, scoped like a module's, and a client that speaks the bus directly.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The architecture — subjects, streams, accounts, the enrolment handshake, the person's client — is
|
||||
written as a to-be design before code, and reviewed.
|
||||
- Eight records are touched: 0002 and 0033 (the substrate names a broker — now NATS), 0041 and
|
||||
0042 (events keep their shape; the exchange becomes a subject prefix), 0043 (accounts as
|
||||
configuration), 0078 (the broker module is `nats`), 0083 (JetStream carries the guarantee), 0039
|
||||
(unchanged, and the reason this is possible).
|
||||
- The lab beds that prove the bus are re-run on NATS; none is skipped.
|
||||
- Until the cutover, nothing changes on any node.
|
||||
|
||||
## How it is checked
|
||||
|
||||
A lab bed raises a mesh on NATS from genesis and proves: a node enrols over TLS with a claimed
|
||||
token; a push composes, is held while the store restarts and applies after; an upgrade rolls out;
|
||||
a module's tools are invoked from another node and from a person's client; an event dead-letters
|
||||
after its deliveries are exhausted; a module's account cannot publish outside its `emits` nor
|
||||
subscribe outside its `consumes`. Then the cutover bed: a mesh on AMQP with the predecessor's
|
||||
compatibility broker beside it moves its bus in one rollout with every node reporting afterwards.
|
||||
|
||||
## Progressive insight
|
||||
|
||||
> **Progressive insight — 2026-09-26.** *The compatibility broker was not single-purpose when this
|
||||
> was written.* This record says the adopted AMQP broker is "kept as a module with one purpose —
|
||||
> the predecessor's clients". Two modules of the new mesh also depended on it, through a `requires:
|
||||
> ["amqp"]` grant its provisioner answered with a private vhost — `amqp-ping` and
|
||||
> `amqp-email-forwarder`. On the retirement condition below, both would have been left requiring
|
||||
> something no provider answers.
|
||||
> [ADR 0125](0125-the-bus-is-the-only-broker.md) resolves it by moving them onto the bus and
|
||||
> retiring the interface, which makes this record's sentence true rather than merely intended. The
|
||||
> decision — the bus is NATS, the AMQP broker becomes the predecessor's compatibility broker and
|
||||
> retires with the last of them — is unchanged.
|
||||
|
||||
> **Progressive insight — 2026-09-26, correcting the one above.** *The broker is not a
|
||||
> compatibility module at all, and the sentence does not become true.* The insight above said
|
||||
> [ADR 0125](0125-the-bus-is-the-only-broker.md) would make "one purpose — the predecessor's
|
||||
> clients" true by moving the mesh's own modules off it.
|
||||
> [ADR 0127](0127-amqp-is-a-provision-not-the-bus.md) supersedes that: nothing moves off, because
|
||||
> a module may legitimately need an AMQP broker as a backing service the way it needs a database.
|
||||
> The broker becomes **an ordinary provider module** — no seat, not foundation, not raised at
|
||||
> genesis, and with no retirement condition, because the day its last client disappears is not a
|
||||
> day anything is waiting for. What this record decided — the mesh's bus is NATS — is untouched
|
||||
> by both; what was wrong was the sentence describing what happens to the old server, twice.
|
||||
|
||||
## References
|
||||
|
||||
- [research 014](../01-RESEARCH/014-the-bus-on-nats/00-overview.md)
|
||||
- [ADR 0002](0002-nodes-communicate-over-a-broker.md), [0033](0033-the-substrate-is-a-store-and-a-broker.md),
|
||||
[0039](0039-what-the-sdk-holds-and-refuses.md), [0043](0043-a-module-broker-account-is-scoped-by-emits-and-consumes.md),
|
||||
[0078](0078-the-store-and-broker-are-modules.md), [0083](0083-one-push-leaves-the-mesh-consistent.md)
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-09-24
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
---
|
||||
|
||||
# 107. Persistent data is a directory bind, never a named volume
|
||||
|
||||
## Context
|
||||
|
||||
Measured 2026-09-24, mid-migration: four modules mount a named Docker volume for real state —
|
||||
`mesh-store` (every database the mesh holds), `mesh-broker` (its data and TLS material),
|
||||
`mesh-registry` (every image), and `searxng`'s cache. Every other module in the catalogue —
|
||||
more than forty — mounts a host directory, `/var/lib/<module>/...`.
|
||||
|
||||
The predecessor did not make this choice. HAL's own `postgres` bound `./db-data`, and its
|
||||
`lavinmq` bound `./data` — directories, both. The mesh's adoption of them
|
||||
([`a63ef3d`](https://git.novox.be/novox/mesh-catalog/commit/a63ef3d), "the postgres module
|
||||
adopts mesh-store instead of raising its own", 2026-09-16) introduced the named volume; three
|
||||
other modules followed the same shape since. Full account of what was found and fixed:
|
||||
[issue 115](../04-ISSUES/115-a-named-docker-volume-is-invisible-and-one-flag-from-gone/00-report.md).
|
||||
|
||||
## The two guarantees are not the same guarantee
|
||||
|
||||
A named volume and a host directory both survive **ordinary** container recreation — a rebuild,
|
||||
a `take`, the `docker rm -f` and push this migration already uses routinely. Neither loses data
|
||||
to that. That was never the question.
|
||||
|
||||
What they do not both survive:
|
||||
|
||||
- **`docker rm -fv`, `docker volume rm`, `docker system prune --volumes`** all target a named
|
||||
volume specifically. The first is one character from the command this migration's own rules
|
||||
already call for after every address change. A host directory has no equivalent command that
|
||||
destroys it by accident — removing it is always a deliberate `rm -rf` on a path someone typed.
|
||||
- **Visibility.** Every tool this migration has used all night to find and verify data —
|
||||
`ls`, `find`, `grep`, a backup job — reaches a host directory for free. A named volume requires
|
||||
knowing to ask Docker (`docker volume inspect`) before its bytes, at
|
||||
`/var/lib/docker/volumes/<name>/_data`, are reachable at all.
|
||||
|
||||
## Decision
|
||||
|
||||
**A container mount holding data that must survive is a host directory bind. A named volume is
|
||||
permitted only for data that is disposable if lost** — a cache, a scratch space, something the
|
||||
module rebuilds on next start without consequence. `searxng`'s `valkey` cache is close to this
|
||||
line and was converted anyway, for consistency and because it costs nothing to.
|
||||
|
||||
Ownership is the one thing a host directory does not get for free that a named volume does:
|
||||
Docker initialises a fresh named volume's ownership to what the container's first process needs;
|
||||
a host directory is whatever created it. A directory made for this purpose must be given the
|
||||
image's expected UID before the container using it starts — read from the running instance being
|
||||
replaced when one exists, rather than guessed.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The four modules were converted: `distribution`, `lavinmq`, `postgres`, `searxng`. Data copied
|
||||
and verified byte-for-byte before each manifest changed; `mesh-store` stopped cleanly first, so
|
||||
its copy is crash-consistent rather than a live read of a running postgres.
|
||||
- **The ownership gap above was not theoretical — it is what happened.** The new directories
|
||||
were created by the operator's tooling running as root; `mesh-store` crash-looped on
|
||||
`mkdir: ... Permission denied` until its directory's ownership was set to match what the
|
||||
original volume already had. Worth a check at `module add` time — nothing catches this today
|
||||
beyond the container failing to start.
|
||||
- Old named volumes were not deleted. They remain the rollback path until confidence in the new
|
||||
mounts is established over time, not one clean start.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 115](../04-ISSUES/115-a-named-docker-volume-is-invisible-and-one-flag-from-gone/00-report.md)
|
||||
- `mesh-catalog` PR #54
|
||||
@@ -1,102 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-25
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0007-connectivity.md
|
||||
---
|
||||
|
||||
# 108. A route carries the policy applied to a request, and names a secret rather than holding one
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0007](0007-connectivity.md) settled that **a route is a grant**: a module contributes the name
|
||||
it wants and the port it listens on, and the proxy hands back the public name. That governs the
|
||||
**grant**. It says nothing about what a request arriving at the name is permitted to do, and the
|
||||
mesh's proxy currently permits everything: its request path is a host lookup and a forward, with no
|
||||
authentication, no source check, no redirect handling and no middleware anywhere in it.
|
||||
|
||||
The standing requirement is that the mesh does **at minimum** what the system it replaces already
|
||||
does. Measured against the predecessor's live configuration and its module catalogue
|
||||
([issue 116](../04-ISSUES/116-route-proxy-has-no-auth-or-ip-restriction/00-report.md)), four
|
||||
capabilities are relied on and absent:
|
||||
|
||||
| Capability | Dependents, counted |
|
||||
|---|---|
|
||||
| Authentication | **three** modules, each gating a credential-less admin surface — a key-value browser UI, a **database web UI**, and the replaced ingress's own dashboard |
|
||||
| Refusal scoped to a path | **one**, and it is a live **incident mitigation** closing a write primitive that was abused |
|
||||
| Path-scoped routing with priority | **two** — the refusal above, and a certificate-challenge path on a host that otherwise routes elsewhere |
|
||||
| Redirect | **two** live routes canonicalising a `www` name onto its apex |
|
||||
|
||||
Counting needed two sources and neither alone is complete: the catalogue cannot see what was
|
||||
hand-written on a node, and a node's configuration directory cannot see what modules declare as
|
||||
container labels. An earlier count read one source and undercounted authentication by two.
|
||||
|
||||
**The third row is a prerequisite, not a sibling.** The proxy's table maps a host to exactly one
|
||||
target, so a host cannot be routed two ways. Adding authentication and a source filter would not
|
||||
make the refusal rule expressible.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Leave policy out of the mesh; keep the affected routes on the adopted ingress.** *Rejected.*
|
||||
The mesh would run two reverse proxies indefinitely with no principled division between them, and
|
||||
one of the routes held back is a live mitigation — leaving it on a component being decommissioned
|
||||
means its removal date is whenever somebody forgets.
|
||||
2. **A separate, proxy-side settings layer keyed by route name.** *Rejected.* It keeps the grant
|
||||
literally clean, but answering *"what protects this route"* then requires reading two files that
|
||||
nothing keeps in step. A route's protection is part of what a route is.
|
||||
3. **The grant carries a reference; the detail lives in a second layer.** *Rejected.* Both costs of
|
||||
option 2, plus a naming indirection to maintain.
|
||||
4. **Carry the password hash in the declaration.** *Rejected.* It would be the first credential
|
||||
value in a declaration, and a precedent is easier to set than to withdraw. A hash is not a
|
||||
plaintext password, but it is sufficient to pass the gate it protects.
|
||||
5. **An open middleware surface the proxy applies.** *Rejected.* It recreates the thing being
|
||||
replaced, makes the proxy's behaviour unbounded, and an open surface is far harder to narrow later
|
||||
than a closed one is to widen.
|
||||
|
||||
## Decision
|
||||
|
||||
**A route contribution carries the policy applied to requests arriving at its name**, alongside the
|
||||
name, the port and the location it already carries.
|
||||
|
||||
**The set is closed, and it is these four:** authentication; refusal scoped to a path; path-scoped
|
||||
routing with priority; redirect. A fifth is an amendment to this record, deliberately — each
|
||||
addition should be earned by a dependent that exists.
|
||||
|
||||
**Where policy needs a credential, the declaration names a secret the mesh mints and holds. It never
|
||||
carries the value.** This keeps the existing secret machinery as the only thing that holds
|
||||
credentials, and keeps hashes out of anything regenerated, synced or committed.
|
||||
|
||||
**Consequently the routing table is keyed by host and path, with priority** — not by host alone.
|
||||
This follows from the decision rather than being a separate one: two of the four capabilities need a
|
||||
single host routed more than one way.
|
||||
|
||||
## Consequences
|
||||
|
||||
**What this makes possible.** The affected routes can leave the adopted ingress, and "at minimum"
|
||||
becomes a satisfiable claim rather than a standing exception. The incident mitigation gets a durable
|
||||
home in the mesh, which its own note already asked for.
|
||||
|
||||
**What got harder.** The contribution shape grows, and every provider of `route` must understand
|
||||
more of it. The table is no longer a flat map, and priority introduces ordering that has to be
|
||||
deterministic rather than incidental — equal priorities must resolve the same way every time or the
|
||||
proxy becomes non-reproducible. A closed set means a new need is a decision, not a patch.
|
||||
|
||||
**What does not change.** The proxy remains a reference implementation: the contract is the file the
|
||||
mesh writes, not the program that reads it. Another proxy may implement the same file.
|
||||
|
||||
**How this is checked.** A rule states how it is verified, so this one does. A lab bed must show,
|
||||
against a mesh that declared them: a route with authentication refusing an unauthenticated request
|
||||
and admitting an authenticated one; a path-scoped refusal shadowing an ordinary route on the same
|
||||
host while that ordinary route still serves every other path; a redirect answering with the
|
||||
redirect; and — the negative case, which is the one that rots quietly — **a declaration carrying a
|
||||
credential value rather than a reference being refused**, so option 4 cannot return by accident.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 116](../04-ISSUES/116-route-proxy-has-no-auth-or-ip-restriction/00-report.md) — the count,
|
||||
the evidence, and the structural finding about the table.
|
||||
- [ADR 0007](0007-connectivity.md) — a route is a grant. This record extends it to the request.
|
||||
- [ADR 0009](0009-modules-and-the-graph.md) — the provision vocabulary a contribution belongs to.
|
||||
- [To-be 08 §3](../03-DESIGN/01-to-be/08-connectivity.md) — where exposure is specified.
|
||||
@@ -1,106 +0,0 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-09-25
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0075-two-stores-and-which-provides-what.md
|
||||
---
|
||||
|
||||
# 109. A package registry seat is one per ecosystem, not one for all of them
|
||||
|
||||
## Context
|
||||
|
||||
Fixing `builder`'s consumption of `package-registry` tonight surfaced the shape ADR 0075 actually
|
||||
left implicit. 0075 split `artifact-store` from `package-registry` and said the second is "an
|
||||
ecosystem's own registry — npm, cargo, PyPI, Go" — but it defined one provision for all four,
|
||||
not one each.
|
||||
|
||||
What that produces, read from the manifests as they stand:
|
||||
|
||||
- `gitea`'s `module.json` declares `provides: package-registry` once, and its `serves` block
|
||||
carries exactly one path: `npm-path`. Nothing names a cargo or PyPI endpoint, though gitea's own
|
||||
package API serves both.
|
||||
- `builder` had, until tonight, a hand-written JSON fragment standing in for a real grant —
|
||||
`{"provision": "package-registry", "from": "gitea", "at": "127.0.0.1", ...}` — because nothing
|
||||
in the interface gave it a real one to ask for. The fragment named `npm-path` specifically; there
|
||||
was nowhere to put a second ecosystem's endpoint even if one had been wired.
|
||||
- The fix applied tonight declares `requires: ["artifact-store", "package-registry"]` and lets the
|
||||
mesh mint the grant properly — correct for what exists today, but it is one seat standing in for
|
||||
what should be several, the same conflation 0075 itself named and did not resolve for this
|
||||
provision specifically.
|
||||
|
||||
**The version-skew problem 0075 wrote down for the artifact-store/package-registry split repeats
|
||||
one level down, inside "package registry" itself:** npm resolves by name and range from one
|
||||
namespace, cargo from another, and a single grant conflates them exactly the way one store for
|
||||
both digests and ranges would have.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. One `package-registry` provision, gitea answers every ecosystem it can.** What exists today.
|
||||
Simplest to grant — one credential, one binding, done once per consumer. Rejected: a consumer
|
||||
that only ever needs npm still receives a grant shaped to cover cargo and PyPI, and there is no way
|
||||
to hand off *only* npm to a different provider (verdaccio, say) without renegotiating the whole
|
||||
provision for every consumer of any ecosystem.
|
||||
|
||||
**2. One provision, parameterised by ecosystem.** `requires: package-registry` plus a declared
|
||||
`ecosystem: npm` alongside it, still one interface. Rejected: the `provides`/`requires` refusal
|
||||
mechanism this mesh already uses (two providers of one provision is a naming conflict until
|
||||
resolved) would need to become conditional on a parameter it does not otherwise carry anywhere in
|
||||
the mesh's resolution — a special case for exactly one provision, rather than the mesh's existing
|
||||
mechanism applied again.
|
||||
|
||||
**3. One provision per ecosystem — `npm-package-registry`, `cargo-package-registry`,
|
||||
`docker-package-registry`, and so on, each independently `provides`/`requires`.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**A package registry seat is one per ecosystem.** `npm-package-registry`, `cargo-package-registry`,
|
||||
`docker-package-registry` — each its own provision, resolved, granted, and refused exactly the way
|
||||
`artifact-store` and today's single `package-registry` already are. Adding an ecosystem is adding a
|
||||
provision, not widening one.
|
||||
|
||||
**Gitea may hold several seats at once.** Nothing here says gitea answers only one; ADR 0075
|
||||
already established that a provider may answer more than one named thing on one machine ("a mesh
|
||||
running gitea for git and packages alongside a registry serving artifacts is an ordinary
|
||||
arrangement"). Gitea fulfilling `npm-package-registry` and `cargo-package-registry` both is the
|
||||
expected shape, not an exception.
|
||||
|
||||
**Each seat's grant is independent.** A consumer that only needs npm holds only the
|
||||
`npm-package-registry` grant. Moving that one ecosystem to a different provider — verdaccio,
|
||||
named directly as the motivating case — means assigning `npm-package-registry` to verdaccio and
|
||||
leaving every other seat exactly where it was. No consumer of `cargo-package-registry` observes
|
||||
the change; no manifest naming `package-registry` broadly needs to be found and re-read.
|
||||
|
||||
**`builder`'s fix tonight is the interim shape, not the target.** It correctly consumes the one
|
||||
seat that exists today (`package-registry`, npm in practice). Splitting it becomes, later,
|
||||
replacing that one line with the ecosystems `builder` actually uses — a manifest change, not a
|
||||
redesign of how `builder` asks for anything.
|
||||
|
||||
## Consequences
|
||||
|
||||
- `gitea`'s `module.json` gains a `provides` entry per ecosystem it actually serves, each with its
|
||||
own `serves` block (`npm-path`, a cargo path, a PyPI path) in place of the one `package-registry`
|
||||
entry with a single `npm-path` inside it.
|
||||
- `gitea`'s provisioner (`modules/gitea/provisioner/index.ts`) currently runs one `runProvisioner`
|
||||
registration for `package-registry`; each seat needs its own registration, or one provisioner
|
||||
keyed by which seat's `create`/`remove` fired — the mesh's `Provision` type does not yet carry
|
||||
which named provision a call is for when a module answers more than one, and that is worth
|
||||
checking before assuming the harness already supports it.
|
||||
- Every consumer's `requires` moves from the one name to however many ecosystems it actually uses.
|
||||
`builder` is the only known consumer today; widening later is one manifest line per module, not
|
||||
a migration.
|
||||
- `verdaccio`'s role sharpens: not "package-registry, an alternative for npm alone" (0075's phrasing)
|
||||
but a named `npm-package-registry` *provider*, a straight swap against gitea's answer to the same
|
||||
seat.
|
||||
- Not solved here: whether `cargo-package-registry` and `pypi-package-registry` are needed at all
|
||||
before something actually consumes them. This record names the shape; building unused seats is
|
||||
its own decision.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0075](0075-two-stores-and-which-provides-what.md) — the record this extends; defined
|
||||
`package-registry` as the second provision without splitting it per ecosystem.
|
||||
- `mesh-catalog modules/builder/module.json` — tonight's fix, the interim single-seat shape.
|
||||
- `mesh-catalog modules/gitea/module.json`, `modules/gitea/provisioner/index.ts` — today's
|
||||
single-provision, npm-only implementation.
|
||||
@@ -1,235 +0,0 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-25
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0009-modules-and-the-graph.md
|
||||
---
|
||||
|
||||
# 110. A seat is held by one assignment, from a closed set, and it may deliver a provision
|
||||
|
||||
> **Superseded in part — 2026-09-27, by [ADR 0122](0122-a-seat-is-data-a-rename-is-a-database-update.md).**
|
||||
> "The controller carries the set in code" no longer holds: the seats are rows of a table in the controller's
|
||||
> store, seeded by a migration, and a seat is referenced by a stable id, its name only a label. The closed set,
|
||||
> one holder per scope and a delivering seat as the single answer for its provision stand as decided here.
|
||||
|
||||
> **Narrowed, not replaced — 2026-09-27, on merging two lines of work.** This was marked superseded by
|
||||
> [ADR 0126](0126-a-module-declares-its-own-seats.md), and that overstated it: 0126 says in as many
|
||||
> words that *"everything 0110 decided about what a seat is stands untouched"*. What moved is where the
|
||||
> set lives and who may add to it —
|
||||
> [0126](0126-a-module-declares-its-own-seats.md) lets a module declare one and makes the set derived,
|
||||
> [0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) names the mesh's own
|
||||
> for their scope, and [0122](0122-a-seat-is-data-a-rename-is-a-database-update.md) moves them out of
|
||||
> code into a table. **What a seat *is* — one holder at its scope, a definition saying what a module
|
||||
> can hold against an assignment saying what it does, a role made singular rather than a module — is
|
||||
> this record and still current**, which is why those three rest on it.
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0009](0009-modules-and-the-graph.md) introduced claims: a module declares something
|
||||
singular, at a scope, and a second holder is refused. [ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md)
|
||||
named the foundation's three after their servers. That mechanism is enforced and works. What it
|
||||
means has drifted, and four things are now true of it that no record says.
|
||||
|
||||
**Any well-formed name becomes a seat by being claimed.** The controller's manifest check
|
||||
refuses a claim only for a malformed name or an unknown scope. Nothing says which seats a mesh has.
|
||||
The names in use were each invented by the module that claims them: `the-showcase`,
|
||||
`the-build-machine`, `the-intrusion-prevention`.
|
||||
|
||||
**Nothing can say what a mesh has, or who fills it.** There is no seat table and no command that
|
||||
lists seats. Holdings are assembled while planning, one node at a time, and discarded afterwards.
|
||||
The only way to answer "which seats does this mesh have, and which module holds each" is to read
|
||||
every manifest in two repositories, because the controller's own manifest lives in its own
|
||||
repository ([ADR 0069](0069-a-module-is-a-repository-and-a-path.md)), and then the controller's code,
|
||||
because one module it ships has its manifest composed there. While this record was being prepared,
|
||||
that enumeration was done by hand, and it missed both of the last two sources: eleven claims were
|
||||
reported where there are thirteen.
|
||||
|
||||
**A claim in a definition makes a module singular, not a role.** The store module's definition
|
||||
claims `mesh-store`, so every assignment of it claims the seat, and a second store module on any other
|
||||
node is refused. What is singular is *the store the mesh itself uses*, not postgres. Any module can
|
||||
run on any node whose capabilities match, which is a core principle of the module system, and a claim
|
||||
written into the definition breaks it for every module that claims anything.
|
||||
|
||||
**Some seats are the mesh's one of something everyone consumes, and nothing uses that fact.** A
|
||||
requirement for a mesh-scoped provision with more than one provider is refused until a person pins,
|
||||
**per consumer node**, which provider to use. [ADR 0109](0109-a-package-registry-seat-is-one-per-ecosystem.md)
|
||||
anticipates exactly that case, gitea and verdaccio both answering npm, and under today's resolution it
|
||||
would mean a pin on every machine that builds anything.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Leave seats as free-form exclusion, claimed in definitions.** Rejected. The overview stays
|
||||
unanswerable, a module that claims a seat can run on only one node, and a second provider of anything
|
||||
costs a pin per consumer node.
|
||||
|
||||
**2. Two concepts: seats for exclusion, and a new word for the mesh's one consumable thing.**
|
||||
Rejected. Both mean "this mesh's one X". Every existing claim would first have to be classified into
|
||||
one or the other, and the overview a person wants is one list, not two.
|
||||
|
||||
**3. A seat is held by one assignment, from a closed set, and holding it may deliver a provision.**
|
||||
Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh defines a closed set of seats.** Each entry has a name, a scope, what holding it delivers
|
||||
(if anything), and the decision that made it a seat. A seat outside the set is refused wherever it
|
||||
is named. Adding a seat is a decision, for the same reason adding a shape to the host's vocabulary is
|
||||
one: the set is what a person reads to learn what a mesh can have, and a name added without an
|
||||
argument is a name nobody can explain later.
|
||||
|
||||
**A definition says which seats a module *can* hold. An assignment says which it *does* hold.** The
|
||||
store module can hold `mesh-store`, and it may be assigned to every node. Exactly one of those
|
||||
assignments holds the seat, because that assignment said so. A second assignment saying so, at the
|
||||
seat's scope, is refused. So a seat makes a *role* singular, never a module, and moving the role is
|
||||
changing which assignment holds it, with no definition changed and nothing unassigned.
|
||||
|
||||
**What the mesh knows about a seat's holder is what it knows about that assignment**: its node, the
|
||||
node's settings for it, and what it serves. Holdings are not stored separately. The seat points at an
|
||||
assignment, and a second record of the same fact would be a second thing to disagree with the first.
|
||||
|
||||
**Holding a seat may deliver a provision.** A seat that delivers a provision may only be held by an
|
||||
assignment of a module that provides it, at the seat's scope.
|
||||
|
||||
**A requirement may name a seat, and then the seat's holder answers it.** Naming the seat asks for
|
||||
*the mesh's* one, not for whichever provider is nearest, so the holder answers **even when another
|
||||
provider runs on the consumer's own node**, and nothing is asked of anyone. Unheld, the requirement is
|
||||
refused, naming the seat. A builder asks for the mesh's npm registry this way, and is served by the
|
||||
holder of `npm-package-registry` wherever it runs, with no pin on any machine.
|
||||
|
||||
**A requirement that names no seat resolves as [ADR 0084](0084-which-provider-serves-a-consumer.md)
|
||||
has it**: a pin, then the provider on the consumer's own node, then the only provider. Where several
|
||||
remain and none is local, **a person chooses when the module is assigned**. Assignment lists the
|
||||
candidates, with the holder of a seat that delivers the provision suggested first, and records the
|
||||
answer on the assignment as its pin. Without an answer the module is not assigned. This keeps
|
||||
[ADR 0009](0009-modules-and-the-graph.md)'s rule that a requirement with several answers is never
|
||||
guessed. The choice is made either by the requirement naming the seat, or by a person at assignment,
|
||||
and never silently by what happens to run nearby. That is the failure
|
||||
[issue 106](../04-ISSUES/106-the-vault-claims-no-seat/00-report.md) names for the vault.
|
||||
|
||||
**A seat delivers a provision only where the mesh has one answer for everyone.** The artifact store,
|
||||
the npm registry, git and the vault are each one per mesh by their own records, so their seats
|
||||
deliver them.
|
||||
|
||||
**The foundation's seats deliver nothing.** `mesh-controller`, `mesh-store` and `mesh-broker` name
|
||||
which assignment the mesh *itself* uses: the controller, the store holding its records, the broker
|
||||
carrying its bus. The store and broker modules may run on other nodes too, and a database or `amqp`
|
||||
consumer that names no seat is served by co-location from whichever runs on its own node, the seat's
|
||||
holder included. Were `mesh-store` to deliver, a consumer could name it and be sent to the store the
|
||||
mesh keeps its own records in. That is not a store for consumers.
|
||||
|
||||
**A seat may reserve its provision.** Where a second provider would break the reason the provision
|
||||
exists, only an assignment holding the seat may provide it at all: the parser refuses a definition
|
||||
that provides it without being able to hold the seat, resolution refuses an assignment providing it
|
||||
without holding the seat, and a pin cannot choose anyone else: a requirement for it always names the seat. `secret` is the one
|
||||
reserved provision.
|
||||
The vault is one per mesh because a second *"would be a second place to lose"*
|
||||
([ADR 0085](0085-a-secret-is-a-provision.md), as amended), and a second `secret` provider is exactly
|
||||
that, whether a pin chose it or not.
|
||||
|
||||
**Seats are also informational.** The controller lists every seat in the set, what it delivers, and
|
||||
which assignment holds it, including seats nobody holds. An unheld seat is an answer, "this mesh has
|
||||
no X", not an error.
|
||||
|
||||
**The first set is the twelve seats already claimed, plus two.** Thirteen claims are in use, and they
|
||||
name twelve seats because two alternative modules claim `the-resolver-configuration`. This record
|
||||
admits every seat the catalogue and the controller claim today, so no definition is refused by it:
|
||||
|
||||
| seat | scope | delivers | can be held by | made a seat by |
|
||||
|---|---|---|---|---|
|
||||
| `mesh-controller` | mesh | — | `mesh-controller` | [0079](0079-the-foundation-seats-are-named-after-their-servers.md) |
|
||||
| `mesh-store` | mesh | — | `postgres` | [0079](0079-the-foundation-seats-are-named-after-their-servers.md) |
|
||||
| `mesh-broker` | mesh | — | `lavinmq` | [0079](0079-the-foundation-seats-are-named-after-their-servers.md) |
|
||||
| `mesh-vault` | mesh | `secret`, reserved | `mesh-vault` | this record, for [issue 106](../04-ISSUES/106-the-vault-claims-no-seat/00-report.md) |
|
||||
| `the-artifact-store` | mesh | `artifact-store` | `distribution` | [0075](0075-two-stores-and-which-provides-what.md) |
|
||||
| `the-catalogue` | mesh | — | `mesh-catalog` | this record |
|
||||
| `npm-package-registry` | mesh | `npm-package-registry` | `gitea` | [0109](0109-a-package-registry-seat-is-one-per-ecosystem.md) |
|
||||
| `the-build-machine` | node | — | `builder` | this record |
|
||||
| `the-dns-port` | node | — | `dnsmasq` | this record |
|
||||
| `the-intrusion-prevention` | node | — | `fail2ban` | this record |
|
||||
| `the-packet-filter` | node | — | `nftables` | this record |
|
||||
| `the-private-network` | node | — | the controller's private-network module | this record |
|
||||
| `the-resolver-configuration` | node | — | `resolv-conf` or `resolved-split-dns` | this record |
|
||||
| `the-showcase` | node | — | `showcase` | this record |
|
||||
|
||||
There are two additions. `npm-package-registry` is [ADR 0109](0109-a-package-registry-seat-is-one-per-ecosystem.md)'s
|
||||
seat, named after the provision it delivers, as 0079 named the foundation's seats after what they are.
|
||||
A gitea assignment holds it. verdaccio provides the same provision and cannot hold the seat, so it is
|
||||
the second provider this record exists to make harmless. Moving npm to it would take a definition
|
||||
saying it can hold the seat, and then an assignment saying it does.
|
||||
|
||||
`mesh-vault` answers [issue 106](../04-ISSUES/106-the-vault-claims-no-seat/00-report.md). The vault is one
|
||||
per mesh ([ADR 0085](0085-a-secret-is-a-provision.md), as amended), and until now that was enforced by
|
||||
nothing. The seat is named after its server, by the 0079 convention.
|
||||
|
||||
`the-dns-port` is listed as delivering nothing, although `dnsmasq` provides `wildcard-resolution`.
|
||||
That provision is node-scoped and answered on the machine, so no preference between providers arises.
|
||||
|
||||
## What this changes in earlier records
|
||||
|
||||
On acceptance, each of these is amended by this record, not edited:
|
||||
|
||||
- [ADR 0009](0009-modules-and-the-graph.md): a claim in a definition says a module *can* hold a seat;
|
||||
the assignment says it does.
|
||||
- [ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md) and
|
||||
[ADR 0078](0078-the-store-and-broker-are-modules.md): "a mesh runs one postgres and one
|
||||
lavinmq" becomes one holder of `mesh-store` and one of `mesh-broker`. The store and broker modules may
|
||||
run on other nodes.
|
||||
- [ADR 0084](0084-which-provider-serves-a-consumer.md): a requirement may name a seat, which its holder
|
||||
answers; and where several providers remain and none is local, the choice is asked when the module is
|
||||
assigned and recorded as a pin, rather than refused until someone pins it.
|
||||
- [ADR 0109](0109-a-package-registry-seat-is-one-per-ecosystem.md): one provision per package
|
||||
ecosystem stands. Where 0109 says *seat*, it means that provision. Only `npm-package-registry` is
|
||||
also a seat in this set. A cargo or docker registry becomes one by a record, as any seat does.
|
||||
"Gitea may hold several seats" reads: gitea may provide several ecosystems, and hold the seat of
|
||||
each one that is a seat. Moving npm to verdaccio is not "assigning `npm-package-registry` to
|
||||
verdaccio". It takes verdaccio's definition saying it can hold the seat, and then an assignment
|
||||
holding it.
|
||||
- [To-be 23](../03-DESIGN/01-to-be/23-choosing-a-provider.md): the same two changes, in the design
|
||||
that describes choosing a provider.
|
||||
- [To-be 21](../03-DESIGN/01-to-be/21-the-installation-in-full.md): genesis assigns the foundation's
|
||||
store, broker and controller holding their seats, where their definitions claim them today.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The controller carries the set in code. A test asserts its size, and that every entry names the
|
||||
record that made it a seat, so changing the set means finding the argument rather than a number.
|
||||
- An assignment gains the seats it holds. Genesis assigns the foundation's store, broker and
|
||||
controller holding their seats, where today their definitions claim them.
|
||||
- Manifest validation refuses an unknown seat, a seat named at the wrong scope, and a delivering seat
|
||||
named by a module that does not provide the provision. Resolution refuses a second holder, and an
|
||||
assignment holding a seat its module cannot hold.
|
||||
- Resolution answers a requirement naming a seat with its holder. Assignment asks a person where
|
||||
several providers remain, suggesting the seat's holder first, and records the answer as a pin. A
|
||||
provider record gains the module it came from.
|
||||
- A `seats` command lists the set with each seat's holder, derived from assignments.
|
||||
- **What got harder:** a module wanting a new singular role can no longer invent a name. It needs a
|
||||
record. And an assignment has one more thing to say. Both are the point.
|
||||
- **Not changed:** the controller's seat placeholder stays as it is. It exists so the controller can
|
||||
reach a foundation it made before any module existed.
|
||||
|
||||
## How it is checked
|
||||
|
||||
| Rule | Checked by |
|
||||
|---|---|
|
||||
| The set is closed, and every entry names its decision | A controller unit test asserts the set's size and a non-empty decision for every entry. |
|
||||
| A seat outside the set is refused | Manifest-validation tests for an unknown seat, the wrong scope, and a delivering seat named by a module that does not provide it. |
|
||||
| Every module in use names a seat in the set | A controller test parses every catalogue manifest and fails on any refused seat. The lab's beds read the same manifests ([ADR 0089](0089-a-bed-reads-the-catalogue-it-proves.md)). |
|
||||
| A seat is held by one assignment, not by a module | A resolution test: the store module assigned to two nodes resolves, with one assignment holding `mesh-store`; a second assignment asking to hold it is refused. |
|
||||
| A requirement naming a seat is answered by its holder | Resolution tests: two providers with the seat held; a second provider on the consumer's own node, where the holder still answers; the seat unheld with two providers, and with **one** provider, both refused naming the seat. |
|
||||
| An assignment holds only a seat its module can hold | A resolution test: an assignment holding a seat its definition does not name is refused. |
|
||||
| Every seat is listed with its holder | A `seats` command test: every seat in the set is listed with its scope, what it delivers and its holder, and an unheld seat is listed as unheld. |
|
||||
| Several providers and none local is a person's choice | An assignment test: the candidates are listed with the delivering seat's holder first; the answer is recorded as a pin; with no answer the module is not assigned. |
|
||||
| The foundation's seats route nobody | A resolution test: with the store module on two nodes, a database consumer is served by the one on its own node, whichever holds `mesh-store`; a requirement naming `mesh-store` is refused, because it delivers nothing. |
|
||||
| A reserved provision has no other provider | The parser refuses a definition providing `secret` that cannot hold `mesh-vault`; resolution refuses an assignment providing it without holding the seat, and a pin on a `secret` requirement. |
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0009](0009-modules-and-the-graph.md): claims, scopes, and "refused, never guessed"
|
||||
- [ADR 0079](0079-the-foundation-seats-are-named-after-their-servers.md): a seat named after what it is
|
||||
- [ADR 0109](0109-a-package-registry-seat-is-one-per-ecosystem.md): the per-ecosystem registry seats
|
||||
- [ADR 0084](0084-which-provider-serves-a-consumer.md), [to-be 23](../03-DESIGN/01-to-be/23-choosing-a-provider.md):
|
||||
pins, co-location and refusal
|
||||
- `mesh-controller internal/catalogue/resolve.go` (`checkClaims`, the brokered branch of `Resolve`),
|
||||
`internal/catalogue/manifest.go` (claim validation), `cmd/mesh-controller/plan.go` (holdings)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user