Compare commits
204
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0944437d4e | ||
|
|
2e6a989031 | ||
|
|
ec373556d0 | ||
|
|
f242b8763b | ||
|
|
bb6af05f5c | ||
|
|
672011ab92 | ||
|
|
05ff547358 | ||
|
|
1def984c07 | ||
|
|
76a671b8c5 | ||
|
|
19bd7fd960 | ||
|
|
f82a90dfa0 | ||
|
|
17c27ddf83 | ||
|
|
5c6b645c6b | ||
|
|
b6cf49d8ce | ||
|
|
1484ff7682 | ||
|
|
c10efa4d8c | ||
|
|
24fd567c31 | ||
|
|
be5284f7b8 | ||
|
|
21c4d212ec | ||
|
|
1bd6f4611e | ||
|
|
9d59c2ce14 | ||
|
|
64e357da7d | ||
|
|
9a9a78f882 | ||
|
|
e816b61cb7 | ||
|
|
610be93803 | ||
|
|
d5ddd1e715 | ||
|
|
a6db6a70cf | ||
|
|
e44ec3f612 | ||
|
|
dbc7406490 | ||
|
|
2001d6e063 | ||
|
|
90c6793096 | ||
|
|
499407c669 | ||
|
|
2cf1c64e24 | ||
|
|
e1987b3654 | ||
|
|
c87ab9af0f | ||
|
|
4e0dd1b0d3 | ||
|
|
b5d3f23947 | ||
|
|
c753ff068a | ||
|
|
7a1d3d89f0 | ||
|
|
cb79a28a80 | ||
|
|
0b9fda46cc | ||
|
|
6b58a6af82 | ||
|
|
9a39785116 | ||
|
|
c0b1c47be6 | ||
|
|
06440c1522 | ||
|
|
0e7eb9c2da | ||
|
|
cf73f96797 | ||
|
|
b0cfd08f7c | ||
|
|
0a6613f0d4 | ||
|
|
aaa0647ec3 | ||
|
|
de32fb82ea | ||
|
|
85a729c607 | ||
|
|
fb284fa6ee | ||
|
|
2b04a5f0a1 | ||
|
|
addbd74c39 | ||
|
|
6bf18062c4 | ||
|
|
0f63a73def | ||
|
|
d0ee14712e | ||
|
|
f6199fafce | ||
|
|
d901156614 | ||
|
|
e934a0286f | ||
|
|
8dbfd8d8ff | ||
|
|
2cea6c9b0c | ||
|
|
79dcc0a6ec | ||
|
|
65f69d96a9 | ||
|
|
b83a1e748e | ||
|
|
84b464d109 | ||
|
|
88d7db36c8 | ||
|
|
974b3e7d42 | ||
|
|
c61f01d63e | ||
|
|
4325467b4d | ||
|
|
0a010b8b79 | ||
|
|
cc0503ef66 | ||
|
|
56d8d546c7 | ||
|
|
edff1f25a1 | ||
|
|
85f0aecf5a | ||
|
|
0dc8cab178 | ||
|
|
98b731d5d1 | ||
|
|
375e77690b | ||
|
|
24701df23e | ||
|
|
dd5a065295 | ||
|
|
748641dd22 | ||
|
|
75d3ac0f82 | ||
|
|
b6a191d0a6 | ||
|
|
73b9cc3257 | ||
|
|
f795356ae1 | ||
|
|
aa4bfe289e | ||
|
|
6f0db74ee4 | ||
|
|
e7854a31a6 | ||
|
|
8bfb11da2c | ||
|
|
96f2ae72c1 | ||
|
|
78353ad272 | ||
|
|
fa5fd9fc69 | ||
|
|
f9c240a957 | ||
|
|
fff89b7c62 | ||
|
|
80e36c7595 | ||
|
|
3c49b389d1 | ||
|
|
222fbe95f4 | ||
|
|
b29008ad36 | ||
|
|
101d7f30f7 | ||
|
|
225eee14d0 | ||
|
|
12e75cdfa2 | ||
|
|
377fa3f88a | ||
|
|
11e32ce5be | ||
|
|
f1081c4f97 | ||
|
|
152cc0c0e7 | ||
|
|
a8b951f3b4 | ||
|
|
0eadc27fc7 | ||
|
|
4e437eb148 | ||
|
|
0e7a115183 | ||
|
|
18428baf4b | ||
|
|
7b5f3582be | ||
|
|
6dcfa2226e | ||
|
|
1ddc2f5cbf | ||
|
|
06998cc8d9 | ||
|
|
353bfdcef3 | ||
|
|
91f82cd38e | ||
|
|
300be92814 | ||
|
|
427fe4c9c5 | ||
|
|
a65e1076a2 | ||
|
|
09ec150bd6 | ||
|
|
1efa37957d | ||
|
|
26073f30b0 | ||
|
|
1f5a12e07e | ||
|
|
7ac5194bcb | ||
|
|
330d341618 | ||
|
|
cf56649845 | ||
|
|
b9af4e373d | ||
|
|
e7417cf124 | ||
|
|
47f7321c05 | ||
|
|
f95043e379 | ||
|
|
cb98a08384 | ||
|
|
4e822c56a0 | ||
|
|
c5fa1de264 | ||
|
|
ea359ea4e6 | ||
|
|
c22b58143b | ||
|
|
d9dd8a63d8 | ||
|
|
43c3f2c098 | ||
|
|
45e2d7e972 | ||
|
|
9844020589 | ||
|
|
535fd8bb54 | ||
|
|
cb4c18cedc | ||
|
|
8f8f1b4337 | ||
|
|
08d6476e52 | ||
|
|
80b647a2ed | ||
|
|
a4c5ef69bb | ||
|
|
ada703e790 | ||
|
|
2a61c57258 | ||
|
|
f8c0676b3a | ||
|
|
e4a55051cc | ||
|
|
9bfd57174f | ||
|
|
a227f98bdd | ||
|
|
409fa9bbc9 | ||
|
|
747fc546c1 | ||
|
|
046bbd3b30 | ||
|
|
99243ec47e | ||
|
|
0d66edb24c | ||
|
|
5ff97ec3aa | ||
|
|
75ac87c72b | ||
|
|
437523554e | ||
|
|
058d76ee8a | ||
|
|
db6430ccbf | ||
|
|
458f248c05 | ||
|
|
464833d0ef | ||
|
|
99a9da0950 | ||
|
|
de8a7176ab | ||
|
|
a7d4c39535 | ||
|
|
048f14540a | ||
|
|
c2ee44255e | ||
|
|
ddf23a7861 | ||
|
|
f3bbfd4bf6 | ||
|
|
893d90fb41 | ||
|
|
1e577168bf | ||
|
|
f349c940f0 | ||
|
|
1699375476 | ||
|
|
912fd428fb | ||
|
|
a75faad882 | ||
|
|
9f4757a8e4 | ||
|
|
93f8bbfc8e | ||
|
|
58edc4b4fb | ||
|
|
010a34cc33 | ||
|
|
35d5b2814f | ||
|
|
3c5edc3879 | ||
|
|
ab5025d508 | ||
|
|
6c8532ce70 | ||
|
|
c032cae100 | ||
|
|
071fc1c29d | ||
|
|
c2d5323dea | ||
|
|
21b79a04a2 | ||
|
|
bafc7b7d05 | ||
|
|
5e493934a5 | ||
|
|
c957fcc6d7 | ||
|
|
dcb5d5ce38 | ||
|
|
bb8075b37b | ||
|
|
40a39c8602 | ||
|
|
e5b98e3b0a | ||
|
|
2f3e6e6641 | ||
|
|
1405dd39e8 | ||
|
|
11294372b7 | ||
|
|
fd416cef91 | ||
|
|
74e45e9b48 | ||
|
|
c2a7016b62 | ||
|
|
18ce19b85d | ||
|
|
a0eccc713a |
@@ -0,0 +1,44 @@
|
||||
---
|
||||
name: hq-cleanup
|
||||
description: Use when the operator asks to clean up the forge and the working copies — stale or merged branches, old pull requests, finished worktrees, leftover folders, stashes. Triggers on "cleanup", "clean up our repos", "stale branches", "old worktrees", "tidy the forge". Reads first, deletes only what the operator approves, and reports what it cleaned, kept and left.
|
||||
---
|
||||
|
||||
# hq-cleanup
|
||||
|
||||
Runs playbook [09 — Cleanup](../../../00-META/process/09-cleanup.md); the playbook is authoritative. This skill
|
||||
adds the mechanics.
|
||||
|
||||
## 1. Take stock (one read-only agent, smaller model)
|
||||
|
||||
Brief it to change nothing and to write one report to the session's scratchpad:
|
||||
|
||||
- **Forge.** `gitea.gitea_list_repos`; per repository its default branch and **every branch from the git refs**
|
||||
(`gitea.gitea_api` GET `/repos/{owner}/{repo}/git/refs/heads`; the list tool misses branches, issue 949);
|
||||
`gitea.gitea_branch_protection_get`; open pull requests (`gitea.gitea_open_pulls`, titles and update times
|
||||
from `gitea.gitea_api` `/repos/{owner}/{repo}/pulls?state=open`). Per branch: contained in the default branch
|
||||
(`gitea.gitea_contains`, with the default branch's sha), patch-equal (`git cherry` in a local clone), tip date.
|
||||
- **Local.** Under the projects folder: each clone's `git worktree list --porcelain`, `git branch -vv`,
|
||||
`git branch --merged origin/<default>`, `git stash list`, `git status --porcelain` per worktree; folders and
|
||||
loose files beside the clones with `du -sh` and newest file. A worktree under another session's scratchpad is
|
||||
Kept.
|
||||
- A **Missing tools** section: each read it did without a mesh tool.
|
||||
|
||||
## 2. Classify and ask
|
||||
|
||||
Classes as playbook 09 step 2: Safe, Likely safe, Kept, Unknown. Tell the operator the counts and the report's
|
||||
place, and ask one question: delete the Safe list? Likely-safe and Unknown items are asked separately.
|
||||
|
||||
## 3. Delete the approved list only
|
||||
|
||||
- Forge branch: `gitea.gitea_delete_branch`.
|
||||
- Worktree: `git worktree remove <path>` (never `--force`), then `git worktree prune`.
|
||||
- Local branch: `git branch -d <name>` (never `-D`).
|
||||
Once design 58 milestone 7 is built (issue 884), each deletion is a call held as an ask: the operator approves
|
||||
the exact list on Telegram and the mesh carries it out on the warrant. Until then a session's permission check
|
||||
refuses these; hand the operator the approved list as a file, with the commands, and do not work around the
|
||||
refusal.
|
||||
|
||||
## 4. Report
|
||||
|
||||
Four lists: **cleaned** (with its evidence), **kept** (why), **unknown** (what it looks like), **the operator
|
||||
kept** (their word). Open each missing tool with `issue-tracker.open`, or note it on an existing issue.
|
||||
+214
-70
@@ -69,7 +69,8 @@ is listed under [Homonyms](#homonyms), with the qualified form each domain uses.
|
||||
**Recorded in** the catalogue, one manifest per module. **Upstream of** every other domain; its words
|
||||
are a published vocabulary, fixed by the manifest's schema. **Decided by** ADR 0040, 0126, 0174, 0188,
|
||||
0207, 0236, 0262.
|
||||
**Uses:** module, seat, provision (Provisioning), data class (Data), health (Health and repair).
|
||||
**Uses:** module, seat, config value (Config), provision (Provisioning), data class (Data), health (Health
|
||||
and repair).
|
||||
|
||||
- **manifest** — the file a module is defined by (`module.json`): what it is, what it provides and
|
||||
requires, the seats it claims, the resources it declares, its tools, its data and its health. Say
|
||||
@@ -83,34 +84,77 @@ are a published vocabulary, fixed by the manifest's schema. **Decided by** ADR 0
|
||||
it guards: the `mesh-controller`, `postgres` and `nats` modules claim the `mesh-controller`,
|
||||
`mesh-store` and `mesh-broker` seats (ADR 0079,
|
||||
ADR 0116).
|
||||
- **setting** — a value a manifest declares and an assignment gives, one of the two ways a node varies
|
||||
a module (ADR 0174).
|
||||
What a "flavor" once varied is a setting or a separate module.
|
||||
*Not:* ~~flavor~~
|
||||
- **preference** — a setting whose manifest gives it a default, because its value is true of the
|
||||
software and only tuned per node: a font size, a width
|
||||
(ADR 0262).
|
||||
A setting with no default is the operator's own value: a domain, an identity.
|
||||
*Not:* ~~tunable~~ (hq)
|
||||
- **kept region** — a marked block in a managed file the mesh writes *into*, where the operator's own
|
||||
lines survive every send and are given back when the module goes (ADR 0174). The other way a node
|
||||
varies a module.
|
||||
varies a module, besides a config value; a **managed** file's marked block is one (Provisioning).
|
||||
- **bundle** — the artifact a module's own code is built into — its tools, a seat's implementation, a
|
||||
daemon — in any language the mesh has a toolchain for; never an image. A **tools bundle** speaks MCP to
|
||||
the tool runner. One module may declare several
|
||||
(ADR 0188).
|
||||
- **tool** — one capability a module serves through the tool runner, addressed `<module>.<tool>`. A
|
||||
seat's **verb** is the same thing carried by a seat rather than a module.
|
||||
- **mesh-sdk** — the library a module's own code is written against, including the harness that serves a
|
||||
tools bundle to the tool runner. The harness is part of it, not a library of its own.
|
||||
- **mesh-sdk** — the library a module's own code is written against, including the SDK harness that serves a
|
||||
tools bundle to the tool runner. The SDK harness is part of it, not a library of its own.
|
||||
*Not:* ~~tools-sdk~~
|
||||
- **invokes** — the manifest word for the tools a module calls, `<module>.<tool>` each or `*` for every
|
||||
one. A grant on the publish side and nothing else; a module that declares none calls nothing.
|
||||
- **run for** — an assignment, or a start of a module, that serves one agent identity with that identity's machine
|
||||
account, home and bus password. Only a module whose manifest declares it may be run for an identity; running for
|
||||
the operator is the default run, as the operator account, with no setting (ADR 0412).
|
||||
- **upgrade policy** — how a module's new builds reach its nodes: `roll` (one node first, judged, then
|
||||
the rest), `together`, or `record` (register the build, send it nowhere)
|
||||
(ADR 0236).
|
||||
The controller's verb `upgrade` shows it.
|
||||
|
||||
## Config — the values someone may change
|
||||
|
||||
**Purpose.** To hold, check and resolve every value a module reads that someone may change, and to say
|
||||
where each resolved value came from. **Recorded by** the `mesh-config` module, in its buckets on the bus,
|
||||
with its store as the history behind them. **Upstream of** Provisioning, which places the values;
|
||||
downstream of Module and Placement. **Decided by** ADR 0262, 0277, 0348.
|
||||
**Uses:** module, node, manifest (Module), assignment (Placement), provision (Provisioning), proposal
|
||||
(Operator and conversation).
|
||||
|
||||
- **config value** — anything a module reads that someone may change: a font size, a domain, an address.
|
||||
A module declares every one it reads under `config` in its manifest, with a type, a description,
|
||||
validation, whether it is trusted, optionally the predefined key it falls back to, and a default or
|
||||
`required`; it lands in a template file as `${config:<key>}` or is read by the module's code at runtime.
|
||||
It resolves from four sources, the most specific winning: the default in the manifest; **mesh config**,
|
||||
the value of the predefined key it falls back to for the whole mesh; **node config**, that key's value
|
||||
on one node; and **module-assignment config**, the module's own value on one node. There is no module
|
||||
value for the whole mesh. *Config* is the concept, the verb (`config`) and the manifest field
|
||||
(ADR 0348). Mesh config is a source level, the value set for the whole mesh; `mesh-config` is the module
|
||||
that keeps every level. The controller holds, mirrors and resolves no config value (ADR 0348 §9).
|
||||
Older records and documents say "setting" for a config value and "settings" for the concept; new writing
|
||||
says config value and config. The word goes on this entry's *Not:* line once the documents in force that
|
||||
still use it in running prose are reworded, which is issue 683; until then it is checked by review. The
|
||||
manifest field and the verb "settings" are kept as aliases until every module has moved (ADR 0348 §1,
|
||||
issue 630), and are named on the line below with the rest once issue 683 is done.
|
||||
*Identifier until renamed:* `takes-settings`, `${setting:}` — manifest forms kept as aliases until every
|
||||
module has moved to their config names (ADR 0348 §1, issue 630).
|
||||
What a "flavor" once varied is a config value or a separate module.
|
||||
*Not:* ~~flavor~~
|
||||
- **predefined key** — a key of the catalogue's `predefined.json`, the fixed list of keys for which mesh
|
||||
config and node config hold values, each with its type, description, validation and whether it is
|
||||
trusted, such as `font-size`. A config value may name the predefined key it falls back to, of its own
|
||||
type; one that names none resolves from its default and the module's own value only. The list is the
|
||||
catalogue build's: a change to it is validated with every manifest naming a key, and removing a key still
|
||||
named or still holding a value is refused (ADR 0348).
|
||||
- **trusted** — said of a config value that decides what runs or who can reach what. At every level it is
|
||||
set only at the controller's terminal or through `config` with propose, on the operator's Approve; a
|
||||
module value that falls back to a trusted predefined key is itself trusted (ADR 0277, ADR 0348 §8). Not
|
||||
the trusted sender of an operator message (see [Homonyms](#homonyms)).
|
||||
- **preference** — a config value whose manifest gives it a default, because its value is true of the
|
||||
software and only tuned per node: a font size, a width
|
||||
(ADR 0262).
|
||||
A config value with no default is `required`: the operator's own value, such as a domain or an
|
||||
identity, and while nothing sets it the module's files are not written (ADR 0348).
|
||||
*Not:* ~~tunable~~ (hq)
|
||||
- **`mesh-config`** — the module that owns every config value, and not the source level mesh config: it
|
||||
holds the seat `mesh-config` once for the mesh, validates, takes a trusted value only from the
|
||||
controller's terminal or the operator's Approve, keeps the history with who set it and why, and resolves
|
||||
the four sources (ADR 0348).
|
||||
|
||||
## Core — the mesh's own machinery
|
||||
|
||||
**Purpose.** To keep the controller, the bus, the store and every node's engine running and agreed, so
|
||||
@@ -178,10 +222,11 @@ every other domain has something to run on. **Recorded by** the controller and t
|
||||
(ADR 0229).
|
||||
|
||||
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
|
||||
records (nodes, assignments, walks, builds, settings), after each change commits in its store and only
|
||||
records (nodes, settings, declared config, predefined keys), after each change commits in its store and only
|
||||
when the value differs, with one event per write. The store stays the record; the self-check compares
|
||||
the two and raises `mirror-drifted` on any difference
|
||||
(ADR 0289). Shortened to *mirror* where the
|
||||
(ADR 0289, ADR 0356). Builds, rollouts and assignments are no mirrors: they are
|
||||
the chain, kept on the bus as their only source (ADR 0408). Shortened to *mirror* where the
|
||||
text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is
|
||||
a different thing, a copy of an outside image in the artifact store.
|
||||
|
||||
@@ -189,10 +234,11 @@ every other domain has something to run on. **Recorded by** the controller and t
|
||||
|
||||
**Purpose.** To decide, send and apply what each node runs, and to say why.
|
||||
**Recorded by** the controller's `inventory` database. **Upstream of** Provisioning, Connectivity, Data
|
||||
and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 0252, 0261.
|
||||
**Uses:** node, machine, module, seat, setting (Module), manifest (Module).
|
||||
and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 0252, 0261, 0411.
|
||||
**Uses:** node, machine, module, seat, manifest (Module), config value (Config).
|
||||
|
||||
- **assignment** — a module put on a node, with the settings that node gives it.
|
||||
- **assignment** — a module put on a node. Its config values on that node (module-assignment config,
|
||||
Config) are `mesh-config`'s, never part of what the controller records (ADR 0348 §9).
|
||||
- **pending assignment** — an assignment of a module whose build is still running, kept, and made by the
|
||||
controller when that build registers the module, or ended with why when it does not
|
||||
(ADR 0261).
|
||||
@@ -221,15 +267,15 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
|
||||
other things (see [Homonyms](#homonyms)).
|
||||
- **apply** — the node-engine making its node match its declaration; **reconcile** is doing so again
|
||||
until nothing differs.
|
||||
- **operator account** — the login name of the person who works on a node, stated on the node record;
|
||||
empty for a node nobody logs into. Everything the mesh places under a person's home is resolved
|
||||
against this account's home and owned by it
|
||||
(ADR 0181).
|
||||
- **operator account** — the machine account of the operator's person identity on a node, read from the
|
||||
identity and no longer stated on the node record; empty for a node nobody logs into. Everything the mesh
|
||||
places under a person's home is resolved against this account's home and owned by it
|
||||
(ADR 0181, ADR 0411).
|
||||
Not a name a manifest carries.
|
||||
- **agent account** — the login every coding agent on a node runs as, when the node names one of its own;
|
||||
stated on the node record at the controller's terminal, never by a verb or a setting. Empty means agents
|
||||
run as the operator account. It cannot become root without a person, and the node-engine judges that on
|
||||
every look (ADR 0266).
|
||||
- **agent account** — the machine account of the agent identity the identity seat marks as a node's agent account,
|
||||
on the operator's Approve, never by an assignment or a config value. Where none is marked, agents run as the
|
||||
operator account. It cannot
|
||||
become root without a person, and the node-engine judges that on every look (ADR 0266, ADR 0411).
|
||||
- **account group** — a group of the machine's group database that a module puts an account in, by
|
||||
declaring the account with that group and nothing else of it. Several modules may each add one to the
|
||||
same account; the mesh takes back only an account group it added
|
||||
@@ -238,15 +284,21 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
|
||||
|
||||
## Provisioning — one module serving another
|
||||
|
||||
**Purpose.** To resolve which module serves a provision for which consumer, and to wire the two.
|
||||
**Recorded by** the controller. **Upstream of** Data; downstream of Module, Placement and Identity.
|
||||
**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263.
|
||||
**Uses:** module, seat, credential (Identity and access).
|
||||
**Purpose.** To resolve which module serves a provision for which consumer, to wire the two, and to
|
||||
place every value a module needs into its files on its node. **Recorded by** the controller and
|
||||
`mesh-provisioner`. **Upstream of** Data; downstream of Module, Placement, Config and Identity.
|
||||
**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263, 0349.
|
||||
**Uses:** module, seat, node-engine (Core), declaration (Placement), config value (Config), credential
|
||||
and vault (Identity and access).
|
||||
|
||||
- **provision** — a service one module `provides` and others `require`; the mesh resolves a provider
|
||||
and wires the two with an endpoint and a credential. A provision is a service you offer, a seat is a
|
||||
role you occupy, and the two meet where a seat delivers a provision: occupying the seat is what makes
|
||||
a module *the* provider of it.
|
||||
Every value a module needs is a provision from the domain that owns it: placement from the controller,
|
||||
config values from `mesh-config`, shared secrets from the vault, a consumer's values from the module
|
||||
providing it, an item of a collection from its contributor. A module says only what it needs and where
|
||||
it lands (ADR 0349).
|
||||
- **provider** and **consumer** — the module that provides a provision, and the module that requires
|
||||
it. Which of several providers serves a consumer is resolved
|
||||
(ADR 0084).
|
||||
@@ -264,27 +316,70 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
|
||||
node seat `node-mounts`, as a mount unit and an automount unit, mounted on access (ADR 0263).
|
||||
- **occasional source** — a share on a device that is not a node and comes and goes, such as a USB disk;
|
||||
its absence is a state, *not present*, never a fault (ADR 0263).
|
||||
- **mesh-provisioner** — the module that places provisions on the nodes: it holds the seat
|
||||
`mesh-provisioner` once for the mesh, keeping which domain answers which kind of placeholder, what each
|
||||
module on each node needs, what is applied where and the items of every collection, and the seat
|
||||
`node-provisioner` on every node, which its provisioner client serves (ADR 0349).
|
||||
- **provisioner client** — mesh-provisioner's part on one node. For each module there it reads the
|
||||
placeholders of its template files, records the needs, watches each owner's values for the node, fills a
|
||||
template only once every value is there, has the node-engine write the file through a narrow local verb,
|
||||
makes the program take the change as the file declares (`restart-on`, `reload-on`, `read-on-use`), and
|
||||
reports `prov.applied`. It runs as a service with a bus account of its own, never in the tool runner
|
||||
(ADR 0349).
|
||||
- **collection** — a provision with many providers. The **collector**, a module that takes additions from
|
||||
others (a shell's snippets, a bar's blocks), requires it; each **contributor** ships its **item** as a
|
||||
template in its own repository and provides it. The provisioner client fills the collector's template
|
||||
from the items of the contributors placed on its node, and an item goes when its contributor leaves the
|
||||
node (ADR 0349).
|
||||
- **owned** — a placed file that is the mesh's whole: the provisioner writes it back whenever it differs.
|
||||
It is one of three answers a manifest gives for each file it places (`owner-at-runtime`): **owned**,
|
||||
**seed** or **managed**. Whichever it is, the file has one writer: the node-engine writes it only on the
|
||||
provisioner client's word (ADR 0349).
|
||||
- **seed** — a placed file the provisioner writes once; from then on it is the program's.
|
||||
- **managed** — a placed file of which only its named keys or a marked block (a kept region, Module) are
|
||||
the mesh's; the program keeps the rest.
|
||||
|
||||
## Identity and access — who may do what
|
||||
|
||||
**Purpose.** To issue, hold, rotate and check the credentials and grants by which modules, nodes, agents
|
||||
and the operator act. **Recorded by** the controller's `identity` and `licences` databases and the vault.
|
||||
**Upstream of** Provisioning, Change and delivery, and Operator and conversation.
|
||||
**Decided by** ADR 0085, 0113, 0183, 0225, 0234.
|
||||
**Uses:** module, node, operator, consumer (Provisioning).
|
||||
**Decided by** ADR 0085, 0113, 0183, 0225, 0234, 0386, 0387, 0388, 0411, 0413, 0414.
|
||||
**Uses:** module, node, seat, tool (Module), operator, consumer and provision (Provisioning), agent and ask (Operator and conversation).
|
||||
|
||||
- **credential** — anything that proves an identity: a password, a key, a token. A **pair credential**
|
||||
is the two ends of one credential, the consumer's and the provider's.
|
||||
- **secret** — a credential or other value the vault keeps sealed and the node-engine unseals into a
|
||||
file at apply (ADR 0085).
|
||||
- **secret family** — a module's own secrets declared as one name ending in `-*`; each member is the prefix
|
||||
and a name (`smb-password-<source>`, one per source the module's settings name), given by its full name
|
||||
and a name (`smb-password-<source>`, one per source the module's config values name), given by its full name
|
||||
and never made by the mesh
|
||||
(ADR 0283).
|
||||
- **vault** — the module that owns every secret (`mesh-vault`,
|
||||
[24 — The secrets vault](../03-DESIGN/01-to-be/24-the-secrets-vault.md)).
|
||||
- **grant** — what an identity may call or reach, bounded by the provisions it requires
|
||||
(ADR 0225).
|
||||
- **identity** — a person, an agent or a module as the mesh knows it: a name, a kind, a role, and whether it
|
||||
is switched on; a person or an agent also has a login name, the machines it is present on and, for a person,
|
||||
the login kept on a machine where it differs. Every identity and role is held by the identity seat, in the
|
||||
bus's key-value store, and changes only through its verbs (ADR 0386, ADR 0411). Each person and agent
|
||||
identity has a bus account of its own, carrying its role (ADR 0413).
|
||||
- **machine account** — an identity's login on one machine, made by the mesh from the identity: the
|
||||
controller composes it and the node-engine applies it, adopting a login that exists and never renaming or
|
||||
deleting one (ADR 0411).
|
||||
- **role** — what an identity may call, as a set of grants (ADR 0386). Not a seat: a bare *role* is a
|
||||
homonym (see [Homonyms](#homonyms)).
|
||||
- **grant** — one line of a role: an effect everywhere, an effect on one seat, or one named tool, and may be
|
||||
limited to named machines; a dangerous tool is granted only by name (ADR 0386). A module's role
|
||||
is made from what its manifest declares it calls (ADR 0386 §4).
|
||||
- **effect** — a tool's mark, set by its module in its manifest: **reads**, **acts** or **dangerous**; a
|
||||
tool served without one counts as dangerous (ADR 0387).
|
||||
- **break-glass agent** — the one agent that holds every tool and never asks: sealed until the operator, or
|
||||
a condition the operator named, opens it, open until the operator closes it, and every act it takes told
|
||||
at once (ADR 0387).
|
||||
- **signed login** — a bus login signed by a key the bus trusts, carrying its expiry, which the bus checks by
|
||||
its signature instead of reading a list (ADR 0388). The identity seat signs with one **signing key** per
|
||||
role.
|
||||
- **root key** — the key at the top of the signed logins, which vouches for the signing keys: sealed in the
|
||||
vault and used only to replace them, on the operator's Approve (ADR 0388).
|
||||
- **bus account** and **membership** — a module's or a node's account on the bus, and the subjects it
|
||||
is issued. The bus server calls its accounts *users*; the mesh says bus account.
|
||||
- **rotate** — replacing a credential without a consumer holding one the provider does not know.
|
||||
@@ -306,8 +401,8 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
|
||||
|
||||
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
|
||||
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
|
||||
planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323.
|
||||
planner, the rollout). **Downstream of** Module, Core, Health and repair and Data.
|
||||
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323, 0401–0408.
|
||||
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
|
||||
|
||||
- **merge check** — the two statuses a pull request carries before it may merge: the **merge gate**
|
||||
@@ -322,7 +417,10 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
**builder**. Any node may hold the seat, so there is no build node by nature
|
||||
(ADR 0237).
|
||||
*Not:* ~~build machine~~
|
||||
- **build queue** — the controller's queue of builds to run; one entry in it is a **build request**.
|
||||
- **build queue** — the builds waiting to run, kept by the bus as three work queues a builder takes in order:
|
||||
delivery builds, then checks of pull requests whose Approve is asked or given (ADR 0421 §10), then every other check; each check queue keeps one
|
||||
request per pull request, the newest (ADR 0406); one entry in it is a
|
||||
**build request**.
|
||||
The queue's verbs still say *ask* for an entry; the conversation's ask is another thing
|
||||
(see [Homonyms](#homonyms)).
|
||||
- **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by **version**.
|
||||
@@ -348,7 +446,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
- **delivery** — one commit in one repository on its way to the nodes, from its pull request's head
|
||||
being announced to one of its final states: one pull request, one status, one note
|
||||
(ADR 0239).
|
||||
From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk
|
||||
From its merge on it is delivered by the rollout of the batch that answers it, and ends with that rollout
|
||||
(ADR 0276).
|
||||
Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`,
|
||||
`held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`;
|
||||
@@ -357,13 +455,21 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
mechanism, which the as-is designs describe; it is never a word for a delivery.
|
||||
*Not:* ~~deployment~~ (hq)
|
||||
- **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was
|
||||
sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own
|
||||
sent to every machine running it by the rollout of a later merge into the same trunk, delivered on its own
|
||||
passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery
|
||||
whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan
|
||||
whose rollout a newer delivery took over before it ended; nor *delivered*, which its own rollout or an empty plan
|
||||
ends in (ADR 0270).
|
||||
- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's
|
||||
repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One
|
||||
level: a group holds deliveries, never groups. Its state is derived from its members, never set.
|
||||
- **hotfix** — one pull request in one repository, delivered at its merge (ADR 0401).
|
||||
- **feature** — one branch name across its repositories: in two or more, its pull requests are one delivery group,
|
||||
merged together and delivered member by member in the group's order; in one, a single pull request delivered as a
|
||||
lone delivery (ADR 0401). Work is never held back on a branch other than a trunk: the trunk is what runs.
|
||||
*Not:* ~~release branch~~ (hq)
|
||||
- **toolchain** — a language's compiler and tools, installed by that language's build module on every node holding
|
||||
the build seat, pinned to one version that is the same on every builder; its version is part of a build's reuse
|
||||
key (ADR 0405). No longer an image.
|
||||
- **delivery plan** — what a delivery does to the mesh: its **build plan** (the modules it moves and
|
||||
their dependents, in tiers), its **deploy plan** (per node, what it receives and what waits for a
|
||||
person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a
|
||||
@@ -371,19 +477,21 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
*Not:* ~~change plan~~, ~~release plan~~
|
||||
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
|
||||
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
|
||||
10 minutes after its first merge (ADR 0276).
|
||||
- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more
|
||||
than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*,
|
||||
10 minutes after its first merge (ADR 0276). Kept as two keys with a time to live on the bus; the first to
|
||||
expire closes the batch (ADR 0407).
|
||||
- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more
|
||||
than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*,
|
||||
which is a delivery group.
|
||||
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
|
||||
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
|
||||
- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
|
||||
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
|
||||
ADR 0276). One walk is open at a time; it names every merge it answers. A primitive the delivering stage
|
||||
asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules
|
||||
built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own:
|
||||
a module *rolls out* by its `roll` policy.
|
||||
*Not:* ~~rollout~~ (hq)
|
||||
- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
|
||||
repository, tier by tier, to every node that runs each module at once (ADR 0276, ADR 0360). Rollouts that share a
|
||||
repository, a module or a graph edge run one after another; others run side by side (ADR 0407). It names every
|
||||
merge it answers, and its record is one key of the bus's rollouts bucket, a link of the chain (ADR 0408). A primitive the delivering stage
|
||||
asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules
|
||||
built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll`
|
||||
policy.
|
||||
*Not:* ~~walk~~ (hq)
|
||||
- **first-node gate** — the judgement on a delivery's first node: healthy three times, no witness put it
|
||||
back, no condition raised since the send (ADR 0236 §2). Not *the gate* bare (see [Homonyms](#homonyms)).
|
||||
- **core module** and **leaf module** — a core module is one on the controller's own path (the controller,
|
||||
@@ -394,7 +502,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
the core modules add mesh-delivery and the resolver.
|
||||
- **delivery time** — from a delivery's merge to every node of its deploy plan that is heard from running its
|
||||
modules' builds and reporting them healthy. Its **delivery budget** is what it is held to: three minutes for
|
||||
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the walk's time, which ends at the send. A
|
||||
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the rollout's time, which ends at the send. A
|
||||
measure inside the fix budget, not the operator's target (ADR 0313).
|
||||
- **fix time** — from an issue's first move to `fixing` to the moment the last of its fixes is delivered: the
|
||||
operator's measure of a fix, the agent's work, the review and the checks included, the delivery time its last
|
||||
@@ -422,7 +530,7 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283.
|
||||
running session, started before, does not have that group yet. Said by the node-engine as the module's
|
||||
resource of kind `account`; it clears at the first look after a new login (ADR 0252).
|
||||
- **waiting** — a resource's health when its module's tool check says it waits for the operator: a named own
|
||||
secret or setting of the module, not given yet, without which a part cannot work. Not a fault: the
|
||||
secret or config value of the module, not given yet, without which a part cannot work. Not a fault: the
|
||||
controller checks it and says it as the condition `needs-operator`, naming the act, and the first-node gate
|
||||
reads it as *waits for a person*
|
||||
(ADR 0283).
|
||||
@@ -503,8 +611,8 @@ it should not. **Recorded by** the controller. **Upstream of** Provisioning and
|
||||
trust. **Recorded by** the router (asks, warrants and the operator's identities).
|
||||
**Downstream of** Health and repair, Change and delivery, and Identity and access. An authorising
|
||||
answer acts in whichever domain asked, through that domain's own verb; the conversation owns the
|
||||
asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
|
||||
**Uses:** operator, person, proof (Identity and access), release (Change and delivery).
|
||||
asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272, 0383, 0384, 0385, 0390.
|
||||
**Uses:** operator, person, node, module, seat, tool (Module), proof and identity (Identity and access), release (Change and delivery).
|
||||
|
||||
- **mesh MCP server** — the endpoint on a node's loopback through which every agent and person on that
|
||||
node reaches the mesh: the MCP server named `mesh`, with its five tools `mesh_overview`,
|
||||
@@ -518,8 +626,8 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
|
||||
line as the **controller's terminal** or as an ordinary call
|
||||
(ADR 0272).
|
||||
- **nox** — `mesh-cli`'s alias, in interactive zsh only. A script, `sudo` and a document meant to work everywhere
|
||||
say `mesh-cli`. The operator also calls the mesh itself *nox mesh*; on this page and in the records *nox* is the
|
||||
alias alone. It is the one command the operator types: every line the mesh addresses to the operator — a
|
||||
say `mesh-cli`. The operator also calls the mesh itself *nox mesh*. *nox* is also the agent (ADR 0414 §7): see
|
||||
[Homonyms](#homonyms), and say which is meant. It is the one command the operator types: every line the mesh addresses to the operator — a
|
||||
condition's words, a verb's answer, a README, a design — names a `nox` line, never a daemon's (the controller's
|
||||
or the node-engine's binary followed by a command), which `words.py` and the catalogue's check refuse
|
||||
(ADR 0272,
|
||||
@@ -546,7 +654,7 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
|
||||
published for that asker alone. The only thing an asker acts on (ADR 0259).
|
||||
- **proposal** — a change an agent, or anyone the bus admits, puts forward through a verb and that takes
|
||||
effect only on the operator's warrant, bound to the exact thing shown: a shared instruction section (ADR
|
||||
0274), a trusted setting's layer (ADR 0277).
|
||||
0274), a trusted config value (ADR 0277, ADR 0348).
|
||||
A proposal changes nothing by itself; declined, expired or refused, it is discarded.
|
||||
- **approval mark** — a tool's or verb's declaration in its manifest (`needs-approval`) that each call to it runs
|
||||
only on the operator's warrant for that one call, with its exact arguments; the tool runner that serves it, or
|
||||
@@ -559,11 +667,24 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
|
||||
instructions.
|
||||
- **reference** — an opaque token in a message's words standing for a detail the content rule keeps out
|
||||
of them (a path, an address); opened with `detail` only on a `private` channel or through the mesh MCP server.
|
||||
- **agent** — a coding agent: the operator's, or one the mesh runs. Not the node-engine, and not the
|
||||
- **agent** — a coding agent: the operator's, or one the mesh runs; to the mesh, an identity of kind agent
|
||||
(ADR 0386). Not the node-engine, and not the
|
||||
build seat's holder, whose seat name `node-build-agent` is an identifier to rename.
|
||||
- **thread** — one line of agent work, owned by the mesh: its turns, tool calls and results, its workspace
|
||||
and usage, kept whole for good on the bus by the thread seat; it outlives any session, node or front end
|
||||
(ADR 0383, ADR 0384).
|
||||
- **session** — one run of an agent harness on a thread.
|
||||
- **agent harness** — the program that runs an agent's sessions, its context manager and loop: Claude Code
|
||||
today (ADR 0384). Not the SDK harness (see [Homonyms](#homonyms)).
|
||||
- **thread handler** — the node seat whose holder runs threads' sessions headless, on the always-on nodes
|
||||
(ADR 0384).
|
||||
- **workspace toolkit** — the tools a session works with on its thread's workspace, the same for every agent
|
||||
harness: the node seat `workspace` (ADR 0385).
|
||||
- **sandbox** — the small virtual machine one session runs in, with its thread's workspace mounted
|
||||
(ADR 0390).
|
||||
- **instruction section** — one named section of the managed instruction file, which every agent session on the
|
||||
nodes it is for reads first, as the mesh's word: set at the controller's terminal in the agent module's
|
||||
settings, or **proposed** by an agent and made active on each node only by the operator's warrant, which that node
|
||||
config, or **proposed** by an agent and made active on each node only by the operator's warrant, which that node
|
||||
proves itself (ADR 0274). A hand-over between sessions is one.
|
||||
- **personal rule** — one short text in the operator's own words about how the operator wants their agent
|
||||
to work with them, registered for the operator (the **operator scope** of the agent module) and rendered
|
||||
@@ -578,10 +699,10 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
|
||||
## Media — what a person watches, listens to and reads
|
||||
|
||||
**Purpose.** To find, fetch and file copies of the works a person watches, listens to and reads, each in
|
||||
the library its kind names. **Recorded by** the `media-importer` module's settings and its memory of what
|
||||
the library its kind names. **Recorded by** the `media-importer` module's config values and its memory of what
|
||||
it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Decided by** ADR 0295, 0296,
|
||||
0302.
|
||||
**Uses:** module, seat, setting (Module).
|
||||
**Uses:** module, seat, config value (Config).
|
||||
|
||||
- **content** — a work a person watches, listens to or reads: a film, an album and its tracks, an
|
||||
episode, a book, a session of a sport event. It is known by an identity from a catalogue outside the
|
||||
@@ -592,7 +713,7 @@ it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Dec
|
||||
- **media release** — one copy of a work an engine can fetch, such as an NZB or a video. Downloaded, it
|
||||
sits in the downloads folder, not yet filed. ADR 0302 says *release*.
|
||||
- **library** — one folder of one kind, filed by one importer: the media-importer's, named in its
|
||||
settings, or a servarr app's. Several libraries may hold one kind, as TV Shows and Anime both hold
|
||||
config, or a servarr app's. Several libraries may hold one kind, as TV Shows and Anime both hold
|
||||
series.
|
||||
- **media file** — a file on disk together with its metadata: its kind, the fields that identify the
|
||||
work, and what part of the work the file is. It is what the media-importer takes.
|
||||
@@ -611,7 +732,7 @@ it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Dec
|
||||
|
||||
Not a domain of the mesh but of the way it is built, listed because its words meet the mesh's.
|
||||
**Recorded in** this repository, the issue tracker and the knowledge base. **Decided by** ADR 0019, 0080, 0244, 0275,
|
||||
0284, 0290, 0291.
|
||||
0284, 0290, 0291, 0423, 0424.
|
||||
|
||||
- **research effort**, **graduation**, **hand-off**, **playbook** — an investigation, held whole in the
|
||||
knowledge base and named "research NNN";
|
||||
@@ -624,17 +745,29 @@ Not a domain of the mesh but of the way it is built, listed because its words me
|
||||
`changed`, `superseded`): a later record may change one clause and leave the rest
|
||||
(ADR 0290). Not a *rule*, which is a line of
|
||||
`00-META/how-we-build.md`.
|
||||
- **knowledge base** — the role that holds the decision records and research efforts whole (number, status, topic,
|
||||
typed links, clauses and reasoning, with every version), takes them as proposals, has them reviewed and accepted,
|
||||
- **knowledge base** — what holds the decision records and research efforts whole (number, status, topic,
|
||||
typed links, clauses and reasoning, with every version), and, once they are moved, the designs and the glossary
|
||||
entries too (ADR 0423); takes them as proposals, has them reviewed and accepted,
|
||||
and answers what holds: the seat `knowledge-base`, held by the module `mesh-knowledge`
|
||||
([55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md)). Not **the record**, the literal reader
|
||||
of this repository.
|
||||
- **design** — a document in `03-DESIGN/`: **as-is** (what runs) or **to-be** (what is being built).
|
||||
- **design** — a record of the knowledge base, named "design NN", with a **to-be** section (what is being built)
|
||||
and an **as-is** section (what runs), its owning repositories and its links to its decisions and other designs
|
||||
(ADR 0423). Until the move, a document in `03-DESIGN/`, in `01-to-be/` or `00-as-is/`.
|
||||
- **design state** — of a design: proposed, designed, in progress or implemented, worked out by the knowledge base
|
||||
from the issues with a design link to it and whether their deliveries are done; never written by anyone
|
||||
(ADR 0424).
|
||||
- **glossary entry** — one entry of the glossary as a record of the knowledge base: its head word or words, its
|
||||
domain, its meaning, its *Not:* line and its identifier line (ADR 0423). Until the move, an entry of this page.
|
||||
- **milestone** — a group of a design's issues delivered together, as one delivery group, or one pull request when
|
||||
it touches one repository; named by what the operator or the mesh can do once it is delivered (playbook 04,
|
||||
ADR 0401).
|
||||
*Not:* ~~epic~~ (hq)
|
||||
- **issue** — a question about the mesh that someone must answer, numbered and tracked by the issue tracker, which
|
||||
holds it whole: its report, diagnosis and resolution included (ADR 0291);
|
||||
the folders in `04-ISSUES/` are imported there and removed.
|
||||
An **incident** is a past event that taught a rule.
|
||||
- **issue tracker** — the role that holds the mesh's issues and hands out their numbers: the seat `issue-tracker`,
|
||||
- **issue tracker** — what holds the mesh's issues and hands out their numbers: the seat `issue-tracker`,
|
||||
held by the module `mesh-issues` ([53 — The issue tracker](../03-DESIGN/01-to-be/53-the-issue-tracker.md)).
|
||||
Not hits's *item*; bare *ticket* is a homonym (see [Homonyms](#homonyms)).
|
||||
*Not:* ~~ticketing system~~ (hq), ~~ticket system~~ (hq)
|
||||
@@ -672,7 +805,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
|---|---|---|---|
|
||||
| plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement |
|
||||
| | what a delivery would do to the mesh | **delivery plan** | Change and delivery |
|
||||
| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery |
|
||||
| | the sending of one batch of commits across nodes | **rollout** (verb `plans`) | Change and delivery |
|
||||
| | a step a person starts, like the bus's | **planned step** | Core |
|
||||
| membership | the subjects a bus account is issued | **membership** | Provisioning |
|
||||
| | an account in a group of the machine | **account group** | Placement |
|
||||
@@ -686,8 +819,13 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
| | the pull request status | **merge gate** | Change and delivery |
|
||||
| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement |
|
||||
| tier | a level of the mesh | **layer** | Core |
|
||||
| | a step of a walk | **tier** | Change and delivery |
|
||||
| | a step of a rollout | **tier** | Change and delivery |
|
||||
| | how much proof an ask needs | **assurance level** | Identity and access |
|
||||
| harness | the part of `mesh-sdk` that serves a tools bundle | **SDK harness** | Module |
|
||||
| | the program that runs an agent's sessions | **agent harness** | Operator and conversation |
|
||||
| | the TypeScript `provisioner` helper for database providers | **`provisioner`**, renamed by design 57 | Provisioning |
|
||||
| role | what a seat names, at a scope | **seat**; the bare word in older designs is tracked by issue 872 | Module |
|
||||
| | what an identity may call | **role** | Identity and access |
|
||||
| ask | a request for the operator's input | **ask** | Operator and conversation |
|
||||
| | an entry in the build queue | **build request** | Change and delivery |
|
||||
| store | the one database server | **store** | Core |
|
||||
@@ -696,6 +834,8 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
| record | a numbered decision | **decision record** | The record |
|
||||
| | the literal reader of this repository, searched first for a symptom | **the record** (the `records` module) | Operator and conversation |
|
||||
| | the upgrade policy that sends nowhere | `record` | Module |
|
||||
| nox | the mesh's one agent identity, its logins and accounts (ADR 0414) | **`nox`, the agent** | Identity and access |
|
||||
| | the operator's terminal command, `mesh-cli` | **the `nox` command** | Operator and conversation |
|
||||
| knowledge base | the module holding the decision records and research efforts whole | **knowledge base** | The record |
|
||||
| | the predecessor's operational memory, in older playbooks (issue 430) | none: it no longer exists | — |
|
||||
| check | a pull request's two statuses | **merge check**, **merge gate**, **repository check** | Change and delivery |
|
||||
@@ -712,6 +852,10 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
| | a question someone must answer | **issue** | The record |
|
||||
| deploy | the per-node part of a delivery plan | **deploy plan** | Change and delivery |
|
||||
| | a wrapped program's own deploy | its own word | — |
|
||||
| managed | a placed file of which only named keys or a marked block are the mesh's | **managed** (`owner-at-runtime`) | Provisioning |
|
||||
| | a file the mesh writes, such as the agent's instruction file or `managed_settings` | the file's own name, or *a file the mesh places* | — |
|
||||
| trusted | a config value that decides what runs or who can reach what | **trusted** config value | Config |
|
||||
| | a sender on the operator's own list | **trusted** sender, its message an **operator message** | Operator and conversation |
|
||||
|
||||
## How this page is kept
|
||||
|
||||
|
||||
+26
-1
@@ -1,9 +1,10 @@
|
||||
---
|
||||
status: canonical
|
||||
updated: 2026-10-11
|
||||
updated: 2026-10-16
|
||||
derives: the constitution page
|
||||
decisions:
|
||||
- "ADR 0020"
|
||||
- "ADR 0391"
|
||||
---
|
||||
|
||||
# How we build
|
||||
@@ -172,6 +173,30 @@ quietly stop being true, and nobody will learn that from a document.
|
||||
Not test-driven development as a blanket rule — a test written first against undiscovered
|
||||
behaviour asserts a guess. The obligation is that every decision has a defender.
|
||||
|
||||
### Tests run at the lowest level that proves them, within a time budget
|
||||
|
||||
Three levels, by how much real system a test uses (ADR 0391):
|
||||
|
||||
| Level | Uses | Budget | Runs |
|
||||
|---|---|---|---|
|
||||
| **Unit** | one piece of logic alone: no database, network, bus, running service or other process | well under a second per test | locally, the tests of what changed, while working; and in the forge's checks |
|
||||
| **Integration** | the real thing beside it: a database, the bus, a catalogue tree | the repository's whole suite in seconds, never over sixty | only in the forge's checks, once per push |
|
||||
| **End to end** | the whole mesh, or a lab mesh, from the outside | — | only when the operator asks |
|
||||
|
||||
- **Test each behaviour at the lowest level that can prove it.** A defect found higher up gets a test at the
|
||||
lowest level that reproduces it, before its fix.
|
||||
- **A test that needs longer than a second, or a database, the network or a service, is not a unit test.**
|
||||
- **A suite over sixty seconds is a defect with its own issue**: it is made faster or split, not accepted.
|
||||
- **Never run a merge check, a throwaway database or an integration suite locally.** The forge's checks do,
|
||||
on the builder.
|
||||
- **A report names its tests and how long they took**, so a test over its budget is seen when it is added.
|
||||
|
||||
### Tests within a time budget — how it was earned
|
||||
|
||||
On 2026-10-16 a feature's fixing agents ran the catalogue's whole merge check (about ten minutes), a
|
||||
throwaway database and a locally built gate during local development. Each round cost tens of minutes and
|
||||
caught neither real failure; the forge's checks found both.
|
||||
|
||||
### A report is read from the system, never from what asked for it
|
||||
|
||||
The same rule as the two above, pointed at reporting rather than at verification. **Anything
|
||||
|
||||
@@ -50,13 +50,15 @@ the expensive half.
|
||||
|
||||
| # | Playbook | Trigger |
|
||||
|---|---|---|
|
||||
| [01](01-research.md) | Research | An idea worth investigating before committing to design |
|
||||
| [02](02-graduation.md) | Graduation & design change | Research concludes, or a design must change |
|
||||
| [01](01-research.md) | Research | An idea from the operator, or one worth investigating before committing to design: setup questions, findings, a direction the operator confirms |
|
||||
| [02](02-graduation.md) | Graduation & design change | Research concludes (its direction confirmed), or a design must change: decision records, the to-be design |
|
||||
| [03](03-issues.md) | Issues | Something is wrong — often with the owner unknown |
|
||||
| [04](04-build-handoff.md) | Build handoff | A design is ready to be built |
|
||||
| [04](04-build-handoff.md) | Work breakdown and handoff | A design is ready to be built: milestones usable on their own, issues per repository, each milestone one feature step |
|
||||
| [05](05-constitution-sync.md) | Constitution sync | `how-we-build.md` changed a rule the mesh enforces |
|
||||
| [06](06-writing-a-module.md) | Writing a module | Something that runs today must run on the mesh |
|
||||
| [07](07-feature-branches.md) | Feature branches across repos | Work that changes code, in one repo or several at once |
|
||||
| [07](07-feature-branches.md) | Features | Work that changes code across issues or repositories: prepared context, local development, one review, one check, one delivery |
|
||||
| [08](08-small-fixes.md) | Small fixes | One issue in one repository, about an hour of work: one review, one push, one delivery |
|
||||
| [09](09-cleanup.md) | Cleanup | Merged branches, stale pull requests, finished worktrees and folders: read first, delete only what the operator approves, report the rest |
|
||||
|
||||
## The cycle is checked
|
||||
|
||||
|
||||
+113
-39
@@ -1,66 +1,140 @@
|
||||
# Playbook 01 — Research
|
||||
|
||||
**Trigger.** An idea, technology or approach worth investigating before it is committed to
|
||||
design. Also: an as-is document that raises a question nobody can answer.
|
||||
**Trigger.** An idea the operator expresses, or an idea, technology or approach worth investigating before
|
||||
it is committed to design. Also: an as-is document that raises a question nobody can answer. A symptom (a
|
||||
bug, something broken) is not research: it follows playbook [03](03-issues.md).
|
||||
|
||||
**Who runs it.** Anyone.
|
||||
**Who runs it.** Anyone; in an agent session, the session that heard the idea leads it.
|
||||
|
||||
**Where it lives.** A research effort lives whole in the knowledge base, the seat `knowledge-base`, reached
|
||||
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none: an effort
|
||||
is written with the knowledge base's verbs, never as a folder here.
|
||||
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none.
|
||||
|
||||
## The cycle it starts
|
||||
|
||||
```
|
||||
idea ─► research (01) ─► decisions and design (02) ─► work breakdown in milestones (04)
|
||||
─► each milestone built and delivered as one feature step (07) ─► what is reported afterwards (03, 08)
|
||||
```
|
||||
|
||||
An idea arrives in one of two forms, and the first step differs:
|
||||
|
||||
- **A new idea**, where only the mesh's own context matters: read what the knowledge base holds on it.
|
||||
- **An idea for an existing application**: read that application's to-be and as-is designs, its decision
|
||||
records and its open issues first; the effort `touches` them.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Read what is known first.** `knowledge-base.ask` with the question in plain words returns the decision
|
||||
records and research efforts that bear on it; `knowledge-base.get` with `kind: research` and a number
|
||||
reads one effort, and `reasoning: true` (or `section` with a document's name) reads its documents.
|
||||
2. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The
|
||||
number comes from the issue tracker's counter and is spent whether or not the effort is ever proposed;
|
||||
one never proposed is withdrawn after a week. Never pick a number by hand.
|
||||
3. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number` reserve gave, a one-line
|
||||
`title`, `status: active`, `date` (the day it began), and `sections`: its documents in order, each a
|
||||
`name` and its markdown `text`. The first is `00-overview.md`: **what** is being investigated, **why**,
|
||||
and **what it touches**. Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`,
|
||||
`"issue 365"`, or a design's path), not only in prose, and what it names for another reason as `cites`.
|
||||
4. **Do the research** in further documents of the same effort: notes, option analyses, evidence, draft
|
||||
designs. Each change is `knowledge-base.propose` again with the effort whole and `why`, one line saying
|
||||
why this version. Every version is kept; `knowledge-base.history` lists them and gives the difference
|
||||
between two.
|
||||
1. **Read what is known.** `knowledge-base.ask` with the idea in plain words returns the decision records and
|
||||
research efforts that bear on it; `knowledge-base.get` reads one (`reasoning: true`, or `section`, for its
|
||||
documents). Read the issue tracker for open issues in the same area (`issue-tracker.list`).
|
||||
2. **Ask the operator to set it up.** A few questions, one at a time, each answerable with a choice and
|
||||
carrying a recommendation: what comes first, what is out of scope, which options must be tried, what
|
||||
done looks like. Three to five questions is usual; the answers are the effort's first findings.
|
||||
- **Check every option against the mesh's own rules before offering it.** An option that breaks one
|
||||
(the core values in [`mission.md`](../mission.md), among them *the mesh owns the truth* and
|
||||
*sovereignty: external dependencies are deliberate and few*; the rules in
|
||||
[`how-we-build.md`](../how-we-build.md); and everything between modules over the bus, research 036) is
|
||||
dropped, or offered only with the rule it breaks named. A recommendation the operator has to argue back
|
||||
costs a version.
|
||||
- **Explain each question with a concrete picture** in the operator's words: what they would see, do or
|
||||
lose under each option. No internal names (a record number, a verb, a word the glossary lacks) without
|
||||
what it is beside it.
|
||||
3. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The number
|
||||
is spent whether or not the effort is ever proposed; one never proposed is withdrawn after a week. Never
|
||||
pick one by hand.
|
||||
4. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number`, a one-line `title`,
|
||||
`status: active`, `date` (the day it began), and `sections`, each a `name` and its markdown `text`:
|
||||
- `00-overview.md`: the idea in the operator's words, what exists, the operator's answers (dated), and the
|
||||
words the effort uses that the glossary lacks, each defined in one line;
|
||||
- `01-questions.md`: the questions to answer, the first ones first;
|
||||
- `02-method.md`: what will be tried and measured, and on what.
|
||||
|
||||
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
|
||||
nothing, a retired word of the glossary, a daemon's line), naming every reason at once and writing nothing.
|
||||
The knowledge base then reviews the proposal itself, through a judgement that sees only the record, the
|
||||
records it links and the rules; on `changes` the effort stays proposed with the findings, and its author
|
||||
proposes it again. An effort that needs the operator's word is proposed with `needs-operator: true` and
|
||||
waits for the operator's Approve on a channel that proves who answered.
|
||||
Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`, `"issue 365"`, or a
|
||||
design's path), and what it names for another reason as `cites`.
|
||||
5. **Do the research.** Try, measure and read, smallest question first. Each finding goes into
|
||||
`04-findings.md`: what was tried, what it showed (with numbers), what it means. Each change of the effort
|
||||
is `knowledge-base.propose` again, whole, with `why`; every version is kept, and `knowledge-base.history`
|
||||
lists them and gives the difference between two. Until issue 844 is delivered, the knowledge base
|
||||
takes a new version of an accepted, active effort only as a progressive insight: give `insight`, saying
|
||||
what the effort held before the new finding, and mark the finding in the text with its date.
|
||||
- **Propose a version per settled set, not per answer.** While the operator answers a run of questions,
|
||||
or while experiments run, keep the answers and findings in the session's save point and propose once
|
||||
when the set is settled. Each version is reviewed and resent whole, so a version per answer multiplies
|
||||
the reviews.
|
||||
- **Pre-check before proposing**, against what the review sends back most: every record, issue or design
|
||||
the text names is in `links`; work put off names the issue that tracks it (an acceptance test is not
|
||||
put-off work: it is what the build proves, step 7); each figure is given as measured (a range, not a
|
||||
rounded claim); no word on a glossary *Not:* line.
|
||||
- **Try at the cheapest level that answers the question.** A program's own behaviour (a library, a
|
||||
server, a protocol) is tried in process, on the machine's loopback, in minutes. Lab runs (about 12 to
|
||||
16 minutes fresh) are for the mesh's own code; before that code exists, a lab run tests nothing an
|
||||
in-process test does not, and its lab scenario is written as the build's acceptance test instead.
|
||||
6. **Place it in the domains.** Before the direction, write `03-domains.md`: the domain or domains the idea
|
||||
belongs to (the glossary's sections, ADR 0244); the modules and seats it lands in, each new, changed or
|
||||
retired; what it does to each domain's words, each word added (with its domain and a one-line meaning),
|
||||
changed or retired, and none on a *Not:* line or already meaning something else (the glossary's homonyms);
|
||||
and whether a domain grows or shrinks, a concept moving between domains included. The direction names its
|
||||
decisions in those words. Asked when the idea is set up, too: an option that lands a concept in the wrong
|
||||
domain is offered with that cost named, like one that breaks a rule.
|
||||
7. **Settle the direction with the operator.** When the questions are answered, write `05-direction.md`:
|
||||
the proposed decisions in a few lines each, the options rejected and why, and the build's acceptance
|
||||
tests: what the build must prove before it is done, carried by its milestones (playbook
|
||||
[04](04-build-handoff.md)). Nothing else is left open: each question closes by a test, by a decision the
|
||||
effort records with its reason (a small design limit is the lead's), or by the operator. Ask the
|
||||
operator to confirm or change the direction, one point at a time, then propose it once. Then the effort
|
||||
graduates through playbook [02](02-graduation.md).
|
||||
|
||||
## Bounds and models
|
||||
|
||||
| Step | Bound | Model |
|
||||
|---|---|---|
|
||||
| reading and the setup questions (1, 2) | 30 minutes | the session's own, larger |
|
||||
| proposing (3, 4) | 15 minutes | larger |
|
||||
| each experiment or reading task (5) | 30 minutes | smaller, one agent at a time unless the operator allows more |
|
||||
| placing it in the domains (6) | 15 minutes | larger |
|
||||
| the direction (7) | 30 minutes | larger |
|
||||
|
||||
A step past its bound is said to the operator at once, with what is being done about it. The lead reports
|
||||
as playbook 07 does: every ten minutes and at once when something is found, stops or needs the operator,
|
||||
with the allowance used.
|
||||
|
||||
## What makes research worth reading
|
||||
|
||||
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"*
|
||||
outranks *"the dependency rule is not followed"*. An effort that measured nothing has not
|
||||
finished.
|
||||
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"* outranks *"the
|
||||
dependency rule is not followed"*. An effort that measured nothing has not finished.
|
||||
|
||||
Research describes real observations but never identifies the mesh it observed. The shape of a
|
||||
finding survives anonymisation intact — *a node publicly named but behind a household NAT*
|
||||
carries the whole lesson without naming anything.
|
||||
Research describes real observations but never identifies the mesh it observed. The shape of a finding
|
||||
survives anonymisation intact — *a node publicly named but behind a household NAT* carries the whole
|
||||
lesson without naming anything.
|
||||
|
||||
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
|
||||
nothing, a retired word, a daemon's line), naming every reason at once and writing nothing. The knowledge
|
||||
base then reviews the proposal itself; on `changes` the effort stays proposed with the findings and its
|
||||
author proposes it again, at most three times before the findings go to the operator with a
|
||||
recommendation. An effort that needs the operator's word is proposed with `needs-operator: true` and waits
|
||||
for the operator's Approve on a channel that proves who answered.
|
||||
|
||||
## Do not
|
||||
|
||||
- Do not put status in prose. It is the effort's `status`, and the prose must not restate it.
|
||||
- Do not give a `became` link while the effort is open — playbook 02 gives it at closure.
|
||||
- Do not write into `03-DESIGN` from an open effort.
|
||||
- Do not write an effort, or a copy of one, into this repository: the knowledge base is the only place one
|
||||
is read, and `records.py` refuses a new path under `01-RESEARCH/`.
|
||||
- Do not write an effort, or a copy of one, into this repository: `records.py` refuses a new path under
|
||||
`01-RESEARCH/`.
|
||||
|
||||
## Closing
|
||||
|
||||
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated`
|
||||
or `abandoned`, always with a `became` link to what it turned into. Nothing is deleted —
|
||||
what was rejected, and why, is the more expensive half to rediscover.
|
||||
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated` or
|
||||
`abandoned`, always with a `became` link to what it turned into. Nothing is deleted — what was rejected,
|
||||
and why, is the more expensive half to rediscover.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The rules on an effort are the knowledge base's: `propose` refuses one that breaks a rule that judges one
|
||||
record (the module's table test holds one case per rule), and the rules that judge the whole record set run
|
||||
after every acceptance and raise a condition when one fails. That this repository holds no effort is
|
||||
`records.py`'s `no-records` check, on every pull request.
|
||||
after every acceptance and raise a condition when one fails. The habits of steps 2, 5, 6 and 7 (options checked
|
||||
against the rules, one version per settled set, the cheapest level, nothing left open) are the lead's, read by
|
||||
the reviewer; of the pre-check, `propose` refuses a link to nothing and a retired word, and the review returns
|
||||
a record named but not linked, an untracked deferral and a figure not as measured. That this repository holds
|
||||
no effort is `records.py`'s `no-records` check, on every pull request. An active effort with no new version
|
||||
for seven days is the smell of research left behind: its lead closes or continues it.
|
||||
|
||||
@@ -13,12 +13,30 @@ clauses in force for a topic or a record).
|
||||
|
||||
## Graduating research
|
||||
|
||||
The effort's `05-direction.md`, confirmed by the operator (playbook [01](01-research.md), step 7), is what
|
||||
graduates. The lead writes the records and the design on the larger model; it is one sitting, bounded at
|
||||
about an hour, and said to the operator if it runs past.
|
||||
|
||||
1. **Check it against GENESIS.** An effort graduates only if its conclusion is traceable to
|
||||
[`mission.md`](../mission.md), [`context.md`](../context.md) and
|
||||
[`effect.md`](../effect.md). If it conflicts, either the effort is wrong or GENESIS is —
|
||||
say which, in writing, before proceeding.
|
||||
2. **Record the decision** (below, *Writing a decision record*).
|
||||
3. **Write the design.** Create the document under `03-DESIGN/01-to-be/` with frontmatter:
|
||||
2. **Record the decisions** (below, *Writing a decision record*): one record per choice that could be
|
||||
reversed on its own, not one record for the whole effort.
|
||||
- **Write them together, not one after another.** Reserve every number first, then propose each record
|
||||
as soon as it is written, without waiting for the review of the one before; the reviews run side by
|
||||
side, and the operator's asks for the records that need them arrive together. A record that
|
||||
`rests-on` or `extends` a sibling is proposed once that sibling is accepted.
|
||||
- **Take the clauses from the confirmed direction** and cite the effort (`cites`, `"research NNN"`) for
|
||||
the evidence instead of retyping its findings: a record's Context holds the few facts its decision
|
||||
rests on, with the effort named for the rest.
|
||||
- **Pre-check each record before proposing it**, as for research (playbook [01](01-research.md), step
|
||||
5): every record, issue or design it names is in `links`; each deferral names its issue; each figure
|
||||
is given as measured; no word on a glossary *Not:* line.
|
||||
3. **Write the design.** **One design per thing** (the operator, 2026-10-16): when a to-be design already
|
||||
covers what the decisions change, they rewrite that design, as *Amending an existing design* below says,
|
||||
and no second design is written beside it that refers to it. Only for something no design covers, create
|
||||
the document under `03-DESIGN/01-to-be/` with frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
@@ -31,7 +49,16 @@ clauses in force for a topic or a record).
|
||||
---
|
||||
```
|
||||
|
||||
4. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
|
||||
The design lists the direction's acceptance tests: what the build proves before it is done.
|
||||
|
||||
The words of the effort's `03-domains.md` go into the glossary in the same change, each in its domain: a
|
||||
word added gets its entry, a word whose meaning changed gets its entry rewritten, and a retired word moves to
|
||||
its replacement's *Not:* line. A word that already means something else in another domain is named as a
|
||||
homonym with the qualified form each domain uses.
|
||||
|
||||
4. **Hand it to the work breakdown.** The design goes on through playbook [04](04-build-handoff.md):
|
||||
milestones, issues, and each milestone built as one feature step (playbook [07](07-feature-branches.md)).
|
||||
5. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
|
||||
`became` link to the decision record (`"NNNN"`) and to the design document (its path). An effort that
|
||||
stops without a decision is proposed with `status: abandoned` and a `became` link to what replaced it.
|
||||
|
||||
@@ -66,7 +93,8 @@ clauses in force for a topic or a record).
|
||||
superseded record cited alone, an unknown topic, a retired word of the glossary, a daemon's line, a
|
||||
deferral no issue tracks. The knowledge base then reviews the proposal itself, through a judgement that
|
||||
sees only the record, the records it links and the rules, and answers `approve` or `changes`. On
|
||||
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`.
|
||||
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`;
|
||||
at most three rounds, then the findings go to the operator with a recommendation.
|
||||
On `approve` it is accepted, or, when it says `needs-operator`, it waits for the operator's Approve on a
|
||||
channel that proves who answered. Nothing proposed is in force. `knowledge-base.get` shows a record's
|
||||
reviews and the operator's ask when it waits for one.
|
||||
@@ -133,4 +161,6 @@ record (its module's table test holds one case per rule), and the four rules tha
|
||||
changes, an accepted record extending a superseded one) run after every acceptance and raise a condition
|
||||
when one fails. In this repository, `records.py` checks that a design's `decisions:` entries and every
|
||||
"ADR NNNN" in the text are well formed, and that no record is written here; `index.py` checks the reading
|
||||
order's six topics, which the knowledge base reads.
|
||||
order's six topics, which the knowledge base reads. The three habits of graduating step 2 (side by side, clauses
|
||||
from the direction, the pre-check) are the lead's, read by the reviewer; `propose` refuses a link to nothing,
|
||||
a deferral no issue tracks and a retired word, and the review returns a record named but not linked.
|
||||
|
||||
@@ -1,35 +1,73 @@
|
||||
# Playbook 04 — Build handoff
|
||||
# Playbook 04 — Work breakdown and handoff
|
||||
|
||||
**Trigger.** A to-be design is settled and work is about to start in a code repository.
|
||||
**Trigger.** A to-be design is settled (playbook [02](02-graduation.md)) and its building is about to start.
|
||||
|
||||
**Who runs it.** Whoever starts the build.
|
||||
**Who runs it.** The lead who will run the feature, on the larger model. It is one sitting, bounded at about
|
||||
an hour, and said to the operator if it runs past.
|
||||
|
||||
## What it produces
|
||||
|
||||
A design becomes **milestones**, and each milestone becomes **issues**:
|
||||
|
||||
- **A milestone is usable on its own.** Once its delivery plan is done, the operator or the mesh can do
|
||||
something new, said in one sentence ("a Claude Code session is mirrored into a thread on the bus").
|
||||
A milestone that changes nothing anyone can use is not a milestone; fold it into the next.
|
||||
- **A milestone is one release** of playbook [07](07-feature-branches.md): prepared, developed whole, reviewed,
|
||||
checked and delivered as one, never cut for its size or for time (the operator, 2026-10-17).
|
||||
- **Milestones are cut by release order.** A milestone ends where one repository's change must be released before
|
||||
another can use it (a new manifest word before the catalogue reads it, an SDK change before its users, until
|
||||
design 47's milestones 3 and 6 lift those orders); the lead checks this when cutting, so no milestone needs a
|
||||
second release.
|
||||
- **An issue is one repository's part of a milestone,** sized so a fixing agent finishes it inside the
|
||||
milestone's development time. Issues that must land in order are linked `depends-on`.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every
|
||||
claim in it traces to a decision record in the knowledge base (`knowledge-base.get` for each entry of
|
||||
its `decisions:`). An open question in the text is a
|
||||
reason to run playbook [01](01-research.md), not to start building around it.
|
||||
2. **Name the owner.** Set `code:` in the design's frontmatter to the repositories from
|
||||
[`repos.md`](../repos.md). If the repository does not exist yet, add it to `repos.md` in
|
||||
the same change.
|
||||
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being
|
||||
replaced is stated there; if it is not, write it before changing it. Building against an
|
||||
undocumented as-is is how a shipped behaviour gets lost.
|
||||
4. **Flip the status.** `status: in-progress`, `updated:` today.
|
||||
5. **Build in the code repository.** HQ is not a code repository and never carries
|
||||
implementation.
|
||||
6. **On completion**, run the "when something ships" section of playbook
|
||||
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every claim in it traces
|
||||
to a decision record in the knowledge base (`knowledge-base.get` for each entry of its `decisions:`). An
|
||||
open question in the text is a reason to go back to playbook [01](01-research.md), not to build around it.
|
||||
2. **Name the owners.** Set `code:` in the design's frontmatter to the repositories from
|
||||
[`repos.md`](../repos.md). A repository that does not exist yet is added to `repos.md` in the same change.
|
||||
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being replaced is stated
|
||||
there; if it is not, write it before changing it. Building against an undocumented as-is is how a shipped
|
||||
behaviour gets lost.
|
||||
4. **Break it into milestones.** Write them into the design, in a section *Milestones*: for each, its number,
|
||||
its one-sentence outcome, the acceptance tests of the design it proves, the repositories it touches, and
|
||||
what it depends on. Order them so each is usable as soon as it is delivered, the smallest useful one
|
||||
first. A milestone is named by what it lets someone do; how it ships is shaped by the plan its changes would
|
||||
have, asked through `mesh-delivery.what-if`: the modules it moves and their fan-out (ADR 0401 §6).
|
||||
5. **Ask the operator to confirm the milestones**, in a short table: number, outcome, the acceptance tests it
|
||||
proves, repositories, the parts each repository builds. Change what they change.
|
||||
6. **Open the issues.** For each confirmed milestone, `issue-tracker.open` one issue per repository part, its
|
||||
title the symptom or capability in plain words, its report naming the design, the milestone and the
|
||||
decisions it builds with; link the order with `depends-on`. Write the issue numbers into the milestone's
|
||||
entry in the design.
|
||||
7. **Flip the status.** `status: in-progress`, `updated:` today; the design change goes through playbook
|
||||
[08](08-small-fixes.md) (one hq pull request).
|
||||
8. **Hand each milestone to playbook 07, one at a time.** The milestone is one release of playbook 07; its issues
|
||||
are the release's issues; the next milestone starts when the previous one's delivery plan is done. A design waits
|
||||
while another design's milestones are being built: one design is built at a time (playbook
|
||||
[07](07-feature-branches.md)), so this step starts only when the design before it is delivered.
|
||||
9. **On completion of the last milestone**, run the "when something ships" section of playbook
|
||||
[02](02-graduation.md).
|
||||
|
||||
## Rules
|
||||
|
||||
- Every merge is a human checkpoint, without exception.
|
||||
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that
|
||||
leaves it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue
|
||||
names who owns what is not. the knowledge base's rule `deferral` checks the decision record when it is proposed
|
||||
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that leaves
|
||||
it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue names who
|
||||
owns what is not. The knowledge base's rule `deferral` checks the decision record when it is proposed
|
||||
([issue 354](../../04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/00-report.md)).
|
||||
- Never open a pull request unprompted. A permissions list saying it is allowed is not a
|
||||
request.
|
||||
- Work in an isolated worktree, never a shared checkout. A failed `cd` in a shared checkout
|
||||
commits to the wrong branch, and the error scrolls past.
|
||||
- HQ is not a code repository and never carries implementation.
|
||||
- How the building is done — worktrees, agents, review, checks, the Approve and the delivery — is playbook
|
||||
07's, not this one's.
|
||||
|
||||
## How it is checked
|
||||
|
||||
Read by the lead and the reviewer today; a check in hq's merge-check is tracked by issue 845.
|
||||
|
||||
- A design in `status: in-progress` has a *Milestones* section, each milestone with its outcome and its
|
||||
issues; an in-progress design without one is the smell of building without a breakdown.
|
||||
- Every issue a milestone names exists in the tracker (`issue-tracker.get`), and an issue in `fixing` names
|
||||
its milestone in its report.
|
||||
- A milestone whose issues are all `resolved` has had its delivery plan done; its outcome sentence and
|
||||
acceptance tests are proven on the mesh before the next milestone starts.
|
||||
|
||||
@@ -1,69 +1,401 @@
|
||||
# Playbook 07 — Feature branches across repos
|
||||
# Playbook 07 — Features
|
||||
|
||||
**Trigger.** Work that changes code — in one code repo or in several at once (`mesh-sdk`,
|
||||
`mesh-controller`, `mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a decision rides along).
|
||||
**Trigger.** A feature: work that changes code in one repository or several (`mesh-sdk`, `mesh-controller`,
|
||||
`mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a design rides along), too large to be one small fix.
|
||||
A small fix (one repository, one issue, about an hour of work) follows
|
||||
[playbook 08](08-small-fixes.md) instead.
|
||||
|
||||
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly —
|
||||
it is the guard against the failure it was written for.
|
||||
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly.
|
||||
|
||||
## The failure it prevents
|
||||
## The failures it prevents
|
||||
|
||||
A feature was worked as a branch-and-MR per *unit of thought* — one per decision, one per
|
||||
stacked increment — and each MR was treated as finished when it was *opened*, not when it was
|
||||
*merged*. Across repos the same feature took a different branch name in each. The MRs piled up
|
||||
unmerged: one session left **sixteen** stacked intermediate MRs that had to be consolidated and
|
||||
closed by hand. An MR is a review checkpoint, not a scratchpad.
|
||||
- **Pull requests as scratchpads.** A feature worked as one pull request per unit of thought left sixteen
|
||||
stacked pull requests unmerged, consolidated by hand afterwards. A pull request is the end of a feature,
|
||||
not a step in it.
|
||||
- **Waiting on checks mid-feature.** A feature worked as one pull request per issue into protected
|
||||
branches spent most of a day waiting for merge checks while the code was ready.
|
||||
- **Context bought again and again.** A fresh agent per small issue spends most of its run reading the
|
||||
playbook, the issue and the code before it writes a line. On 2026-10-14 an issue sized for five to ten
|
||||
minutes took forty-one.
|
||||
- **Finished work that never ships.** A feature that kept taking new issues held thirteen landed issues
|
||||
undelivered for a day: an issue done at 21:09 was still not on the machines at 15:00 the next day.
|
||||
- **Work that waits and nobody notices.** Most lost time was not work at all, but a check, a delivery plan
|
||||
or an agent waiting on something that had stopped, unseen.
|
||||
- **Review that keeps hardening.** One fix went through seven review rounds in five and a half hours, each
|
||||
asking for more hardening than its issue asked (issue 704, 2026-10-14). A round is for what makes the
|
||||
change wrong; everything else is an issue of its own.
|
||||
- **The first check after the whole branch is written.** A feature's catalogue branch, pushed once after
|
||||
hours of local work, met its first check on 2026-10-16 and failed on 29 problems, TypeScript errors among
|
||||
them, that a unit suite run continuously while working finds at once.
|
||||
- **A green pull request nobody marks.** An approved, green pull request that was not marked ready waited
|
||||
12 h 17 min for someone to merge it (mesh-catalog #300, 2026-10-14). There is no ready mark since ADR 0421:
|
||||
the operator's Approve is asked as soon as the checks pass.
|
||||
- **A module proven only after it reached the machines.** The mail modules were delivered with unit tests
|
||||
only, then failed live three times (issues 859, 865, 868) and took five more hours of fixes.
|
||||
|
||||
## The rule
|
||||
## The shape
|
||||
|
||||
One feature is **one branch name**, **one worktree per repo**, **one MR per repo**, opened
|
||||
**once, at the end**.
|
||||
One feature is **one branch name**, **one worktree per repository**, **one pull request per repository**,
|
||||
opened **once**, after the review. It runs in five steps, and its time is bounded. A milestone has no phases
|
||||
(the operator, 2026-10-17): these are its steps, and a report names the step it is in:
|
||||
|
||||
1. **Name the feature once.** `feat/<slug>`. The *same* branch name in every repo the feature
|
||||
touches — never a different name per repo, never a fresh branch per increment within the
|
||||
feature.
|
||||
2. **Isolate each repo.** One git worktree per touched repo under `.work/<slug>/<repo>`, branched
|
||||
off `main`:
|
||||
```
|
||||
git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main
|
||||
```
|
||||
Parallel features never collide, and no shared checkout is edited.
|
||||
| Step | What happens | Bound |
|
||||
|---|---|---|
|
||||
| 0. Prepare | the lead settles the open questions and writes one context file per repository | 15 minutes |
|
||||
| 1. Develop | one fixing agent per repository builds every issue locally, the whole unit suite before every commit | the milestone's issues, 30 minutes each |
|
||||
| 2. Review | one reviewer reads all of it; reports go both ways; at most 3 rounds | 15 minutes |
|
||||
| 3. Check | push; the forge's merge checks run the integration tests; no further review | one check run |
|
||||
| 4. Deliver | merge; the delivery plan builds, publishes and sends the whole feature | its delivery plan |
|
||||
|
||||
**Add the untouched siblings the lab reads.** The lab beds find the other repositories by
|
||||
sibling path from the lab checkout (`../mesh-tools/module.json`, `../mesh-sdk`, …), the way
|
||||
the main layout has them. A `.work/<slug>/` directory holding only the touched repos fails a
|
||||
bed at once with `no manifest for mesh-tools at …/.work/<slug>/mesh-tools/module.json`, after
|
||||
genesis has already passed. Give the directory those repos as **detached worktrees on `main`**
|
||||
— never symlinks:
|
||||
```
|
||||
git worktree add --detach .work/<slug>/mesh-tools main
|
||||
```
|
||||
3. **Commit as you go — locally.** Increments land on the one branch. Nothing is pushed and no
|
||||
MR is opened mid-feature.
|
||||
4. **Finish, then publish.** When the whole feature is done — every repo, tests green — push
|
||||
every branch and open **one MR per touched repo**, together.
|
||||
5. **Merge promptly, once approved.** Every merge into `main` is notified and approved
|
||||
(ADR 0023); once it is, merge —
|
||||
do not leave it sitting. The branch is deleted on merge.
|
||||
6. **Leave nothing behind.** After the MRs merge, no `feat/<slug>` branch and no `.work/<slug>`
|
||||
worktree survive.
|
||||
**One milestone is one release** (the operator, 2026-10-17). A milestone is developed whole, reviewed once,
|
||||
pushed once and delivered once. It is never cut into steps for its size or for time: every cut adds a review, a
|
||||
merge check, an Approve and a delivery plan of waiting, so the fewest cuts is the fastest. Milestones are cut by
|
||||
release order when the design is broken up (playbook [04](04-build-handoff.md)); a release-order dependency found
|
||||
inside a milestone is a fault of that cut, said to the operator, never a silent extra step.
|
||||
|
||||
## What this is not
|
||||
**One design is built at a time** (the operator, 2026-10-17). While the milestones of one design are being
|
||||
built, no milestone of another design starts: its issues stay `open`, and nobody moves them to `diagnosing` or
|
||||
`fixing`. The next design starts when the last milestone of the one being built has its delivery plan done. A
|
||||
lead who finds another design's milestone already in `fixing` asks its owner before starting its own. A small
|
||||
fix ([playbook 08](08-small-fixes.md)) and an issue that is in no design's milestones are not held by this rule.
|
||||
|
||||
- **Not a licence to batch unbounded work.** A feature is a *bounded* unit; if it sprawls for
|
||||
days, end-of-feature bloat merely replaces per-increment bloat. Split it into features, each
|
||||
its own branch and MR.
|
||||
- **Not a second trunk.** Every repo branches off `main`. There is no longer an `initialization`
|
||||
trunk.
|
||||
**Scope is fixed at the prepare step.** An issue found during a feature goes to the next milestone or the next feature,
|
||||
never into the running one, unless it blocks the running one.
|
||||
|
||||
## Step 0 — Prepare (the lead, larger model)
|
||||
|
||||
1. **Name the feature once:** `feat/<slug>`, the same branch name in every repository it touches (but see the
|
||||
exception for a new manifest word in phase 3: a release that must merge first has a name of its own). The
|
||||
branch is never protected. **One milestone is one branch per repository** (the operator, 2026-10-17), with one
|
||||
folder `.work/<slug>/`; never a second name for part of it (`-2`, `-step2`).
|
||||
2. **Make the worktrees:** one per touched repository under `.work/<slug>/<repo>`, branched off `main`
|
||||
(`git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main`). The lab beds find the other
|
||||
repositories by sibling path, so add the untouched siblings the lab reads as detached worktrees on
|
||||
`main` (`git worktree add --detach .work/<slug>/mesh-tools main`), never symlinks. **A repository's own
|
||||
clone stays on `main`** (the operator, 2026-10-17): no work is done in it and no branch is checked out
|
||||
there, so a session that opens in it reads what holds now; fetch and fast-forward it before reading.
|
||||
3. **Settle every open question** in the feature's issues before any agent starts: on the issue, through
|
||||
the knowledge base, or by asking the operator. A fixing agent that meets an undecided design question
|
||||
stops, and its run is spent again.
|
||||
4. **Write one context file per repository**, `.work/<slug>/<repo>.context.md`. It is what a fixing agent
|
||||
reads instead of rediscovering the code, and it is reused by every agent and every round:
|
||||
- the issues it builds, in order, each with: the files and functions it changes, the decisions it builds
|
||||
with (quoted, with where each was decided), and its acceptance criteria (item 5), each with the test that
|
||||
proves it;
|
||||
- the repository's map for this feature: the packages and files involved, with one line each on what they
|
||||
hold, and the patterns to copy (file:line);
|
||||
- what not to touch, and what lands beside it in other repositories;
|
||||
- the local facts: the worktree, the branch, the check commands, a temporary directory for tests outside
|
||||
the home (`TMPDIR`), and the unit tests to run (never a database server: tests that need one are
|
||||
integration tests, which the forge's checks run).
|
||||
When the reading is large, a read-only agent on the smaller model answers the lead's exact questions with
|
||||
file:line, and the lead writes the file from its answers.
|
||||
5. **Write the findings into the issues** (the operator, 2026-10-17: the better the content of the issues, the
|
||||
better work the agents do, and the less review is needed). Before any agent starts, each issue carries, with
|
||||
`issue-tracker.note`: every file and test that touches the change, found by search and not from memory
|
||||
(file:line); how the change is checked and delivered end to end (which test proves its done-when, which
|
||||
modules its delivery plan moves, to which machines); and for a group, its merge order. The context file
|
||||
points to these notes; an issue reread by another session or a later agent then holds what was found.
|
||||
When the reading is large, it starts with a read-only survey agent on the smaller model.
|
||||
**Each issue carries its acceptance criteria, written by the lead** (the operator, 2026-10-18: the larger
|
||||
model writes them, and they guide every test). Each criterion is one sentence, given, when, then, about what
|
||||
the module or the operator observes, never about how the code does it, and names the level of the test that
|
||||
proves it: unit, integration (a bus, a store or a process started) or end to end (none exist yet, so such a
|
||||
criterion is proven live after the delivery and said so). Together they cover the issue's done-when and the
|
||||
cases its life passes through: on 2026-10-18 mesh-delivery's group life (a group of three, a member's own
|
||||
check, a close in a group, a second pull request at the same commit, a stop and a resend) had no criterion,
|
||||
and each case was found by a failure on the live mesh. Each criterion is numbered in its note and cited as
|
||||
`<issue>/AC<n>` (`1118/AC2`). They are an `issue-tracker.note` headed `Acceptance criteria`, a revision a new note that replaces the older one, until the tracker holds them as a
|
||||
field (issue 1118). A criterion the reviewer adds in step 2 is written onto the issue the same way.
|
||||
**The survey sizes each issue, and an issue sized over an hour is split before any agent starts** (the operator,
|
||||
2026-10-17), into issues of about 30 minutes each, one change each, in the order they land: the bound of a
|
||||
fixing agent's issue is then a real bound. On 2026-10-17 one issue held four changes sized at four to six
|
||||
hours; it ran past its 30-minute bound for over an hour, and the reports could only say "still on part D".
|
||||
6. **Move the issues to `fixing`**, `by` naming the feature's agent for that repository.
|
||||
|
||||
## Step 1 — Develop (one fixing agent per repository, smaller model)
|
||||
|
||||
- **One agent per repository, kept for the whole feature.** It reads its context file once and builds the
|
||||
issues one after another, committing each locally with a message naming the issue ("Keep the assignments
|
||||
on the bus (issue 652)"). It is not replaced per issue: the lead continues the same agent (by its id) with
|
||||
the next instruction, so its context is paid for once.
|
||||
- **Unit tests only, each well under a second** (how-we-build §5, ADR 0391). **The whole unit suite runs before
|
||||
every commit** (the operator, 2026-10-17), not only the tests of what changed: it takes under a second, and a
|
||||
change breaks tests in packages it never touched. Each issue's first test fails before its change and passes
|
||||
after; `gofmt`, `go vet` and the whole unit suite stay green. **The tests are written from the issue's
|
||||
acceptance criteria**: one test or more per criterion at the level it names, each test's comment citing
|
||||
the criterion it proves by its label (`// 1118/AC2`), and no criterion left without its test; an integration criterion is a test
|
||||
that compiles while you work and runs in the one merge check before the push.
|
||||
- **Nothing is pushed** and no pull request is opened in this step. Never land on a red check: run the
|
||||
checks, read the result, then land in a separate step.
|
||||
- **The fixing agent's brief** (the lead gives it with the context file): name your issue in every step and
|
||||
message ("issue 805: landing in mesh-catalog"); reach the mesh only through the mesh MCP server, never ssh;
|
||||
never push, open, merge or close a pull request; never write to a store's database by hand, a schema change
|
||||
is a numbered migration and the consolidated schema; Go only unless the context file says otherwise; never
|
||||
move another issue; **run the integration tests once before every push, not while developing** (the
|
||||
operator, 2026-10-18): compile them with `go vet -tags integration ./...` while you work, and before the push
|
||||
run the repository's whole merge check with `mesh-controller.try` on the pushed branch (the store and bus a
|
||||
pull request's check gets) or `merge-check.sh` against a throwaway store, and push only on its pass. On
|
||||
2026-10-18 three pushes in a row failed their first merge check on faults an integration run finds in a
|
||||
minute (mesh-catalog #361, mesh-controller #380's stale keys, #381's 400 tests without the layout marker),
|
||||
each costing a check, a fix and another check. Report back in under 150 words: commits, tests added and that they failed first, the
|
||||
check results, anything you stopped on.
|
||||
- **An agent that finds the code differs from its context file stops and reports** (file:line); the lead
|
||||
settles it, updates the context file, and continues the agent.
|
||||
- **Two repositories that depend on each other** keep their order in the context files: the reader before the
|
||||
writer, the controller's new manifest word before the catalogue manifest that uses it.
|
||||
- **Several agents in one repository** (only for a large repository with independent areas): each works on a
|
||||
local branch of its own from the feature branch's head and lands by fast-forward under a lock
|
||||
(`flock <feature worktree>.lock git -C <feature worktree> merge --ff-only <its branch>`), after its unit
|
||||
tests pass; if another landed first, it merges again and repeats. In a shared clone, fetch the feature
|
||||
branch by name before branching from it. Never resolve a conflict with a blind `git merge-file --union`.
|
||||
|
||||
## Step 2 — Review (one reviewer, larger model; at most 3 rounds, about 15 minutes)
|
||||
|
||||
1. **Bring every feature branch up to the current trunk** (fetch by name, then merge), so the reviewed tree is
|
||||
the tree that is pushed.
|
||||
2. **One reviewer reads every repository's local commits** together, against the context files and the
|
||||
decisions. **It is one agent, kept for the whole release** like the fixing agents: the lead continues it by
|
||||
its id for every later round, for the extra cycle after a failed check (step 3) and for its review record, so it
|
||||
reads the decisions, the issues, the context files and the commits once (issue 1042). A fresh reviewer per
|
||||
round or per mark pays for all of that again. Its report is per repository: each finding as file:line, the failure it causes, and the fix.
|
||||
It says which round it is in ("round 1 of at most 3").
|
||||
3. **The report goes to the same fixing agents** that wrote the code, continued by their ids, so they fix
|
||||
with the context they already hold. Each answers the reviewer: what it changed, per finding, or why not.
|
||||
4. **Every new integration test is read against the code it drives before the first push.** It has never run: fixing agents only compile it, and its first run is the merge check, so each fault it hides costs a whole check. The reviewer traces each step of each new integration test through the code (fixtures against the manifest rules, what the store keeps and reads back, who holds a seat), and says which steps it could not trace. On 2026-10-17 a new test failed on its first two runs, on a fixture and then on a fault of the change; reading it against the code after the first failure found another fault, which would have failed a run too. The reviewer also checks that each new integration test enters through the same path the live trigger takes, not through a state it writes by hand: on 2026-10-17 milestone 4's test saved its delivery plan directly, so it skipped the cut from a batch that every live plan takes, and it passed while the backup step never ran (issue 1086).
|
||||
5. **The reviewer reads the answers and the new commits**, and approves or reports again (step 7 says what a round
|
||||
may ask). After round 3 the lead stops and brings the operator what is left, with a recommendation.
|
||||
6. **The reviewer reads every acceptance criterion against its test**: each criterion has a test at its level
|
||||
that would fail without the change, and no test proves less than its criterion says. A criterion missing
|
||||
for a case the issue's done-when needs is written onto the issue (step 0, item 5) and is a finding; a test
|
||||
missing for a criterion is a finding.
|
||||
7. **A finding outside the issues' done-when and their acceptance criteria is never a change asked, in any round.** More hardening, a
|
||||
further case, a nicer form: the reviewer opens it with `issue-tracker.open` and names the number in its
|
||||
report, and the round does not wait for it. What a round asks is only what makes the change wrong against
|
||||
its issues, their acceptance criteria and its decisions.
|
||||
|
||||
## Step 3 — Check (no review)
|
||||
|
||||
1. **Push every feature branch and open one pull request per repository, together.**
|
||||
The integration tests ran once before the push (step 1); the merge check runs them again as the gate. Pull requests in two or
|
||||
more repositories that share the branch name are one delivery group (ADR 0239 §2).
|
||||
Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the
|
||||
reviewer first confirms that commit's tree equals the reviewed one. **Each pull request's description carries
|
||||
the review record** (ADR 0421 §5), one line the lead writes from the reviewer's approval:
|
||||
`Reviewed-by: <who@node> at <full sha>` (the issue 1022 note gives its form). mesh-delivery reads it at every new
|
||||
head and puts it in the Approve's ask; a missing line, or one naming another commit, is said first in the ask.
|
||||
It is never a gate.
|
||||
2. **The forge's merge checks run the integration tests once** (`mesh/merge-gate`, `mesh/repo-check`). No
|
||||
second review: step 2 reviewed this tree.
|
||||
3. **A check that fails** goes back to the same reviewer, continued by its id, with the failure, for **one more cycle** of step 2 with
|
||||
the same fixing agents; then push once again. **The failure is read from the check's whole log**, never
|
||||
reasoned from its summary: `mesh-delivery.show` names the check's build id, and `mesh-controller.builds` with
|
||||
`log` set to that id prints everything the builder said. A check's summary picks only some failing lines;
|
||||
on 2026-10-17 it showed a store test's `FAIL` without its cause, and a fix was first guessed from the code.
|
||||
**A change to `merge-check.sh` is judged by main's version of it**, so a test the change adds must pass under
|
||||
either version: one that runs the new script's behaviour runs a file of its own, never `merge-check.sh` itself.
|
||||
**So a change that needs the new script to judge it ships the script first, as a small fix of its own**, and the
|
||||
rest waits for its delivery: on 2026-10-17 milestone 3 of design 47 moved its integration tests behind a build
|
||||
tag in the same pull request as the script that runs that tag, and main's script would have skipped every
|
||||
tagged test, its acceptance tests among them. The prepare step looks for this: a change to `merge-check.sh` or to
|
||||
what it selects (build tags, the store-tests list) is cut ahead of the milestone.
|
||||
4. **The milestone's acceptance is proven by its check.** A milestone that adds or changes a module
|
||||
carries an integration test that starts the module whole, its process against the real bus and store, and
|
||||
shows the milestone's acceptance sentence; the forge's checks run it (ADR 0391 §1, §6). Unit tests alone are not
|
||||
that proof; the mesh has no end-to-end tests yet, and the lab is not used (the operator, 2026-10-16).
|
||||
5. **The operator's Approve is asked as soon as the checks pass** (ADR 0421). Nobody marks, labels, comments or
|
||||
rechecks: mesh-delivery asks the Approve on Telegram, the channel that proves who answered today (ADR 0398 §3;
|
||||
an Approve at the terminal is issue 953), once both checks passed, for a delivery group once for all its
|
||||
members, naming the review record. On the Approve it merges the approved heads, a group's in one pass
|
||||
(ADR 0418). A new head ends the Approve and is asked again once its checks pass. A watcher follows the checks,
|
||||
the Approve and the merge, and wakes the lead. Beyond that
|
||||
Approve, the operator's word is waited for only when the milestone changes what the operator sees, removes
|
||||
data, or needs a step only the operator's terminal can take; the lead says which before the milestone starts.
|
||||
|
||||
**One exception: a catalogue change that uses a new manifest word.** The catalogue's merge check reads
|
||||
manifests with the controller the mesh runs, so the controller's pull request is merged and delivered first;
|
||||
then `mesh-delivery.recheck` the catalogue's and merge it on the new verdict. Such work is two releases, the controller's first, and playbook 04 cuts its milestones that way.
|
||||
**A new key the node-engine checks is such a word too** (issue 1051): the controller judges every machine with the
|
||||
node-engine's own validator, vendored at the commit `go.mod`'s `replace` pins, so a key added to the node-engine is
|
||||
refused by the gate until the controller's pin moves to the delivered node-engine (`go mod vendor`); the feature is
|
||||
then three releases, the node-engine's, the controller's, the catalogue's. **Every release that must merge before
|
||||
the catalogue's is on a branch name of its own** (`feat/<slug>-node-engine`, `feat/<slug>-controller`): on the
|
||||
feature's name it joins the catalogue's delivery group, and a group merges only whole, so neither merges.
|
||||
The exception ends when milestone 3 of design 47 is delivered (issue 977, ADR 0402, ADR 0403): the group's composed
|
||||
check, judged by the group's controller, is then each member's gate, and the feature is one group.
|
||||
|
||||
## Step 4 — Deliver
|
||||
|
||||
The merge starts the delivery plan: build, publish, send. A merge into a branch other than `main` is not a
|
||||
delivery: no machine receives it (issue 889 while it still starts one). The lead follows it to done
|
||||
(`mesh-controller.plans` with `wait` and `until: done`), then resolves the feature's issues with `fixed-by`
|
||||
naming its pull requests, and removes what is left: no `feat/<slug>` branch and no `.work/<slug>` worktree
|
||||
survive.
|
||||
|
||||
## Models
|
||||
|
||||
| Who | Model | Why |
|
||||
|---|---|---|
|
||||
| The lead (step 0 decisions, the context files, following and reporting) | larger | judgement |
|
||||
| Read-only questions about the code for the context files | smaller | reading, answered with file:line |
|
||||
| Fixing agents (step 1, and the fixing in steps 2 and 3) | smaller | building to a written context |
|
||||
| The reviewer (step 2, the extra cycle in step 3 and the review record; one agent, continued by its id) | larger | judgement, once per round |
|
||||
| Decision records | larger | judgement |
|
||||
|
||||
Every agent the mesh runs draws on one weekly allowance shared by every session, and the larger model
|
||||
spends it many times faster. Up to five agents on the smaller model at once, at most two subagents on the
|
||||
larger one (the lead not counted).
|
||||
|
||||
## Following and reporting
|
||||
|
||||
Most lost time is waiting nobody saw. The lead therefore keeps a **wait list**: everything the feature waits
|
||||
on, with when it started and its bound.
|
||||
|
||||
| Waiting on | Bound | Past the bound |
|
||||
|---|---|---|
|
||||
| a fixing agent's issue | 30 minutes | ask it where it stands; split or take over |
|
||||
| the review | 15 minutes a round | ask the reviewer; stop at round 3 |
|
||||
| a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log |
|
||||
| the Approve's ask | 5 minutes after both checks pass | read `mesh-delivery.show`: an ask not raised is a fault, said to the operator |
|
||||
| a merge after the Approve | 5 minutes | read `mesh-delivery.show` for the delivery |
|
||||
| a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
|
||||
|
||||
**Every wait on the list has a watcher** (the operator, 2026-10-17). A session runs only when a message reaches
|
||||
it, so a check, a merge, a delivery plan, a record's review or the operator's Approve on a channel that lands
|
||||
while the lead's turn has ended wakes nobody: on 2026-10-17 an approved group sat half merged for 14 minutes
|
||||
unseen (issue 1032). For each wait the lead starts a watcher, an agent on the smaller model that changes nothing,
|
||||
never the smallest (the operator, 2026-10-17).
|
||||
It loops `mesh_wait` on the event families of what it watches (`mesh-delivery.transition` and
|
||||
`mesh-delivery.group` for a delivery, for example) with a timeout under 60 seconds, since the call itself ends near
|
||||
a minute. After every answer or timeout it reads the state from the verb that owns it (`mesh-delivery.show`,
|
||||
`mesh-controller.plans`, `knowledge-base.get`), because an event only wakes it (ADR 0367). It finishes, and so wakes
|
||||
the lead, when the state changes, when the wait passes its bound, **or when nothing changed for two minutes**
|
||||
(the operator, 2026-10-18): a wait that stopped moving is a stall, never a wait. On 2026-10-18 a delivery plan stopped
|
||||
at 12:46 because the bus refused the new controller's writes, its watcher waited for "done or failed", and the stall
|
||||
was found ten minutes later only because the operator asked what was happening (issue 1121). The lead never sleeps or polls by hand to wait.
|
||||
|
||||
**Every ten minutes (the operator, 2026-10-17), every two minutes while a delivery plan runs, and at once when something lands, stops or
|
||||
needs the operator,** the lead reports, in one of the two layouts below (the operator, 2026-10-16: an overview
|
||||
of the issues, often). Times are local and absolute, from the clock (`date`), never guessed; every number has its
|
||||
unit and its bound or baseline; an issue is named by its number and a few words, never by its number alone.
|
||||
|
||||
**The full layout** (in the session) is the whole design as a tree (the operator, 2026-10-17: always the full
|
||||
overview): every milestone of the design with its state, and under the running milestone its repositories and
|
||||
their issues, each issue under the one it waits for, marked ✓ landed or delivered, ▶ fixing, ⏸ waiting, ✗ stopped:
|
||||
|
||||
```
|
||||
Feature <slug> · milestone <n> of <m> · step <name>, <minutes used> of <bound> min
|
||||
Issues: <landed> of <total> landed · <fixing> fixing · <waiting> waiting
|
||||
|
||||
Design <nn> · <its title>
|
||||
├── ✓ milestone 1 <its outcome, a few words> delivered 12:58
|
||||
├── ▶ milestone 2 <its outcome, a few words>
|
||||
│ ├── mesh-controller (agent: fixing, 2 of 3 landed)
|
||||
│ │ ├── ✓ 652 assignments on the bus landed 14:02
|
||||
│ │ ├── ▶ 736 placement words leave the layers fixing since 13:51, 23 of 30 min
|
||||
│ │ │ └── ⏸ 751 config keys named (mesh-catalog) waits for 736
|
||||
│ │ └── ✓ 654 a file names its template landed 13:20
|
||||
│ └── mesh-catalog (agent: waiting)
|
||||
│ └── ⏸ 751 (above, under 736)
|
||||
└── ⏸ milestone 3 <its outcome, a few words> <its issues>
|
||||
|
||||
Opened along the way: <each issue opened during the feature, its number and a few words, and where it goes> — or "none"
|
||||
Past its bound: <the wait, how long, what is being done> — or "none"
|
||||
Needs you: <one line per question, with my recommendation> — or "nothing"
|
||||
Allowance: week <n> % (<+d> since the last report), session <n> %
|
||||
Next: <the next actions up to the delivery, in order>
|
||||
```
|
||||
|
||||
**The short layout** (on a channel such as Telegram, or when nothing changed but time):
|
||||
|
||||
```
|
||||
<slug> milestone <n>/<m> · <step> <used>/<bound> min · <landed>/<total> landed · past bound: <none | what> · needs you: <nothing | what> · week <n> %
|
||||
```
|
||||
|
||||
A report says what changed since the last one first. In the session every report is the full layout, the whole
|
||||
tree of the milestone's issues included, also when only time passed: the operator reads the state from it without
|
||||
asking. The short layout is for a channel.
|
||||
|
||||
## The context of the lead's own session
|
||||
|
||||
At the end of each step, and whenever the session's context grows large or spread over unrelated work, the
|
||||
lead prepares a `/clear` as the mesh's instruction "prepare before clear" says: a save point with the
|
||||
feature's step, its branches and heads, the context files' paths, the agents' ids, the wait list and the
|
||||
next action; then the prompt the operator pastes. The context files make the feature resumable after a clear.
|
||||
|
||||
## The design's state is written back
|
||||
|
||||
Other sessions know what is built only from the design's frontmatter and the issue tracker; what the lead
|
||||
knows and does not write back, nobody else knows. A design merged in one delivery with three of its tasks
|
||||
left, its frontmatter still saying `designed`, read to another session as not built, and a second design
|
||||
was started beside it (issue 1009, 2026-10-17). So the lead writes the design's state back at each of these
|
||||
moments, the same hour, an hq change going through [playbook 08](08-small-fixes.md):
|
||||
|
||||
- **The first milestone starts:** the design's `status: in-progress` ([playbook 04](04-build-handoff.md) step 7),
|
||||
if it is not already.
|
||||
- **A milestone's delivery plan is done:** its issues are resolved (step 4), with `fixed-by` naming its pull
|
||||
requests. The design's milestone entry is not changed: it names the milestone's issues, and whether they are
|
||||
delivered, when and by what is read from the issue tracker when it is asked, never copied into the design
|
||||
(issue 1038; AGENTS.md: cross-cutting views are generated when they are read).
|
||||
- **The building pauses with work left** (another design takes its turn, a hand-over, the session's end): the
|
||||
design stays `in-progress`, and each issue left gets a note with `issue-tracker.note` saying what it waits
|
||||
for and what comes after it ("built after design N is delivered, before design M"). The first issue of
|
||||
a design that waits its turn gets the same note.
|
||||
- **The last milestone is delivered:** [playbook 02](02-graduation.md), "When something ships": the as-is is
|
||||
written and the design says `implemented`.
|
||||
|
||||
## Issues that stop
|
||||
|
||||
An issue stays in `fixing` only while an agent works on it. A milestone that stops, for a wait past its bound, a
|
||||
hand-over, or the session's end, moves each of its issues not delivered back to `open` with
|
||||
`issue-tracker.move` and a note saying how far it got (its branch, its last commit, what is left). A list of
|
||||
issues in `fixing` that nobody works on reads to the operator as stuck work.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The end state is visible, and its absence is the smell:
|
||||
- A feature's issues show their commits on the feature branch (`git log --grep 'issue <n>'`), and no pull
|
||||
request into a feature branch exists. A pull request per issue, or an agent waiting on a merge check in
|
||||
step 1, is the failure this playbook prevents.
|
||||
- A context file exists per touched repository under `.work/<slug>/` while the feature runs.
|
||||
- No issue of a running milestone is sized over an hour in its prepare note; an issue whose fixing agent passes
|
||||
twice its bound was not split.
|
||||
- Every issue of a running milestone has a note from the prepare step naming its files and tests (file:line), how
|
||||
it is checked and delivered, and for a group its merge order (`issue-tracker.get`); an issue moved to `fixing`
|
||||
without one is a prepare step skipped.
|
||||
- Every issue of a running milestone has a note headed `Acceptance criteria` before it moves to `fixing`
|
||||
(`issue-tracker.get`), each criterion naming its test level, and every criterion's label is cited by a test in the
|
||||
pushed tree (`git grep -n '<issue>/AC<n>'`); an issue without them is a prepare step skipped, and a
|
||||
criterion no test names is a review that missed it (until issue 1118 makes the tracker refuse it).
|
||||
- Every fixing agent's report gives the whole unit suite's result and time; a report naming only some packages'
|
||||
tests is the whole-suite rule not kept.
|
||||
- The lead's save point names each watcher with its model; a watcher on the smallest model is the failure of
|
||||
"Following and reporting".
|
||||
- A report or a save point that names a "phase" of a milestone is the old wording: a milestone has steps.
|
||||
- Every repository's own clone says `main` (`git -C <clone> branch --show-current`); another branch there is
|
||||
work done outside a worktree (issue 1010).
|
||||
- After a feature's delivery, `git branch -r | grep feat/<slug>` and `git worktree list` return nothing for it.
|
||||
- A design whose milestone issues are partly resolved says `in-progress`, and one whose issues are all
|
||||
resolved and delivered says `implemented`: read its Milestones section against `issue-tracker.get` for each
|
||||
issue. A design that says `designed` with a resolved milestone issue is a state not written back.
|
||||
- A design's Milestones section names each milestone's issues and nothing of their state: a "delivered", a date
|
||||
or a pull request in an entry is a copy of the tracker's state (issue 1038), read with
|
||||
`grep -n 'delivered' <design>` over its Milestones section.
|
||||
- Every open issue of a design that says `in-progress` and has no agent on it shows, in `issue-tracker.get`, a
|
||||
note saying what it waits for; one without such a note is a pause not written back.
|
||||
- The milestone issues in `fixing` (`issue-tracker.list`) belong to one design at most: milestone issues
|
||||
of two designs in `fixing` at once is the failure of "one design is built at a time".
|
||||
- More than one open pull request in a repository that share no feature name, or a milestone whose issues
|
||||
stay in `fixing` for more than a working day, is the smell: stop and bring it to the operator.
|
||||
- A review report that asks a change outside the issues' done-when and acceptance criteria, or a fourth round, is the failure of the
|
||||
review rules; `issue-tracker.list` with `flag: abandoned` lists issues left in `fixing` with nobody on them.
|
||||
- A pull request green for more than 5 minutes with no Approve asked is a fault of mesh-delivery
|
||||
(`mesh-delivery.show`), said to the operator.
|
||||
- Every review record of a release names one reviewer (`mesh-delivery.show`, the review record), and the lead's
|
||||
save point names one reviewer id per release; a second reviewer within one release is a reviewer not kept
|
||||
(issue 1042). A pull request whose ask says it has no review record at its head is a record not written.
|
||||
- A milestone that adds or changes a module whose pull request has no integration test that starts the module
|
||||
whole is the failure of the check step's fourth item.
|
||||
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it cannot find is a
|
||||
missing sibling worktree, not a mesh fault.
|
||||
|
||||
- After a feature merges, `git branch -r | grep feat/<slug>` and `git worktree list` return
|
||||
nothing for it. A surviving branch or worktree means step 6 was skipped.
|
||||
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it
|
||||
cannot find is a missing sibling worktree, not a mesh fault.
|
||||
- More than one open MR in a repo that share no feature name, or a stack of MRs none of which is
|
||||
merged, is the failure this playbook exists to prevent — stop and consolidate before opening
|
||||
more.
|
||||
When this playbook meets a problem, the lesson is added here, so the next feature avoids it.
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Playbook 08 — Small fixes
|
||||
|
||||
**Trigger.** One issue in one repository, small enough to build in about an hour: a bug with a known cause,
|
||||
a missing check, a wording, a playbook or design edit. Anything larger, or anything that needs a change in a
|
||||
second repository first, is a feature and follows [playbook 07](07-feature-branches.md). A feature is the
|
||||
larger kind: it touches several modules and has a research effort or a design before it. **Needing a decision
|
||||
does not make a fix a feature**: the decision is settled first (step 1) and the fix stays small (the operator,
|
||||
2026-10-16).
|
||||
|
||||
**Related issues in one fix.** A small fix may cover two or three related issues when they are in the same
|
||||
repository, mostly in the same module, and together still fit in about an hour: they touch the same files, or
|
||||
one would be wrong or unsafe without the other (switching a clipboard's history on before its history tool stops
|
||||
handing out what was copied). Each issue is named in the commit and the pull request and moved to `fixing`,
|
||||
and each is resolved on its own once the fix is delivered. More issues, a second repository, or more than an
|
||||
hour is a feature.
|
||||
|
||||
**Who runs it.** Anyone who writes code, engineers and agents alike.
|
||||
|
||||
## The aim
|
||||
|
||||
A small fix spends its time on the fix, and runs the long part once: **one review, one push, one check run,
|
||||
one delivery**. Everything before the push is local and fast.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Read the issue and the code it names** (the session's own model, or one read-only agent on the smaller
|
||||
model answering exact questions with file:line). If a design question is open, it is settled before any
|
||||
code is written: on the issue, through the knowledge base, or by the operator. A fix that turns out to
|
||||
need a second repository, or more than about an hour, stops here and becomes a feature; one that needs a
|
||||
decision stays a small fix. **The session's own model (the larger) then writes the issue's acceptance
|
||||
criteria** as playbook [07](07-feature-branches.md) step 0 item 5 says: each one sentence, given, when, then,
|
||||
naming the level of the test that proves it, as an `issue-tracker.note` headed `Acceptance criteria`. A
|
||||
playbook or design edit has none: its check is `sh merge-check.sh`.
|
||||
2. **Move the issue to `fixing`** with `by` as who@node.
|
||||
3. **Build on a local branch** `fix/<issue>-<slug>`, in a worktree of its own branched from `main`, never in
|
||||
the repository's own clone, which stays on `main` (playbook [07](07-feature-branches.md), step 0, item 2). Write the
|
||||
test that fails today first; make the change; keep `gofmt`, `go vet` and the repository's whole unit suite green
|
||||
before every commit (each test well under a second; no database in the unit tests: how-we-build §5, ADR 0391); before the push the repository's merge check runs once, integration tests included, through `mesh-controller.try` or `merge-check.sh` against a throwaway store, as playbook 07 step 1 says.
|
||||
The tests are written from the acceptance criteria, at least one per criterion at its level, each citing the
|
||||
criterion it proves by its label (`<issue>/AC<n>`, playbook 07 step 0 item 5). Commit, naming the issue. For a playbook or design edit, run the repository's own checks (in hq,
|
||||
`sh merge-check.sh`).
|
||||
4. **One review of the local commit, before anything is pushed** (a reviewer that did not write it, larger
|
||||
model; at most 3 rounds, as playbook 07; from round 2 only what is wrong; it reads each acceptance criterion
|
||||
against its test, as playbook 07 step 2 item 6; a finding outside the issue's done-when and its acceptance
|
||||
criteria is opened as an issue of its own, never asked as a change). The reviewer's findings go back to the same agent that wrote the fix, continued by
|
||||
its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the
|
||||
operator's Approve of its merge is its review, and the review record says so.
|
||||
5. **Push once and open the pull request.** The trunk is what runs and nothing holds merged work back
|
||||
(ADR 0401), so a fix waits for no other change's delivery. The integration tests ran once
|
||||
before the push (step 3); the forge's merge check runs them again as the gate. A failed check
|
||||
goes back to the same agent with the failure, and the reviewer reads only the change it makes.
|
||||
6. **The pull request's description carries the review record** (`Reviewed-by: <who@node> at <full sha>`,
|
||||
playbook [07](07-feature-branches.md) step 3), and mesh-delivery asks the operator's Approve as soon as its
|
||||
checks pass and merges the approved head (ADR 0421); nobody marks, labels, comments or rechecks. A fix to a module that changes how it runs carries an
|
||||
integration test starting the module whole against the real bus and store, run by the forge's checks
|
||||
(ADR 0391 §1, §6); unit tests alone are not that proof, and the lab is not used. The operator's word is waited for only when the fix changes what the operator sees, removes data,
|
||||
or needs the operator's terminal.
|
||||
7. **Follow its delivery plan to done** (`mesh-controller.plans` with `wait` and `until: done`), prove the fix on
|
||||
the mesh (the symptom gone, measured), and resolve the issue with `located-in` and `fixed-by` the same hour.
|
||||
|
||||
A fix that stops before its delivery moves its issue back to `open` with a note saying how far it got
|
||||
(branch, last commit, what is left), as playbook [07](07-feature-branches.md) says.
|
||||
|
||||
**A second fault found by the delivery.** When a fix's delivery shows a second fault that keeps the issue's
|
||||
done-when from holding, and that fault is in the same files, it is taken into the same issue as a second pull
|
||||
request with a review round of its own. A fault in other code is opened as an issue of its own, and the first
|
||||
issue waits for it (issue 937: mesh-catalog #335 was taken in; issue 943 was opened).
|
||||
|
||||
**A fix that moves a file to a new filler compares the values first.** When a fix makes a module's file filled a
|
||||
new way for the first time (the provisioner client in place of the controller, a provider's entry in place of a
|
||||
composed value), the fix lists every value the old way put into that file on each machine it reaches and shows,
|
||||
in its review, that the new way gives the same values, or says which differ and why. A difference nobody looked
|
||||
at is an outage: on 2026-10-16 the forge and the identity provider on the control node were down for 17 minutes
|
||||
because postgres's consumers were served the port inside its container instead of the port it is published on
|
||||
(issue 957). The comparison is read from the machines (the file as it is today) before the merge, not from the
|
||||
code.
|
||||
|
||||
## Bounds
|
||||
|
||||
| Waiting on | Bound | Past the bound |
|
||||
|---|---|---|
|
||||
| building the fix | 1 hour | it is a feature: stop and plan it under playbook 07 |
|
||||
| the review | 15 minutes | ask the reviewer where it stands |
|
||||
| the merge check | 15 minutes | read `mesh-controller.queue` and the check's log |
|
||||
| the Approve's ask | 5 minutes after both checks pass | read `mesh-delivery.show`: an ask not raised is a fault, said to the operator |
|
||||
| the merge after the Approve | 5 minutes | read `mesh-delivery.show` for the delivery |
|
||||
| the delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
|
||||
|
||||
A wait past its bound is said to the operator at once, with what is being done about it.
|
||||
|
||||
## Models
|
||||
|
||||
The fixing agent runs on the smaller model; the reviewer on the larger. A session fixing alone does the
|
||||
reading itself and uses one subagent only for the review.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- A small fix has one pull request with one or few commits naming its issue, opened after its review: a pull
|
||||
request with many heads is a fix reviewed after its push.
|
||||
- Its issue moves to `resolved` within the hour its delivery plan ends.
|
||||
- A pull request with no review record at its head (the Approve's ask says so first), a fix to a module's
|
||||
running pushed with no integration test of the module whole, or a review report
|
||||
asking a change outside the issue's done-when and acceptance criteria, is the failure of steps 4 to 6.
|
||||
- A fixed code issue shows a note headed `Acceptance criteria` (`issue-tracker.get`) older than its first
|
||||
commit, and each criterion's label is cited by a test in the pushed tree
|
||||
(`git grep -n '<issue>/AC<n>'`); one without is step 1 skipped.
|
||||
- An issue in `fixing` with nobody on it shows in `issue-tracker.list` with `flag: abandoned`; a stopped fix
|
||||
moves it back to `open` with its note.
|
||||
- The repository's own clone says `main` (`git -C <clone> branch --show-current`); another branch there is a
|
||||
fix built outside a worktree (issue 1010).
|
||||
@@ -0,0 +1,49 @@
|
||||
# Playbook 09 — Cleanup
|
||||
|
||||
**Trigger.** The operator asks for a cleanup, or the forge and the working copies have gathered what nobody
|
||||
uses: merged branches, stale pull requests, worktrees and folders of finished work.
|
||||
|
||||
**Who runs it.** A session on the operator's word, with the operator deciding every deletion. Agents read and
|
||||
propose; a deletion is the operator's act or is done on the operator's word for the list it was shown.
|
||||
|
||||
## The aim
|
||||
|
||||
Remove what is provably done, and nothing else. A cleanup that deletes one piece of unfinished work costs more
|
||||
than every stale branch it removed. So every item is shown with its evidence, and anything not provably safe is
|
||||
reported and left.
|
||||
|
||||
## Steps
|
||||
|
||||
1. **Take stock, reading only.** One agent on the smaller model reads, changing nothing:
|
||||
- the forge: every repository, every branch **as git refs** (the branch list tool misses some, issue 949),
|
||||
every open pull request with its head, title and last update;
|
||||
- each working copy on the machine: its worktrees, local branches, stashes, and the folders beside the
|
||||
clones, with size and newest file.
|
||||
It writes one report with each item, its class and its evidence, and a list of the tools it lacked.
|
||||
2. **Classify each item**, by evidence, never by name or age alone:
|
||||
- **Safe** — a forge branch whose tip is contained in its repository's default branch, with no open pull
|
||||
request, not protected and not a long-lived branch the mesh treats as a trunk; a worktree that is clean,
|
||||
whose branch is contained in the remote default branch, untouched for two hours and not in another
|
||||
session's working folder; a local branch contained in the remote default branch.
|
||||
- **Likely safe, not proven** — a branch whose every commit is patch-equal to one on the default branch
|
||||
(the forge lands some commits under new hashes): listed, deleted only on the operator's word for it.
|
||||
- **Kept** — unmerged commits, a dirty worktree, a stash, an open pull request, a protected or trunk-like
|
||||
branch, a repository of another owner, another session's worktree.
|
||||
- **Unknown** — a folder or file outside git: described, never opened where a secret may be, left.
|
||||
Stale open pull requests (no update for three days) are reported with their author's issue, never closed.
|
||||
3. **Ask the operator** with the counts per class and the report's place. One question: delete the safe
|
||||
items as listed? The likely-safe and unknown items are separate questions, item by item when few.
|
||||
4. **Delete only the approved list**, each with the command that refuses an unsafe case (a worktree removal
|
||||
that refuses a dirty one, a branch deletion that refuses an unmerged one). A refusal is reported, never
|
||||
forced. The operator approves a deletion on a channel that proves who answered, as an ask bound to that
|
||||
exact list and carried out on the warrant (ADR 0387, design 58 milestone 7, issue 884). Until that is built,
|
||||
an agent session's permission check refuses these deletions, and the approved list is handed to the
|
||||
operator to run.
|
||||
5. **Report**: what was cleaned, what was kept and why, what was unknown, what the operator kept, and the
|
||||
missing tools, each opened as an issue (`issue-tracker.open`) or added to an existing one.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- The report names, for every deleted item, the evidence it was safe (contained, clean, no pull request).
|
||||
- Nothing classed Kept or Unknown is deleted without the operator's word for that item.
|
||||
- A cleanup's missing tools are issues, not notes in a session.
|
||||
+1
-1
@@ -34,7 +34,7 @@ one-for-one — `mesh-catalog`, `mesh-sdk` and `mesh-tools` exist where the tabl
|
||||
| `mesh-foundation` | 1 | the four pinned services, as declarations |
|
||||
| `mesh-controller` | 2 | **exists.** The controller and its contexts — one of seven built (ADR 0006) |
|
||||
| `mesh-surfaces` | 3 | tools, web, cli |
|
||||
| `mesh-sdk` | — | the stable spine modules build against — the tool-serving harness, the messaging/event framework, the contracts and core primitives. Holds nothing per-module and nothing volatile (ADR 0039). |
|
||||
| `mesh-sdk` | — | the stable spine modules build against — the tool-serving SDK harness, the messaging/event framework, the contracts and core primitives. Holds nothing per-module and nothing volatile (ADR 0039). |
|
||||
| `mesh-lab` | — | **exists.** The lab — scenario lifecycle, networking, placement. Ships to nobody; runs on a workstation. |
|
||||
|
||||
Tier 4's shape is open, and deliberately so: see ADR 0019 and
|
||||
|
||||
@@ -1,100 +1,90 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [hal]
|
||||
updated: 2026-08-23
|
||||
code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0010"
|
||||
- "ADR 0010"
|
||||
- "ADR 0010"
|
||||
- "ADR 0239"
|
||||
- "ADR 0267"
|
||||
- "ADR 0276"
|
||||
- "ADR 0282"
|
||||
- "ADR 0289"
|
||||
- "ADR 0304"
|
||||
- "ADR 0350"
|
||||
- "ADR 0358"
|
||||
- "ADR 0360"
|
||||
- "ADR 0364"
|
||||
- "ADR 0382"
|
||||
- "ADR 0395"
|
||||
- "ADR 0398"
|
||||
---
|
||||
|
||||
# Delivery — from a push to a running node
|
||||
# Delivery — from a merge to the nodes, as it runs
|
||||
|
||||
One trigger, three silos, and a fan-out point that is the most consequential boundary in the
|
||||
mesh.
|
||||
What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055
|
||||
(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone
|
||||
by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs.
|
||||
|
||||
## The trigger
|
||||
## Who does what
|
||||
|
||||
A push to the forge. The forge calls a webhook; the receiving module verifies its signature
|
||||
and emits an event. The coordinator resolves which modules the pushed commits affect, orders
|
||||
them into dependency levels, and creates a pipeline per level.
|
||||
- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch
|
||||
name shared across repositories. It records each transition in its state on the bus (key-value buckets
|
||||
`deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked
|
||||
ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved
|
||||
the merge into main on a channel that proves who answered (ADR 0398 §3).
|
||||
- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a
|
||||
merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and
|
||||
reuses a merge check's build of the same reuse key (ADR 0350).
|
||||
- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus
|
||||
and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go
|
||||
or npm installed (issue 626).
|
||||
- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14
|
||||
(issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it
|
||||
announced so it does not announce twice.
|
||||
|
||||
There is no second path. A manual trigger exists for a module the push detection missed, and
|
||||
using it routinely is a sign the detection is wrong rather than a workflow.
|
||||
## From a merge to a rollout
|
||||
|
||||
Detection is the pipeline's most fragile input. It has failed for reasons that have nothing to
|
||||
do with the change: a webhook truncating its commit list on a large merge, a forge address
|
||||
whose port broke the module-path matching. The characteristic outcome is the bad one — **a
|
||||
merge that created no pipeline, and nothing said so**.
|
||||
- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which
|
||||
closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is
|
||||
cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's
|
||||
store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds.
|
||||
- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module
|
||||
graph. Rollouts run one at a time: a batch cut while a rollout is open is queued behind it, whatever it shares
|
||||
with it. Running unrelated rollouts side by side (ADR 0282 §3, ADR 0304 §2) is decided and not built (issue 525). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on
|
||||
the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every
|
||||
30 s.
|
||||
- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On
|
||||
2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released
|
||||
it (issue 921).
|
||||
- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that
|
||||
branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for
|
||||
55.7 h.
|
||||
|
||||
## Three silos
|
||||
## Checks
|
||||
|
||||
Cardinality is the whole point, and the three differ
|
||||
(ADR 0010):
|
||||
- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's
|
||||
`merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active,
|
||||
cleanly merging pull request again (ADR 0395).
|
||||
- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed
|
||||
check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382).
|
||||
- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and
|
||||
deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the
|
||||
current head (issue 726).
|
||||
- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks
|
||||
every minute, although the forge module already publishes labeled, commented and synchronized events.
|
||||
|
||||
| Silo | Runs | Where | Does |
|
||||
|---|---|---|---|
|
||||
| **build** | once per module feature | the build node | compile and bundle into a self-contained output |
|
||||
| **publish** | once per module feature | the build node | package that output and upload it; a package-registry feature publishes here |
|
||||
| **deploy** | once per module feature **per node** | every assigned node | install, configure, start, verify |
|
||||
## Where the record of what runs lives
|
||||
|
||||
Commands and events are addressed per feature, not per module.
|
||||
Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the
|
||||
controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the
|
||||
package registry. The controller writes three record mirrors to the bus: assignments, declared config and
|
||||
predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the
|
||||
code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit
|
||||
runs where, and which delivery brought it, takes three verbs and a join by hand.
|
||||
|
||||
Build hands over a **staged tree**, not a package. Packaging belongs to publish, so a failed
|
||||
upload retries by re-packaging rather than by re-sending something stale, and build never needs
|
||||
to know how each module composes its artifact. The handover is a tree in a known location,
|
||||
because the build's own working directory is reference-counted and may be gone by the time a
|
||||
later stage runs.
|
||||
|
||||
## The artifact
|
||||
|
||||
The artifact is **build output** — compiled and bundled with its dependency graph inlined —
|
||||
never a filtered copy of source (ADR 0010).
|
||||
A deploy is extract-and-run and touches no network.
|
||||
|
||||
The consequence is the whole cost of the decision: **anything not in the build output does not
|
||||
ship.** Every file kind had to be brought into that rule separately, and each was discovered by
|
||||
something silently not happening after deploy — migrations reading a source layout, provisioning
|
||||
scripts reading a source layout, selection files never packaged at all.
|
||||
|
||||
## The fan-out, and the build node
|
||||
|
||||
After publish, work fans out to every assigned node. The build node is the only node that has
|
||||
already passed through two silos when this happens, and that asymmetry has its own defect
|
||||
class: anything advancing a node's stage must account for **both** pre-fan-out stages. Code
|
||||
that knew only about the first parked the build node forever while every other node deployed
|
||||
cleanly — and the recovery sweeper, which knew the same subset, reported nothing to recover.
|
||||
|
||||
A recovery mechanism that knows less than the thing it guards is worse than none, because it
|
||||
reports success over a stall it cannot see.
|
||||
|
||||
## Levels
|
||||
|
||||
A level completes before the next begins, so a module builds against its dependencies as they
|
||||
were just published. The shared library is at level zero, which means anything that breaks it
|
||||
breaks the first module of every cascade.
|
||||
|
||||
## What green proves
|
||||
|
||||
**A green pipeline proves transport, not effect.** The stages report that a message was
|
||||
dispatched and accepted, which is not the same as the thing being running, correct, or present.
|
||||
|
||||
This is the mesh's most consistent failure shape and it is not incidental to the design — it is
|
||||
what the stage reporting currently measures. Documented instances include a service reported
|
||||
started when the container command merely returned, an image pull failure that did not fail the
|
||||
deploy, a package install that 404ed from every mirror while the job went green (see
|
||||
[`04-ISSUES/001`](../../04-ISSUES/001-failed-package-install-reports-success/00-report.md)),
|
||||
and a node left on old code after a failed artifact download with a version marker that had
|
||||
already advanced.
|
||||
|
||||
A verify stage exists to close this gap. It has been built and, for a period, was never
|
||||
scheduled, because the coordinator's stage list did not include it.
|
||||
No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214).
|
||||
|
||||
## What is not covered
|
||||
|
||||
The end-to-end harness for this pipeline has not built since the workspace was removed, and
|
||||
nothing reports that nothing runs it
|
||||
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). The
|
||||
pipeline's coverage is currently assumed rather than checked — which is the same class of claim
|
||||
this repository exists to make people stop making.
|
||||
The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet,
|
||||
and the lab is not used (the operator, 2026-10-16).
|
||||
|
||||
@@ -1,69 +1,144 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [mesh-catalog modules/mesh-console, mesh-tools src/mesh.ts, mesh-tools src/http.ts, mesh-tools src/runtime.ts, mesh-controller internal/broker, mesh-controller cmd/mesh-controller/check.go, mesh-controller cmd/mesh-controller/seatverbs.go]
|
||||
updated: 2026-10-10
|
||||
code: [mesh-tools, mesh-host, mesh-controller]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0152"
|
||||
- "ADR 0095"
|
||||
- "ADR 0037"
|
||||
- "ADR 0175"
|
||||
- "ADR 0195"
|
||||
- "ADR 0197"
|
||||
- "ADR 0367"
|
||||
- "ADR 0154"
|
||||
---
|
||||
|
||||
# The mesh MCP server, as it runs
|
||||
|
||||
**The mesh's tools reach a person through a module the mesh assigned to their machine.** Since
|
||||
2026-09-30 a workstation that is a node can be assigned `mesh-console`; the mesh mints a bus account
|
||||
`<node>.mesh-console`, seals its credential to the machine, and the container binds
|
||||
`127.0.0.1:<port>` with the port the mesh assigned for the manifest's declared one. An agent on the
|
||||
machine is pointed at `http://127.0.0.1:<port>/mcp` and sees the mesh's tools; a person uses the same
|
||||
endpoint. Nothing on the machine holds a credential a person had to carry.
|
||||
**The `mesh-console` container no longer exists.** Read 2026-10-17 from `mesh-tools` (package
|
||||
`node-tools`), `mesh-host` and `mesh-controller` on their forges' main, and from the live mesh
|
||||
through `mesh_overview`, `mesh_machine`, `mesh_runtimes` and `mesh_describe`. Checked: the
|
||||
catalogue (`mesh-catalog`) carries no module named `mesh-console` any more — the name survives only
|
||||
in the glossary's retired-words list and in a test fixture — and none of the four nodes answers a
|
||||
module of that name.
|
||||
|
||||
## What it is now
|
||||
|
||||
**The tool runner is one of the mesh's core components**, named `node-tools` beside
|
||||
`mesh-controller`, `mesh-host` (the node-engine) and `nats` (the bus) — the mesh's own repositories'
|
||||
check treats the four the same way, as the core whose health the mesh itself defines rather than a
|
||||
module's own. `mesh-host` and `mesh-controller` supervise, compose and account for it as a core
|
||||
process rather than as an ordinary assignment (`ComponentNodeTools`/`BusNodeTools` in both, the
|
||||
witness and lease code that restarts and rolls it back, the account named `<node>.node-tools` on
|
||||
the bus).
|
||||
|
||||
**It still carries a manifest, `node-tools/module.json` in `mesh-tools`,** and that manifest is
|
||||
where its grants live: `invokes` names `*`, so its account may call every tool on the mesh;
|
||||
`waits` names the five event families `mesh_wait` may subscribe to (the knowledge base's reviews,
|
||||
mesh-delivery's transitions, the forge's status, the controller's checked event, and the operator
|
||||
channel's decision for the agent module) — the bus refuses a subscription outside this list
|
||||
(ADR 0367 §2); and it
|
||||
`provides` one provision, the MCP endpoint, at node scope, and `listens` on the port it serves from,
|
||||
opened only to the machine itself. Its code lives under `node-tools/` in the `mesh-tools`
|
||||
repository (`internal/console`, `internal/announce`, `internal/bus`).
|
||||
|
||||
**One runtime per node, run by the node-engine as a sibling process, never a container**
|
||||
(ADR 0175). It holds the node's one bus credential and may call every tool on the mesh; its account
|
||||
publishes nothing else. It binds `127.0.0.1:<port>` and refuses to bind anything else, said before
|
||||
binding rather than after (`node-tools/internal/console/http.go`, citing ADR 0152): whoever is on
|
||||
the machine owns the mesh there, and nobody else may reach this port. An agent or a person on the
|
||||
node is pointed at `http://127.0.0.1:<port>/mcp` once; the `serverInfo.name` the handshake answers
|
||||
is `mesh`.
|
||||
|
||||
## What it answers
|
||||
|
||||
`initialize`, `tools/list`, `tools/call`, over HTTP, one JSON body per request, no session and no event
|
||||
stream. `tools/list` is what the running modules answered: every tool runner built on or after that day
|
||||
serves a `tools` verb for its module, and the mesh MCP server asks the catalogue for the roster and each module
|
||||
for its tools. A module that did not answer is named in the list's `_meta.notAnswering`. On the day it
|
||||
shipped that was 36 of 51 modules — those that serve no tools at all, and those whose rebuilt runtime the
|
||||
mesh records rather than rolls out — and 62 tools from the rest.
|
||||
`initialize`, `notifications/initialized`, `ping`, `tools/list`, `tools/call`, over HTTP, one JSON
|
||||
body per request, no session and no event stream (`node-tools/internal/console/mcp.go`).
|
||||
|
||||
`tools/call` reaches any tool by `<module>.<tool>`, listed or not. The mesh MCP server's grant is `*`, so what it
|
||||
may call is every tool on the mesh; its account may publish nothing else and subscribes nothing.
|
||||
**It announces seven tools, not five** (ADR 0195, extended by ADR 0367): `mesh_overview`,
|
||||
`mesh_machine`, `mesh_search`, `mesh_describe`, `mesh_call`, `mesh_runtimes` and `mesh_wait`. The
|
||||
handshake's `instructions` text names all seven and the four address forms. `mesh_overview` is
|
||||
where the mesh-wide count of modules and tools is read live — a number copied into this document
|
||||
would drift; asked on 2026-10-17 it answered 103 modules and 645 tools across the whole mesh, with
|
||||
two module instances (`route-proxy` on two of the nodes) named as not answering rather than silently
|
||||
absent.
|
||||
|
||||
No call is refused for want of a person's approval: a tool runner serves every call it is issued, whoever made
|
||||
it, and cannot tell an agent from the operator (ADR 0259 §6, issue 340). An act waits for the operator's warrant
|
||||
only where its module asks first, in its own code: the agent module's instruction proposals, and the
|
||||
controller's settings proposals and the actions it offers on a condition.
|
||||
**Addresses, exactly as the handshake states them:** `<seat>.<verb>` for a seat held once for the
|
||||
mesh (`mesh-controller.status`); `<node>/<seat>.<verb>` for a seat every machine holds
|
||||
(`<node>/node-packet-filter.rules`); `<node>/<module>.<tool>` for a module assigned to one machine;
|
||||
and `<module>.<tool>` also for a module whose instances are interchangeable. A tool not listed can
|
||||
still be called by address — listing is discovery, calling is the grant — and a module that is not
|
||||
interchangeable is refused with the machines it runs on rather than guessed at.
|
||||
|
||||
**What exists is what announced itself** (ADR 0197). Every node's tool runner answers the NATS services
|
||||
protocol's `$SRV.PING`, `$SRV.INFO` and `$SRV.STATS`, one service per process, naming in each
|
||||
endpoint's metadata which module, seat, scope and machine serves it. `mesh_overview` and
|
||||
`mesh_machine` gather one request's answers: at least 750ms so a runtime that only just answered
|
||||
`$SRV.PING` is still waited for, up to 5s for one that said it is there but has not yet sent its
|
||||
full announcement — a fixed shorter window lost the largest runtime's listing once a live
|
||||
measurement found how slow its round trip over the bus could be. An announcement unchanged since
|
||||
last asked (by its digest) is not re-fetched, which is why `mesh_runtimes` can show a runtime
|
||||
answering in under 100ms for hundreds of kilobytes of discovery.
|
||||
|
||||
## mesh_wait
|
||||
|
||||
**The seventh tool, added by ADR 0367**, lets a session wait on the bus for an event instead of
|
||||
polling: a call names one or more granted event families and an optional filter, and either answers
|
||||
the first match or says it timed out. The tool runner's own bus account hears only the families its
|
||||
manifest's `waits` list names; an event it hears is never taken as the state — a session reads the
|
||||
state from the verb that owns it once woken. It is meant to be called in a loop from a background
|
||||
command, not inline, because one call blocks until it answers or times out.
|
||||
|
||||
## The mesh's own verbs
|
||||
|
||||
*Since 2026-09-30 evening (ADR 0154).*
|
||||
The mesh MCP server asks the `mesh-controller` seat's `tools` verb beside the modules and lists every role's
|
||||
tools as `<seat>.<verb>` — `mesh-controller.status`, `mesh-controller.push` and the other ten. A call
|
||||
to `<prefix>.<name>` reaches the seat when the prefix is a seat declaring that verb, and the module
|
||||
otherwise; `seat:<seat>.<verb>` says so outright. When the controller does not answer, the list
|
||||
names `mesh-controller (seat)` as not answering and carries the modules' tools regardless. The
|
||||
`mesh-controller` *module* is always named as not answering: it serves no module tools, only its seat's.
|
||||
**These already answer directly, not as a stated future.** Beside every assigned module's tools,
|
||||
the tool runner asks the `mesh-controller` seat for its own verbs the same way (ADR 0154): measured
|
||||
live, the mesh-controller runtime is one instance, mesh-wide, and it answers 63 verbs for that one
|
||||
seat (`status`, `push`, `assign`, `build`, `artifacts` and the rest), and the `mesh-controller`
|
||||
*module* answers none of its own — it serves no module tools, only its seat's, exactly as design 33
|
||||
described while it was still only designed. The 56-to-93 seat-verb counts each node's tool runner
|
||||
answers (below) are a different thing: the tool runner serves the verbs of whichever node-scoped
|
||||
seats are held on that node (`node-backup`, `node-build-agent`, `node-service-manager` among them),
|
||||
on behalf of the modules holding them, not the mesh-controller seat's — that one is served by the
|
||||
mesh-controller runtime itself, a separate process, and the tool runner only asks it and relays what
|
||||
it answers.
|
||||
|
||||
## Around it
|
||||
## Where it runs
|
||||
|
||||
- **`invokes`** in a manifest is the grant. It is composed into the bus's user list exactly as a
|
||||
person's account is; the mesh MCP server is the only module that declares it.
|
||||
- **`module check <file|dir>…`** on the controller's binary judges a manifest with no mesh: the strict
|
||||
parse, every per-manifest problem, and the rules between the manifests given. It prints what it cannot
|
||||
judge without a store rather than refusing. The mesh MCP server's own manifest was the first thing checked
|
||||
with it, and the whole catalogue passes.
|
||||
- **The person's client remains.** `operator issue` and `mesh tools|call|mcp` with a credential file
|
||||
still work, for a machine that is not a node and for a mesh not yet able to assign anything.
|
||||
`mesh tools --console <url>` goes through a running mesh MCP server with no credential; it is covered by the
|
||||
runtime repository's tests and was not exercised on the live mesh.
|
||||
**Every node, not a per-operator assignment.** Measured live: all four of the mesh's machines
|
||||
answer a `node-tools` runtime, each with its own module and tool count (between 294 and 312 tools,
|
||||
56 to 93 seat verbs, in this reading). This is a fact that shipped past the design it came from: the
|
||||
to-be document still describes it as assigned to "whichever machines an operator sits at"; what
|
||||
actually runs is the opposite — every node the mesh holds carries one, because it is core
|
||||
infrastructure the mesh itself depends on (discovery, the controller's own verbs, every module's
|
||||
tools), not a convenience assigned to a workstation. There is accordingly no assign or unassign verb
|
||||
for it: removing it from a node is not an operation the controller or the catalogue exposes.
|
||||
|
||||
## What shipped bent
|
||||
## What this repository's prior text got wrong
|
||||
|
||||
- A module registered by hand from the catalogue with `--source <url> --path modules/<m>` records a URL,
|
||||
not a place on the git seat: `--self` takes the forge path form (`<owner>/<repository>`), which the
|
||||
operator did not pass. The rebuild-on-merge matched the URL anyway.
|
||||
- Modules whose upgrade policy is *record* — the forge among them — answered `tools` only once
|
||||
something pushed their rebuilt runtime; until then they are listed as not answering while still
|
||||
callable. That is the policy doing what it says, not a fault of the mesh MCP server.
|
||||
The document this replaces described `mesh-console` as a catalogue module served by a container and
|
||||
assigned only to workstations. That module and its container are gone; the manifest is not — it
|
||||
moved to `node-tools/module.json`, still declares `invokes: ["*"]`, and gained `waits` and the
|
||||
`mcp-endpoint` provision, described above. What also moved is which language builds it: the
|
||||
manifest's `build.artifacts` names a Go binary built from `cmd/node-tools`, and that is what a node
|
||||
runs. The repository still carries the earlier TypeScript sources (`node-tools/src/mesh.ts`,
|
||||
`http.ts`, `runtime.ts`, `announce.ts`, `broker-nats.ts`, `client.ts`, `launch.ts`, `main.ts`,
|
||||
`mcp.ts`) on main, but the manifest does not build from them. The quirks the prior text recorded
|
||||
from the `mesh-console` era (a module registered by hand recording its source as a URL; a
|
||||
*record*-policy module staying listed as not-answering until its rebuilt runtime is pushed) are not
|
||||
re-checked here: they were about the `mesh-console` container's registration path, which no longer
|
||||
exists.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- `mesh_overview` and `mesh_machine <node>`: the live tool and seat-verb counts, and which modules
|
||||
are named not-answering.
|
||||
- `mesh_runtimes`: which runtimes answered discovery, how long they took, how large their
|
||||
announcement was, and whether it was shortened to fit the bus — read this when an address is said
|
||||
to be missing or on another machine.
|
||||
- `mesh_describe <address>` for one tool's schema; a call through `mesh_call` reaches the module or
|
||||
seat over the bus and the answer is its own, unshaped.
|
||||
- The code: `mesh-tools` (`node-tools/internal/console`, `internal/announce`, `internal/bus`),
|
||||
`mesh-host` and `mesh-controller`'s core-component handling of `node-tools` (witness, lease,
|
||||
bus-user naming). Their unit tests hold the loopback-only refusal, the seven-tool handshake, the
|
||||
discovery gather window and `mesh_wait`'s event-then-read-state rule.
|
||||
- `knowledge-base.get` with `kind: decision` and number 152, 175, 195, 197, 367 or 154.
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [mesh-controller, mesh-tools]
|
||||
updated: 2026-10-16
|
||||
decisions:
|
||||
- "ADR 0106"
|
||||
---
|
||||
|
||||
# Logins on the bus, as they run
|
||||
|
||||
**The bus reads one static list of logins from its settings file, in one account, written by the controller from
|
||||
its own records. Every module has a login per assignment; a person's login may call only the tools it was given;
|
||||
the sessions on the operator's machines call through their machine's tool-runner login, which may call every tool.
|
||||
Nothing marks what a tool does.** Read from the controller's and the tool runner's code on main on 2026-10-15
|
||||
(research 053, question 8). Design 58 replaces it.
|
||||
|
||||
## The list
|
||||
|
||||
- **One list, one account.** The controller writes every login from its records into the bus's settings file;
|
||||
the bus re-reads it without dropping connections (ADR 0106 §3).
|
||||
- **A module's login per assignment**, named by machine and module, may publish and subscribe only what its
|
||||
manifest declares it emits, consumes, serves and calls.
|
||||
- **A person's login** may publish only to the tools it was given and listen only on its own inbox.
|
||||
- **The caller is in the address.** A call's address carries the caller's name, so the runtime takes the caller
|
||||
from the address, never from the message. For a seat every machine holds, the address also names the machine.
|
||||
|
||||
## What it cannot do
|
||||
|
||||
- **No expiry.** A login lives until its record is deleted and the list rewritten on the bus's node.
|
||||
- **No per-agent login.** Agent sessions reach the bus through each machine's tool-runner login, which may call
|
||||
every tool on the mesh; what one session did cannot be told from another's.
|
||||
- **No effect.** No tool or seat verb declares whether it reads, acts or is dangerous; some module documents say
|
||||
it in prose, and nothing reads that.
|
||||
- **Asks and warrants exist** (ADR 0259): a module can have one act authorised by the operator, bound to that
|
||||
act; nothing yet asks for a dangerous call by itself.
|
||||
|
||||
## How it is checked
|
||||
|
||||
By reading: the controller's code that renders the bus's settings and the tool runner's caller check, on main.
|
||||
When design 58's switch is done, this document is replaced by the as-is of signed logins.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [mesh-controller, mesh-catalog modules/claude-code, mesh-catalog modules/gitea, mesh-catalog modules/ssh-client]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0266"
|
||||
- "ADR 0399"
|
||||
- "ADR 0409"
|
||||
- "ADR 0410"
|
||||
---
|
||||
|
||||
# Accounts on a machine, as they run
|
||||
|
||||
**A node record names at most two logins: the operator account, always, and an agent account, on one node of
|
||||
four. Where a node names no agent account, agents run as the operator account. Nothing ties an identity to a
|
||||
login, and a module that needs one declares it itself.** Read on 2026-10-17 from the controller's and the
|
||||
catalogue's code on main (both local clones were a few commits behind the forge, so each claim below was
|
||||
also checked against a live read) and from the live node records. The change planned in ADR 0411 to 0414
|
||||
replaces this document when it is built.
|
||||
|
||||
## The node record
|
||||
|
||||
- **The operator account is stated at the controller's terminal** with `node account <node> <account> [home]`.
|
||||
It is the person's login on that machine. The controller refuses to set it to the node's agent account.
|
||||
- **The agent account is stated at the controller's terminal only** with `node agent-account <node> <account>
|
||||
[home]`, and cleared with `--clear`. It is no verb and no setting, so an agent cannot name itself one
|
||||
(ADR 0266 §1). The controller refuses root, a malformed login, a well-known service account and the
|
||||
operator account.
|
||||
- **Five machine facts come from the record.** `account` and `account-home` are the operator's, absent when
|
||||
the node has no operator account. `agent-account` is the agent account, or the operator account where none is
|
||||
named; `agent-home` is its home (`/home/<account>` when no home is stated); `agent-root` is `never` only where
|
||||
the node names an agent account of its own, otherwise empty. Modules read them as `${machine:...}` words.
|
||||
- **One node of four names an agent account today**, the control node. The workstation, the laptop and the
|
||||
home server name none: `mesh-controller.node` says there "agents run as the operator account; no agent
|
||||
account is named", and the plan for the workstation declares the `claude-code` account resource with the
|
||||
operator's name and no `root` field. The tool runner also receives `MESH_AGENT_ACCOUNT` and `MESH_AGENT_HOME`.
|
||||
- **Not verified:** whether the mesh-config snapshot carries either account. The controller's code reads both
|
||||
from its own node table (columns for account, account home, agent account and agent home); no second place
|
||||
that states them was found.
|
||||
|
||||
## What claude-code does with the agent account
|
||||
|
||||
- **It declares a `user` resource** named `${machine:agent-account}` with home `${machine:agent-home}` and
|
||||
`root: ${machine:agent-root}`, and a `~/.claude` directory under that home owned by it, mode 0700. On the
|
||||
control node the plan holds the resource with `root: never`; elsewhere it holds the operator's account.
|
||||
Where the account is new, the mesh creates it with a home, its own group and no password (ADR 0266 §2).
|
||||
- **`claude-agent`** is a file the module writes at `/usr/local/bin/claude-agent`. Run as the agent account
|
||||
it starts `claude`; run as any other account it runs `sudo -iu <agent account> claude` through that
|
||||
account's own sudo, carrying the operator's guard override only if it is set in that shell. The file
|
||||
`/etc/claude-code/agent-account` holds the account's name.
|
||||
- **The guard of ADR 0266 §28** is the module's hook on every tool call. On a node with an agent account
|
||||
of its own it refuses each call of a session that runs as another account, with no override, and tells the
|
||||
session to start with `claude-agent`. Its record goes to the machine's journal. The same guard also refuses
|
||||
ssh to a mesh machine and writes to `/etc/hosts` and the resolver file (ADR 0245).
|
||||
- **Not verified live:** this rests on the guard's code and README, not on a refusal seen on a running machine.
|
||||
|
||||
## The judgement that the agent account cannot become root
|
||||
|
||||
- **The node-engine judges `root: never` on every look**, by reading only: uid 0, a group that grants root,
|
||||
a sudo, doas or polkit rule, a container runtime's socket or API port, a secret the mesh placed for another
|
||||
account, a setuid program no package owns, and kernel link protection off (ADR 0266 §12 to §16). A way found,
|
||||
or a question unanswered, makes the account resource unhealthy. It does not prevent any way to root.
|
||||
- **The controller's self-check holds** that a node naming an agent account has it judged, in a verdict heard
|
||||
within 15 minutes, as unable to become root (ADR 0266 §21 to §22). Otherwise it raises the urgent condition
|
||||
`agent-can-become-root`. `mesh-controller.node` shows the verdict: for the control node it reads "cannot
|
||||
become root without a person", judged the same day.
|
||||
- **On the three nodes without an agent account** the check does not apply. The controller instead holds an
|
||||
urgent condition `root-not-free` on each machine that runs reads or the mail notifier. Two of the three are
|
||||
silenced until 2026-10-23 on the operator's decision; the home server's is open and not silenced.
|
||||
|
||||
## The accounts agents push as on the forge
|
||||
|
||||
- **Today the forge has a user per machine**, made by the gitea module's `forge-account` provision and
|
||||
holding that machine's key, plus a write token the module still mints for each (ADR 0398, ADR 0400). ADR 0410
|
||||
decides one forge user, `nox-agent`, in the agents' write team, each machine's key titled by machine. No tool
|
||||
reads the forge's user list, so whether `nox-agent` exists yet is **not verified**.
|
||||
- **The forge key is meant for the agent account** (ADR 0409 §2), but a module cannot run as that account:
|
||||
the controller refuses it (issue 989, open). So the `ssh-client` module's key keeper still runs as the
|
||||
operator account.
|
||||
- **An ordering deadlock** between `ssh-client`, which wants `forge-account`, and `gitea`, which provides it,
|
||||
stopped every node's declaration from composing on 2026-10-17 at 00:35 until gitea was registered three
|
||||
minutes later (issue 986). The four urgent conditions it raised were not observed again but stay raised,
|
||||
because the controller's declaration check times out and never clears them (issue 992).
|
||||
Git over ssh was proven from the laptop only; the proof on the other three, and retiring the forge token the
|
||||
module still places, wait on issues 989 and 991 (the latter in `fixing`).
|
||||
- **Main stays closed**: a push to the main branch is refused, and a merge waits for a person (ADR 0398).
|
||||
|
||||
## Logins on the bus
|
||||
|
||||
Not here: see [`16-logins-on-the-bus.md`](16-logins-on-the-bus.md).
|
||||
|
||||
## What it cannot do
|
||||
|
||||
- **No identity carries a login.** An agent, a person and a module are not tied to an account on a machine;
|
||||
the node record names a login per node and nothing else.
|
||||
- **A module declares a login itself.** Each module that needs an account writes a `user` resource of its own,
|
||||
and nothing checks that two modules agree or that a login exists on every machine that needs it.
|
||||
- **Agent and operator share a login on three of four machines.** There the agent runs as the operator account,
|
||||
which may become root; the guard's account rule does not apply, and the exposure is a held condition, not a fix.
|
||||
- **No per-agent bus password.** Every session on a machine calls through that machine's tool-runner login
|
||||
(document 16), so one agent's calls cannot be told from another's.
|
||||
- **Detects, does not prevent.** The judgement above lists the ways to root it looks for; a way it does not list
|
||||
is not seen (ADR 0266 §17).
|
||||
|
||||
## How it is checked
|
||||
|
||||
- `mesh-controller.node` with a node name: "agents run as" line, the verdict and when it was judged, the open
|
||||
conditions (`agent-can-become-root`, `root-not-free`, `uncomposable`).
|
||||
- `mesh-controller.plan` with a node name: the `claude-code.agent-account` resource and its `root` field; with
|
||||
`files` set, the files given, including `claude-agent` and `agent-account`.
|
||||
- `mesh-controller.conditions` for the held conditions; `claude_code_guard` for the guard's record, `claude-code`
|
||||
tools for its status.
|
||||
- `knowledge-base.get` with `kind` decision and number 266, 409 or 410; `issue-tracker.get` with 989 or 991.
|
||||
- The code: the controller's node-record and agent-account files, and the `claude-code` module's manifest and
|
||||
guard in the catalogue. Their unit tests hold the node record's refusals and the guard's account rule.
|
||||
@@ -24,6 +24,8 @@ Where the two disagree, the implementation wins and the disagreement is stated.
|
||||
| [`13-the-console.md`](13-the-console.md) | The mesh's tools on the machine a person sits at, served by a module the mesh assigned there |
|
||||
| [`14-a-modules-state.md`](14-a-modules-state.md) | A module's current state on the bus: what the controller creates, the runtime serves, and the first live use showed |
|
||||
| [`15-the-agent-and-its-licences.md`](15-the-agent-and-its-licences.md) | The operator's agent on every machine and the licence manager that keeps its licences |
|
||||
| [`16-logins-on-the-bus.md`](16-logins-on-the-bus.md) | The bus's one list of logins, a login per module assignment, persons limited to their tools, sessions calling through the tool runner's login, and no effect on any tool |
|
||||
| [`17-accounts-on-a-machine.md`](17-accounts-on-a-machine.md) | The operator and agent accounts on the node record, claude-code's own login and guard, the judgement that the agent account cannot become root, and the forge accounts agents push as |
|
||||
|
||||
## What these documents are not
|
||||
|
||||
|
||||
@@ -340,7 +340,7 @@ once, and maintain for a year. The modules are the input; a person reads what on
|
||||
writes what it declares tomorrow.
|
||||
|
||||
**It also changes what "safe" means for the system being retired.** A fix to it has to be safe on
|
||||
its own, because there is no careful walk to sequence it into: the thing is being switched off
|
||||
its own, because there is no careful rollout to sequence it into: the thing is being switched off
|
||||
by hand, not managed into retirement. A change needing three steps in the right order is a change
|
||||
that will be half-applied.
|
||||
|
||||
|
||||
@@ -22,8 +22,10 @@ code:
|
||||
- mesh-host internal/apply (the service that reflects a rule set; a whole file handed to its new owner)
|
||||
- mesh-controller cmd/mesh-controller/nodes.go (a token issued for a machine's tunnel key, ADR 0169)
|
||||
- mesh-host cmd/mesh-host/join_tunnel.go (a machine joins through the tunnel, ADR 0169)
|
||||
updated: 2026-10-13
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0241"
|
||||
- "ADR 0117"
|
||||
- "ADR 0340"
|
||||
- "ADR 0247"
|
||||
- "ADR 0226"
|
||||
@@ -1119,6 +1121,113 @@ against today's behaviour, where both are always composed; an unrouted endpoint
|
||||
named; an assignment naming an endpoint the module does not declare refused where it is said; and a
|
||||
silent assignment rendering byte-identically to today.
|
||||
|
||||
## The five through the network layers
|
||||
|
||||
*2026-10-09. A reading aid: it decides nothing and states no rule. Each claim names the section or the
|
||||
record that decided it.*
|
||||
|
||||
The OSI model is the standard seven-layer picture of a network: every layer wraps what the layer above
|
||||
hands it, and the receiving side unwraps in reverse. The five contexts of this document act at
|
||||
different layers, and seeing which explains why they fail independently. A resolver that stopped
|
||||
answering leaves the private network working, and a tunnel that cannot handshake leaves every name
|
||||
resolving to an address nothing can reach.
|
||||
|
||||
| Layer | What it carries | Where the mesh acts there |
|
||||
|---|---|---|
|
||||
| 7 Application | the message itself | the bus (ADR 0106); resolution (§2); exposure, where the proxy chooses a route by the name a request asks for (§3) |
|
||||
| 6 Presentation | how what is said is encoded and encrypted | certificates (§5): TLS, which is usually drawn between 4 and 7 and placed here by the book |
|
||||
| 5 Session | a conversation kept open | a node's link to the bus, opened once and reopened after the machine sleeps |
|
||||
| 4 Transport | ports, TCP and UDP | filtering (§4), which opens a port to a source; the private network itself travels as UDP |
|
||||
| 3 Network | addresses and the path between them | the private network (§1): every node's address, the peer graph, the hub relaying |
|
||||
| 2 Data link | frames on one network segment | the machine's own uplink (ADR 0117), on the underlay |
|
||||
| 1 Physical | cable, fibre, radio | nothing of the mesh's |
|
||||
|
||||
### Two networks, one inside the other
|
||||
|
||||
**The private network is a layer 3 network carried at layer 4 of the underlay.** A packet for another
|
||||
node's private address is encrypted whole and becomes the payload of a UDP datagram addressed to a
|
||||
peer's underlay address. Every hop on the underlay reads only the outer envelope, up to its layer 3,
|
||||
and never sees inside.
|
||||
|
||||
```
|
||||
underlay │ L1 radio │ L2 frame │ L3 to the hub's public address │ L4 UDP │ ▓▓ encrypted ▓▓ │
|
||||
│
|
||||
decrypted at the hub │
|
||||
┌────────────────────────────────────────────────┘
|
||||
▼
|
||||
private network │ L3 to a private address │ L4 TCP │ L5–L7 a message on the bus │
|
||||
```
|
||||
|
||||
### The shape the layers are drawn on
|
||||
|
||||
The shape is §1's: a hub, direct peering between nodes at the same site, and a roaming node that goes
|
||||
through the hub only. In a small mesh one node is both the anchor and the hub, and holds the bus.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ hub and anchor — reachable from outside │
|
||||
│ holds the bus │
|
||||
└─────▲──────────────────▲─────────────▲───┘
|
||||
│ │ │
|
||||
│ each tunnel is dialled out by the node below it
|
||||
│ │ │
|
||||
┌─────┴─────┐ ┌──────┴──────┐ ┌────┴────────┐
|
||||
│ laptop │ │ home-server │ │ home-server │
|
||||
│ no site: │ └──────▲──────┘ └──────▲──────┘
|
||||
│ hub only │ └──── direct ───┘
|
||||
└───────────┘ same site, peered
|
||||
```
|
||||
|
||||
**Every arrow points at the hub.** A node with no public address still has a tunnel, because it dials
|
||||
out and the hub answers; only the hub's tunnel must be dialable, at a stable address
|
||||
(ADR 0169).
|
||||
|
||||
### One tool call, from the laptop to a home-server
|
||||
|
||||
The laptop wraps the call one layer at a time:
|
||||
|
||||
| Layer | On the laptop |
|
||||
|---|---|
|
||||
| 7 | a message on the bus naming the tool and its arguments |
|
||||
| 6 | the message encoded for the wire |
|
||||
| 5 | the laptop's link to the bus, already open |
|
||||
| 4 | TCP to the bus at its private address |
|
||||
| 3 | a packet from the laptop's private address to the bus's |
|
||||
| 3 inside 4 | the tunnel: that packet encrypted, in a UDP datagram to the hub's public address |
|
||||
| 2 | a frame to the gateway of whatever network the laptop is on |
|
||||
| 1 | radio |
|
||||
|
||||
**At the hub the outer envelope comes off** — frame, packet, datagram — and the tunnel decrypts what was
|
||||
inside. The inner packet is addressed to the bus, which is on this node, so the hub keeps unwrapping up
|
||||
to layer 7. The bus finds the home-server's subscription and sends the message down the home-server's
|
||||
own link, which that node opened earlier through its own tunnel. Neither spoke ever accepted a
|
||||
connection; both dialled the hub.
|
||||
|
||||
**How far the hub unwraps depends on what the packet is for.** A packet addressed to a home-server's
|
||||
private address, not to the bus, is opened only to layer 3: the hub relays it, encrypted again for the
|
||||
home-server's tunnel. That is the whole of the glossary's *a hub relays for nodes that cannot reach
|
||||
each other directly*. Two home-servers at one site skip the hub: their tunnel crosses the home network
|
||||
at layer 2, and nothing of theirs reaches the hub at all.
|
||||
|
||||
**Why a roaming node has one path is a layer 3 fact.** A more specific route to an endpoint that is
|
||||
gone takes the packets and drops them; it does not fall back (§1). A laptop that moves between networks
|
||||
therefore has exactly one route, the hub's.
|
||||
|
||||
### Reading a failure by layer
|
||||
|
||||
The machine's own network checks
|
||||
(ADR 0241)
|
||||
each look at one layer:
|
||||
|
||||
| Check | Layer | Failing alone means |
|
||||
|---|---|---|
|
||||
| **route** — a default route | 3, underlay | the machine has no way out; nothing above can work |
|
||||
| **tunnel** — a handshake with the hub within five minutes | 3 carried at 4 | the underlay works and the private network does not: a key, the hub's address, or a filter on the way |
|
||||
| **bus** — the link is open | 5 to 7 | the private network works and the bus, or the node's account on it, does not |
|
||||
| **resolv-conf** and **names** | 7 | addresses work and names do not, as when a VPN client rewrites the machine's resolver file |
|
||||
|
||||
Read them from the bottom. A layer that fails explains every failure above it, and never one below it.
|
||||
|
||||
## What this removes
|
||||
|
||||
The list is worth having in one place, because it is most of the argument:
|
||||
|
||||
@@ -54,7 +54,7 @@ rather than two kinds of thing
|
||||
node** — it is a machine with a program on it that has not been told which mesh it belongs to.
|
||||
|
||||
There is no state for *the first node*. That is the point of
|
||||
ADR 0004: the first node walks the
|
||||
ADR 0004: the first node rollouts the
|
||||
same path, in an unusual order.
|
||||
|
||||
---
|
||||
@@ -239,7 +239,7 @@ carries, to the machine it is already on.
|
||||
**After step 3 the first node is not special in any way**, which is the property `adopt.sh` and
|
||||
the bootstrap script never had. Its specialness lasted two commands.
|
||||
|
||||
**And enrolment is exercised on node one.** The path every other node depends on is walked
|
||||
**And enrolment is tried on node one.** The path every other node depends on is exercised
|
||||
immediately, against a controller on the same machine, rather than being written and first
|
||||
used months later on node two.
|
||||
|
||||
@@ -524,7 +524,7 @@ remains locally authoritative for *operating*; the copy exists only for this.
|
||||
|
||||
The node-engine is delivered like anything else
|
||||
(ADR 0010), and this is worth
|
||||
walking through because tier 0 looks like it should be special and is not.
|
||||
going through step by step because tier 0 looks like it should be special and is not.
|
||||
|
||||
```
|
||||
push to mesh-host
|
||||
|
||||
@@ -277,15 +277,15 @@ A transition the table does not hold is refused. Two or more deliveries that sha
|
||||
together as one future state of the mesh.
|
||||
|
||||
**This answers the first open question above.** A fit artifact does not declare itself. Off the trunk it is
|
||||
only checked. On the trunk it is published, and its walk across the machines starts when its delivery says
|
||||
only checked. On the trunk it is published, and its rollout across the machines starts when its delivery says
|
||||
so: by itself for the core, which cannot wait for anything to be repaired, and by `mesh-delivery` for
|
||||
everything else while that module is held. A person can always deliver by hand.
|
||||
|
||||
The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the
|
||||
record and the order, and asks.
|
||||
|
||||
**Merges are walked in batches** (2026-10-10, by ADR 0276). The controller collects
|
||||
**Merges are rolled out in batches** (2026-10-10, by ADR 0276). The controller collects
|
||||
merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as
|
||||
one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones.
|
||||
One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it
|
||||
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk.
|
||||
one rollout with at most one commit per repository: the latest of its branch, which contains the earlier ones.
|
||||
One rollout is open at a time, and a rollout that started is never superseded. `plans` shows a batch while it
|
||||
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's rollout.
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller cmd/mesh-builder
|
||||
- mesh-controller internal/builder
|
||||
- mesh-catalog modules/build-agent
|
||||
updated: 2026-10-08
|
||||
- mesh-controller modules/build-agent
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0405"
|
||||
- "ADR 0262"
|
||||
- "ADR 0233"
|
||||
- "ADR 0219"
|
||||
@@ -30,6 +31,8 @@ decisions:
|
||||
|
||||
# Building a module
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's builder (internal/builder, cmd/mesh-builder) implements the recipe kinds described here, and the sections marked built and proven live match the code. Left: not broken down into milestones; issue 1072 found it saying proposed.
|
||||
|
||||
[`12-a-module-repository`](12-a-module-repository.md) says what a module may build — an image, an
|
||||
archive, an upstream mirror — and where the result goes. This says **how a build is modelled**, and
|
||||
why the current model does not fit what a module is.
|
||||
@@ -138,7 +141,7 @@ need to run. Third-party software still arrives as an image, because that is how
|
||||
## What this costs, stated before it is chosen
|
||||
|
||||
**Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the
|
||||
event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the
|
||||
event envelope, tool serving — a build module that installs its toolchain on every builder (ADR 0405), and a bundler the builder understands. And the
|
||||
spine changes rarely but cascades when it does
|
||||
(ADR 0039): with one language a contract
|
||||
change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-sdk src
|
||||
- mesh-tools src/broker-nats.ts (and broker-amqp.ts until the rollout)
|
||||
- mesh-controller internal/link
|
||||
- mesh-controller cmd/mesh-controller (status)
|
||||
- mesh-catalog modules/postgres, modules/keycloak (the Go provisioner loop)
|
||||
updated: 2026-10-06
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0224"
|
||||
- "ADR 0230"
|
||||
@@ -24,6 +24,8 @@ decisions:
|
||||
|
||||
# The module protocol
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: the wire this design describes is in mesh-sdk and mesh-controller's internal/link, and the design's own checks against the code (2026-09-16) agree. Left: not broken down into milestones; issue 1072 found it saying proposed.
|
||||
|
||||
**What a module's code and the mesh say to each other.** An SDK is an implementation of this in one
|
||||
language and nothing more (ADR 0074).
|
||||
|
||||
|
||||
@@ -215,7 +215,7 @@ and recreate a container naming a step after the step ran.
|
||||
*Written 2026-10-02, from ADR 0189
|
||||
and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
|
||||
|
||||
Some work cannot be done underneath a running service: an artifact store's collector walks the
|
||||
Some work cannot be done underneath a running service: an artifact store's collector rollouts the
|
||||
storage and requires every writer stopped. A `run-once` step runs *beside* containers and a
|
||||
scheduled one is the same container again, so until this a module had no way to say it — and the
|
||||
mesh inherited a store that has never collected anything, because the predecessor said it with a
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-host internal/bootstrap
|
||||
- mesh-host cmd/mesh-bootstrap
|
||||
- mesh-lab test/integration/one-node-mesh.test.ts
|
||||
updated: 2026-10-06
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0226"
|
||||
- "ADR 0067"
|
||||
@@ -16,6 +16,8 @@ decisions:
|
||||
|
||||
# The installation, in full
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: the node-engine's bootstrap (`mesh-host` internal/bootstrap) carries phases 2 and 3 of the installation, so the steps this text calls manual are partly automated. Left: not broken down into milestones; the text is not yet corrected against the code.
|
||||
|
||||
Every step from a machine with nothing to a mesh that maintains itself. Written out explicitly
|
||||
because it is the procedure everything else depends on, and because the parts that do not exist
|
||||
yet are easier to see beside the parts that do.
|
||||
|
||||
@@ -6,7 +6,7 @@ code:
|
||||
- mesh-controller
|
||||
- mesh-catalog
|
||||
- mesh-lab
|
||||
updated: 2026-09-15
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0067"
|
||||
- "ADR 0074"
|
||||
@@ -17,6 +17,8 @@ decisions:
|
||||
|
||||
# The work ahead
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Phase 2's blocker (issue 053) is resolved and delivered, and the forge's module declares the package registry in full, Phase 2's done-when. Whether every phase holds is checked under issue 1073, which marks it implemented or names what is left.
|
||||
|
||||
Everything decided this cycle and not yet built, in the order its dependencies allow. Each phase
|
||||
ends at something provable on a running mesh, because a phase that ends at a claim is a phase that
|
||||
went missing without anything complaining.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-09-25
|
||||
status: in-progress
|
||||
code: [mesh-controller]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0084"
|
||||
- "ADR 0027"
|
||||
@@ -11,6 +11,8 @@ decisions:
|
||||
|
||||
# 23 — Choosing a provider
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's internal/catalogue/resolve.go implements the model described here, and designs 26, 27 and 29 rely on it. Not yet checked against this text clause by clause, nor given an as-is: issue 1074.
|
||||
|
||||
A provision is named for what the consumer's code is coupled to
|
||||
(ADR 0027):
|
||||
`postgres-database`, not `database`. That decides *what kind* of provider satisfies a requirement.
|
||||
|
||||
@@ -163,7 +163,7 @@ again restarts nothing. The provider asks when it starts, when the software's va
|
||||
consumer that binds in between holds a value the mesh made, which the software refuses, for at most the hour until the next ask. A change is recorded in the
|
||||
hand-act log as a decision, not a repair, and `rotate` of such a provision is refused. *How it is checked:*
|
||||
the tests ADR 0305 §13 names, in both repositories; the live row is `hand-acts` showing `secret issued` for
|
||||
the media server's provision once both walks are done.
|
||||
the media server's provision once both rollouts are done.
|
||||
|
||||
## Beyond generate and hold
|
||||
|
||||
|
||||
@@ -75,7 +75,7 @@ and uses, and the subjects, streams, consumers and permissions all follow from t
|
||||
(ADR 0039).
|
||||
|
||||
**Added 2026-10-10** (ADR 0289): the controller's own records join
|
||||
the bus as **record mirrors** — nodes, assignments, walks, builds and settings, each a bucket written from
|
||||
the bus as **record mirrors** — nodes, assignments, rollouts, builds and settings, each a bucket written from
|
||||
the controller's store only when the record changed, with one event per write, and audited against the
|
||||
store on every self-check run. Designed in [to-be 54](54-the-controllers-records-on-the-bus.md). The
|
||||
rule it states for every bucket, that a write is made only when what the bucket describes changed, holds
|
||||
|
||||
@@ -709,7 +709,7 @@ healthy while reacting to nothing.
|
||||
Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from
|
||||
a build argument, so the digest was never in the file the builder derived edges from, and every
|
||||
order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of
|
||||
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a
|
||||
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — traversed a
|
||||
graph with no edges. The builder now reports the bases it was handed, the controller records
|
||||
them by artifact path, and the graph is read from the newest build of each module — a recorded
|
||||
manifest carries no `build.on`, so the edge is derived from the build or it does not exist.
|
||||
|
||||
@@ -5,7 +5,7 @@ code:
|
||||
- mesh-controller internal/inventory
|
||||
- mesh-controller internal/catalogue
|
||||
- mesh-controller cmd/mesh-controller
|
||||
updated: 2026-10-02
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0181"
|
||||
- "ADR 0182"
|
||||
@@ -14,6 +14,13 @@ decisions:
|
||||
- "ADR 0051"
|
||||
- "ADR 0113"
|
||||
- "ADR 0117"
|
||||
- "ADR 0399"
|
||||
- "ADR 0400"
|
||||
- "ADR 0412"
|
||||
- "ADR 0415"
|
||||
- "ADR 0416"
|
||||
- "ADR 0417"
|
||||
- "ADR 0422"
|
||||
---
|
||||
|
||||
# 29 — A node has operator accounts, and the mesh owns what lives under a home
|
||||
@@ -56,6 +63,15 @@ These are a **family**, not one module: an `ssh-client` module, a shell module,
|
||||
module, each a *universal-tier* consumer of the account fact — assigned wherever a person logs in,
|
||||
which is every node, unlike the graphical stack that a capability gates.
|
||||
|
||||
**A folder a module owns says how it treats a file it did not write** (ADR 0416, ADR 0422). Declaring a
|
||||
directory under a home owns it (ADR 0182 §1), and the owner alone writes there and reacts to every file
|
||||
that enters it, never losing what the file held. The manifest says how, on the directory itself:
|
||||
`unknown` is `move-out` (the node-engine moves every path the manifest does not declare and `knows` does
|
||||
not name into `~/.local/state/mesh-moved/<module>/`, with a list), `module` (the module's own process
|
||||
reacts, as ssh-client does in `~/.ssh`) or `program` (everything in it is the program's, as in a
|
||||
service's data directory). `move-out` is only for a folder under a home, the home of the account the
|
||||
assignment runs for (ADR 0412 §5). A folder many modules write in is not declared at all.
|
||||
|
||||
## 3. The whole of `~/.ssh` is the mesh's — with one boundary drawn inside it
|
||||
|
||||
The predecessor owned a single file (`~/.ssh/config`) and left the rest alone; it drifted, because
|
||||
@@ -82,6 +98,9 @@ ADR 0117 draws for the uplink and the sshd
|
||||
module draws for the firewall: **the mesh must never be able to arrange the one failure that severs
|
||||
its own way back in.** The carve-out is not a convenience; it is that rule, in `~/.ssh`.
|
||||
|
||||
Since ADR 0415 and ADR 0417 the found class no longer holds in `~/.ssh`: ssh-client moves out what it
|
||||
does not know into a kept folder, and never moves a key or `authorized_keys`, which keeps this boundary.
|
||||
|
||||
## 4. Keys are the mesh's to generate — through a CA, and existing keys are adopted, not replaced
|
||||
|
||||
Key *generation* is the mesh's, not each node's improvising its own. The clean form is an **SSH
|
||||
@@ -220,6 +239,17 @@ each need a decision before the modules that replace the generators can be writt
|
||||
|
||||
The modules §2 calls *a family* — the shell, the terminal, the desktop, everything under a home that is not `~/.ssh` — are designed in [37 — The operator's machine](37-the-operators-machine.md), under ADR 0173 to ADR 0177. This document keeps `~/.ssh`, the CA and the roster files. Two things it listed as not built are decided there: user-scoped services (ADR 0177) and the one-off steps a hook used to run (declared state, or a seat's verb).
|
||||
|
||||
## The forge over SSH — 2026-10-16
|
||||
|
||||
`~/.ssh` gains the account's own key for the forge. ADR 0399 decides that each machine's account has an SSH
|
||||
key made on the machine, that the forge registers its public half on the machine's agent user, and that
|
||||
`ssh-client` writes the forge's SSH key into `known_hosts`, ssh's configuration entry for the forge and git's
|
||||
rule that rewrites the forge's `https://` address to its SSH address. ADR 0400 decides how the public half
|
||||
travels: `ssh-client` writes it to its own state, keyed by node and account, and the forge's provider reads it
|
||||
there, since a manifest's contribution cannot carry a value made at run time. The forge's SSH address is its
|
||||
machine's name on the mesh, where its SSH port is open; the key's file is `~/.ssh/mesh_ed25519`, beside the
|
||||
operator's own keys, which stay theirs (§3). Built under issue 955.
|
||||
|
||||
## References
|
||||
|
||||
- The gap was found generating `~/.ssh/config` from the *HAL* registry (`hal/terminal`'s
|
||||
@@ -236,3 +266,4 @@ The modules §2 calls *a family* — the shell, the terminal, the desktop, every
|
||||
- ADR 0117,
|
||||
ADR 0118 —
|
||||
the never-sever-the-channel rule and the found-vs-owned semantics, applied here to `~/.ssh`.
|
||||
- ADR 0399, ADR 0400 — git over SSH with a key made on each machine, and how its public half reaches the forge.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: proposed
|
||||
code: []
|
||||
updated: 2026-10-06
|
||||
status: in-progress
|
||||
code: [mesh-controller]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0238"
|
||||
- "ADR 0221"
|
||||
@@ -15,6 +15,8 @@ decisions:
|
||||
|
||||
# 30 — The mesh updates itself on a push
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's delivery plans, merge gate and merge window (plan.go, release_plan.go, merge_gate.go, merge_window_bus.go, changeplan.go, with their migrations), which design 47 builds on. Not yet checked against this text nor given an as-is: issue 1075.
|
||||
|
||||
**Today the mesh does not update itself; a person drives the pipeline by hand, and one class of
|
||||
change freezes it.** A code change lands in `mesh-controller` or `mesh-catalog`, and getting it onto
|
||||
the machines is a sequence somebody types. The predecessor's pipelines rebuilt and redeployed on a
|
||||
@@ -141,7 +143,7 @@ Revision, ADR 0218. Three rules on how a plan delivers what it built.
|
||||
A membership that could not be issued fails the send.
|
||||
- **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine,
|
||||
the first by name, and to the rest only once that machine reports the new declaration applied and
|
||||
current. A first machine that fails stops the module's walk there, with the reason in the plan.
|
||||
current. A first machine that fails stops the module's rollout there, with the reason in the plan.
|
||||
- **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository
|
||||
and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed
|
||||
by hand, by its id.
|
||||
@@ -182,7 +184,7 @@ policy or a plan is holding back
|
||||
builds are not known is left too. The push names the machine, the module, the builds and the reason,
|
||||
and says `push <node>` sends it.
|
||||
- The machine a push names, a push of every machine, and `push --behind` send held builds as before.
|
||||
Walking a change through the mesh under `record` is one `push <node>` per machine.
|
||||
Rolling out a change through the mesh under `record` is one `push <node>` per machine.
|
||||
|
||||
## Why now, and why not yet
|
||||
|
||||
|
||||
@@ -253,7 +253,7 @@ the controller creates from the catalogue, mesh-wide, existing from registration
|
||||
watch before the owner runs anywhere ([design 25](25-the-bus-on-nats.md) §3). Every instance of the
|
||||
owner writes; a reader reads and watches. A key may name a machine by the module's own convention;
|
||||
the mesh keeps one bucket per name, not one per machine, because "every server, for every machine"
|
||||
is then one list rather than a walk.
|
||||
is then one list rather than a rollout.
|
||||
|
||||
What a module sees is what it named. Its assignment's membership lists its buckets by those names,
|
||||
with whether it may write, and the runtime answers `get`, `put`, `delete`, `keys` and `watch` for
|
||||
@@ -338,7 +338,7 @@ module, version, digest. An event again: a fact about the builder.
|
||||
|
||||
**The build cascade is that event fanning out through a graph the mesh already has.** A module
|
||||
whose image is built *on* another's artifact declares that in `build.on`. So `built` reaches the
|
||||
controller, which walks the declared graph and submits rebuild jobs for everything downstream. A
|
||||
controller, which traverses the declared graph and submits rebuild jobs for everything downstream. A
|
||||
dependency cascade is not special machinery — it is one event, one derived graph, and the same job
|
||||
queue.
|
||||
|
||||
@@ -351,7 +351,7 @@ entrypoint that brings its state to the shape that version needs — the same vo
|
||||
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
|
||||
runs the module's own code, to completion, in the module's own context, and a version whose preparation
|
||||
did not succeed does not run: the step gates that module and nothing else on the machine
|
||||
(ADR 0136), and the walk stops
|
||||
(ADR 0136), and the rollout stops
|
||||
at the first machine that did not take it
|
||||
(ADR 0135, superseding
|
||||
ADR 0133).
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: [mesh-catalog, mesh-tools, mesh-controller]
|
||||
updated: 2026-10-03
|
||||
status: implemented
|
||||
code: [mesh-tools, mesh-host, mesh-controller]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0197"
|
||||
- "ADR 0195"
|
||||
- "ADR 0175"
|
||||
- "ADR 0367"
|
||||
- "ADR 0160"
|
||||
- "ADR 0159"
|
||||
- "ADR 0152"
|
||||
@@ -163,11 +165,16 @@ matched it by URL.
|
||||
|
||||
## What this does not settle
|
||||
|
||||
- Narrowing a mesh MCP server's grant per assignment. ADR 0046 makes it a setting; nothing reads one yet.
|
||||
- The mesh's own verbs on the bus. Design 33's third family; this document only says where they appear
|
||||
once they exist.
|
||||
- A person's identity behind the mesh MCP server. The mesh sees the mesh MCP server's account; design 15 keeps the
|
||||
question open.
|
||||
- **The mesh's own verbs on the bus.** Design 33's third family: these now answer directly, through
|
||||
the `mesh-controller` seat, under ADR 0154 — settled, not open. [As-is 13](../00-as-is/13-the-console.md)
|
||||
records it live.
|
||||
- **Narrowing the tool runner's grant per assignment.** ADR 0046 makes it a setting; nothing reads
|
||||
one yet. Owned by [design 58](58-agent-work-as-the-meshs-own.md) milestone 6 (issue 883): an
|
||||
identity asked for each assignment is one of that milestone's parts.
|
||||
- **A person's identity behind the mesh MCP server.** The mesh still sees only the tool runner's own
|
||||
account, never the caller's; design 15 raised the question and leaves it open. Owned by the same
|
||||
milestone (design 58 milestone 6, issue 883): its outcome is that every person, agent and module
|
||||
becomes an identity with a role, and a call carries which one made it.
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -2,8 +2,10 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-catalog modules/claude-code]
|
||||
updated: 2026-10-13
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0411"
|
||||
- "ADR 0412"
|
||||
- "ADR 0343"
|
||||
- "ADR 0274"
|
||||
- "ADR 0273"
|
||||
@@ -306,21 +308,27 @@ operator's word — and, for the operator's own home, only at the terminal or on
|
||||
|
||||
## 9. The account the agent runs as
|
||||
|
||||
*Added 2026-10-08 by ADR 0266.*
|
||||
Everywhere above, "the operator's home" and "the operator account" mean **the account the agent runs as**.
|
||||
That is the operator account, unless the node names an **agent account** of its own. The control node does.
|
||||
*Added 2026-10-08 by ADR 0266; changed 2026-10-17 by ADR 0411 and ADR 0412.*
|
||||
Everywhere above, "the operator's home" and "the operator account" mean **the account the agent runs as**:
|
||||
the machine account of the identity the module runs for (ADR 0412). claude-code declares that it can run for
|
||||
an agent identity, in its assignment or when a session is started: started for `nox` it runs with the agent's
|
||||
home; started for the operator, it is the default run, with the operator's configuration. On a
|
||||
machine where no agent is marked as the agent account, its assignment is the default run, as every module's is.
|
||||
|
||||
**The fact.** The operator states the agent account at the controller's terminal. No verb and no setting
|
||||
writes it. The controller resolves the machine facts `agent-account`, `agent-home` and `agent-root`, and gives
|
||||
the tool runner the account and its home. Where the node names none, the facts name the operator account and
|
||||
`agent-root` is empty, so nothing changes there.
|
||||
**The fact.** The machine accounts come from the identity seat (ADR 0411); the operator no longer states the
|
||||
agent account at the controller's terminal. A machine's agent account is the machine account of the agent the
|
||||
identity seat marks as the agent account there, on the operator's Approve: `nox` on the control node. There this module's
|
||||
assignment is run for that agent, and the controller refuses one that is not. The controller resolves the
|
||||
machine facts `agent-account`, `agent-home` and `agent-root` from it, and gives the tool runner the account and
|
||||
its home. Where no agent is marked (every other machine today), the facts name the operator account and `agent-root`
|
||||
is empty, so nothing changes there, though a login `nox` exists for the modules run for it.
|
||||
|
||||
**What the module declares.**
|
||||
|
||||
- A `user` named by `agent-account`, with `root` set from `agent-root`. Where the account is the agent's own,
|
||||
the mesh creates it (home, own group, no password) and the node-engine judges on every look that it cannot
|
||||
become root without a person, by the ways ADR 0266 §3 lists. It detects those ways and no others; it
|
||||
prevents nothing.
|
||||
- No login. The mesh makes the agent's machine account (home, own group, no password) from the identity, and the
|
||||
node-engine judges on every look that it cannot become root without a person, by the ways ADR 0266 §12 to §17 list.
|
||||
It detects those ways and no others; it prevents nothing. The `user` resource named by `agent-account`,
|
||||
which this module declared until ADR 0411, goes.
|
||||
- `~/.claude` under that account's home, owned by it.
|
||||
- `claude-agent`, a command that starts the agent as that account through the operator's own sudo, or
|
||||
directly when already run as it.
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-host, mesh-controller, mesh-catalog]
|
||||
updated: 2026-10-12
|
||||
updated: 2026-10-16
|
||||
decisions:
|
||||
- "ADR 0394"
|
||||
- "ADR 0310"
|
||||
- "ADR 0268"
|
||||
- "ADR 0252"
|
||||
@@ -90,6 +91,7 @@ WP5 assign and prove (operator-gated) needs WP
|
||||
WP6 a module puts the account in a group (mesh-host, mesh-controller, mesh-catalog) issue 247
|
||||
WP7 the control-node withholds execute (mesh-controller, mesh-catalog) ADR 0268
|
||||
WP8 a node's files and accounts are read (mesh-sdk, mesh-catalog, mesh-controller) ADR 0310, issue 418
|
||||
WP9 every file on a node is managed (mesh-catalog, mesh-controller) ADR 0394, issue 900
|
||||
```
|
||||
|
||||
WP1, WP2 and WP4 are independent, and are built in parallel on one feature branch per repository
|
||||
@@ -341,6 +343,10 @@ out, the control-node's value is set to `withhold` and zsh is pushed to it. Then
|
||||
|
||||
*`mesh-sdk`, `mesh-catalog`, `mesh-controller`. ADR 0310, extending ADR 0268; tracked by issue 418.*
|
||||
|
||||
**Replaced by WP9.** ADR 0394 supersedes ADR 0310: `node-files` takes over every read below, with the same
|
||||
bounds, content rule and refusal of secret places; once `node-files` is held on every node, the next delivery
|
||||
removes `node-reads` and `mesh-reads`.
|
||||
|
||||
ADR 0268 §5 has narrow read tools take the common reads over from `execute`, so that an approval is asked for an
|
||||
act and not for a read, and ADR 0286 §44 sets the approval mark on `execute` only once those reads have a tool on
|
||||
every node. Two had none: reading or searching a file, and looking up an account or a group.
|
||||
@@ -366,3 +372,38 @@ under the same issue.
|
||||
`mesh-sdk` test of the content rule and one in each of its users; `mesh_search` for `cat`, `grep` and
|
||||
`getent passwd` finding the verbs. Live, the kernel's RAID status file is read through the seat on the home
|
||||
server, and the agent's credentials file on the laptop is refused.
|
||||
|
||||
## WP9 — Every file on a node is managed through `node-files`
|
||||
|
||||
*`mesh-catalog`, `mesh-controller`. ADR 0394; tracked by issue 900.*
|
||||
|
||||
`node-reads` only reads, and as an account that cannot open a person's home, so nothing in the mesh could copy,
|
||||
move, write or delete a file, and a cleanup needed a shell on the machine. Most files belong to no module: homes,
|
||||
project folders, downloads, logs, scratch.
|
||||
|
||||
- **A node seat, `node-files`, on every node**, held by a module of its own, `mesh-files`, beside `mesh-reads`
|
||||
in the catalogue; the controller holds the seat's definition beside the other node seats. It has a verb for each file action: list,
|
||||
stat, readlink, read, search, write, copy, move, delete, make a folder, change mode, change owner, account and group
|
||||
lookup, and the trash's list and restore. Each says what it replaces (ADR 0245): `ls`, `stat`, `readlink`, `cat`,
|
||||
`grep`, `cp`, `mv`, `rm`, `mkdir`, `chmod`, `chown`, `getent passwd`, `getent group`, `id`, and `ssh <node> cp` for a
|
||||
copy between nodes.
|
||||
- **It may touch every file.** It declares no account of its own: the tool runner starts it as the node's
|
||||
operator account, which has sudo without a password, and it uses `sudo -n` only where that account's own rights
|
||||
do not reach. It keeps each file's owner and modes as well as it can: a copy or a move keeps them, a new file
|
||||
takes its folder's owner. Which files to touch is the caller's responsibility; it judges no permission.
|
||||
- **Nothing is lost at once.** A file deleted or overwritten goes to the node's trash with who asked and why, is
|
||||
restorable for 7 days, and is emptied after.
|
||||
- **A path a module owns is refused**, naming the module and its tool, unless the call says `override` with a
|
||||
reason; an override is recorded and said to the operator.
|
||||
- **It replaces `node-reads`.** Its reads keep WP8's bounds, content rule and kernel reads. Every node is
|
||||
assigned `mesh-reads`, so `node-reads` and `mesh-reads` are removed by the next delivery, once `mesh-files`
|
||||
holds `node-files` on every node.
|
||||
- **No Approve is asked**; every act is recorded with its caller. Read and search never return the content
|
||||
of a known credential file or a secret place (the mesh never prints a token); copying or moving one is
|
||||
allowed.
|
||||
|
||||
**Proof.** The module's tests: each verb against a temporary tree keeps owner and modes, a new file takes its
|
||||
folder's owner, a delete is restorable from the trash and gone after 7 days, an owned path is refused and an
|
||||
override recorded, a credential file's content is never returned. Live: the operator's `~/.claude` on a workstation is listed, a file in `/etc` is read through sudo, and a file copied from one node to another
|
||||
arrives with its owner and modes.
|
||||
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-catalog, mesh-controller, mesh-host]
|
||||
updated: 2026-10-12
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0369"
|
||||
- "ADR 0303"
|
||||
- "ADR 0215"
|
||||
- "ADR 0173"
|
||||
@@ -75,7 +76,7 @@ In order:
|
||||
| 8 | `docker` | the runtime's packages, base configuration, group | four configurations, one owner on one machine |
|
||||
| 9 | `ssh-client` | everything under `~/.ssh` (research 027/03) | a predecessor's entries winning over the mesh's; stale keys |
|
||||
| 10 | scripts | the operator's own scripts, shared and per role (research 027/03) | under no version control, copied by hand |
|
||||
| 11 | `kernel` | kernel, microcode, boot entries | two machines without microcode |
|
||||
| 11 | `kernel` | kernel, microcode, boot entries, and the kernel's run-time settings per machine (transparent huge pages first) | two machines without microcode; huge pages left at the distribution's `always` on a machine whose slow disks they flood (issue 746) |
|
||||
|
||||
`systemd`, `pacman` and `docker` hold the three seats that apply resources
|
||||
(ADR 0207):
|
||||
@@ -83,7 +84,19 @@ every module that declares a service, a package or a container depends on them b
|
||||
They go on every machine before the rest, and once they have, an unmet dependency is refused rather
|
||||
than reported.
|
||||
|
||||
`kernel` is last because a mistake in it costs a boot. `docker` holds the runtime seat
|
||||
`kernel` is last because a mistake in it costs a boot. Its run-time settings are the exception and come
|
||||
first (ADR 0369, issue 746): they change nothing at boot that the kernel does not already accept at run time, and
|
||||
each is a per-machine setting whose default is what the kernel boots with, so assigning the module changes
|
||||
no machine until that machine is given another value. The first is transparent huge pages. On the
|
||||
control-node, with `always`, the journal's memory-mapped files were held in 2 MiB pages, so each small journal
|
||||
write cost a 2 MiB write-back; the journal daemon and a log-reading intrusion-prevention service wrote
|
||||
gigabytes in minutes onto a mirrored pair of spinning disks that also holds the bus's store, and the
|
||||
controller lost its lease. That machine is set to `madvise`. The module writes the mode at boot and, on a
|
||||
change, the new value directly; it reads it back and fails its unit if it did not take, gives back the mode
|
||||
it found when unassigned, and on a running machine rotates the journal once so the open journal files are
|
||||
mapped anew in small pages. How it is checked: the module's tests run its script against
|
||||
a stand-in directory, its unit's health is the unit's state, and its tool `kernel_huge_pages` shows the
|
||||
mode in force and the file pages each unit holds in huge pages. `docker` holds the runtime seat
|
||||
(ADR 0166), with its settings declared as
|
||||
ADR 0164 decides and the capability detected as
|
||||
ADR 0165 decides; what of the three is not built yet is tracked by issue 398.
|
||||
|
||||
@@ -53,9 +53,9 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214).
|
||||
*Added 2026-10-06, ADR 0233.*
|
||||
The mesh composes a second file for the holder: every data item every module on the machine declares,
|
||||
with its class, its path, the path that covers it and its protection. Once an hour and after every
|
||||
night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at
|
||||
night the holder measures each item that is not a cache, by the item's `measure`: a bounded rollout at
|
||||
most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS
|
||||
dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large —
|
||||
dataset's own counters hourly, or the top-level entries only — never an hourly rollout of anything large —
|
||||
and reads the redundant storage it is on (a ZFS pool's health, an md array's
|
||||
members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its
|
||||
newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
|
||||
updated: 2026-10-12
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0357"
|
||||
- "ADR 0325"
|
||||
- "ADR 0323"
|
||||
- "ADR 0318"
|
||||
@@ -100,7 +101,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
|
||||
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
|
||||
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
|
||||
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
|
||||
| the record mirrors: nodes, assignments, walks, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
|
||||
| the record mirrors: nodes, assignments, rollouts, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
|
||||
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) |
|
||||
|
||||
A bus subject two components may publish on is refused when the controller composes grants, unless
|
||||
@@ -175,7 +176,7 @@ process of its own reads them back from the store.
|
||||
*Added 2026-10-08, ADR 0253:*
|
||||
**a condition says itself to the operator in plain words, beside its summary, and says whether the operator
|
||||
is needed.** The headline, needs, explanation, resolved line and actions are made where the condition is
|
||||
made: by the producer when it knows more than the subject (a walk's modules, a failed unit's name, a
|
||||
made: by the producer when it knows more than the subject (a rollout's modules, a failed unit's name, a
|
||||
hand-act's cause), otherwise by the wording registered for the kind, otherwise from the scope, which is
|
||||
reported. The explanation opens with its verdict. All of it is held to the plain rule when the controller
|
||||
takes an observation: no hash, numbered id, dotted name, duration in code, clock time, flag, markup or line
|
||||
@@ -259,25 +260,39 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
|
||||
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
|
||||
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — |
|
||||
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
|
||||
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
|
||||
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (`bus.slow-consumer`, `bus.connection-closed`) | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `connection-closed` (one per bus user), `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
|
||||
| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
|
||||
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
|
||||
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
|
||||
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
|
||||
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — |
|
||||
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — |
|
||||
| S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose walk waits | 30 min; urgent after 4 h | `waiting` (names the walk and `plans go`) | warning | — |
|
||||
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a walk out | warning | — |
|
||||
| S18 | a batch of merges is cut into its walk (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no walk is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
|
||||
| S19 | a batch waiting behind an open walk is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a walk is open | that walk's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the walk) | warning | — |
|
||||
| S16 | a rollout waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose rollout waits | 30 min; urgent after 4 h | `waiting` (names the rollout and `plans go`) | warning | — |
|
||||
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a rollout out | warning | — |
|
||||
| S18 | a batch of merges is cut into its rollout (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no rollout is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
|
||||
| S19 | a batch waiting behind an open rollout is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a rollout is open | that rollout's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the rollout) | warning | — |
|
||||
|
||||
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
|
||||
controller translates each into a condition naming the thing in the mesh's words, as the refused reply
|
||||
of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer
|
||||
is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a
|
||||
system account the controller hears a slow consumer and a refused subject for its own connection only
|
||||
([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md),
|
||||
open question 2).
|
||||
is said only for one the mesh names and expects, and clears when it exists again.
|
||||
|
||||
**Every user's slow consumer, from the bus module (ADR 0357).** The server closes every client it judges
|
||||
a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither
|
||||
in the mesh's account. So the `nats` module's tool bundle reads that record on its own machine every 30
|
||||
seconds and publishes `bus.slow-consumer` for each connection closed as a slow consumer, and
|
||||
`bus.connection-closed` for each the server closed for something its client did against the bus's
|
||||
rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by
|
||||
the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a
|
||||
loop. A closure — its user, connection id and opening time — is said at least once: again when publishing
|
||||
it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last
|
||||
hour of both events back from the events stream on every tick, count each closure once, and raise one
|
||||
condition per bus user and kind, naming the module and its machine; it clears an hour after the newest
|
||||
closure. A closure is a condition at most about a minute after the server closed it.
|
||||
|
||||
**A refused subject is still heard for the controller's own connection only.** The server tells the
|
||||
client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system
|
||||
account (issue 702, open).
|
||||
|
||||
## 4. The self-check: `doctor` (rule 6)
|
||||
|
||||
@@ -525,7 +540,7 @@ except an act recorded by a verb that is a person's decision by design, which is
|
||||
counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and
|
||||
each says whether it records a repair or a decision, and why: approving or rejecting a retirement and
|
||||
deleting what was retired (ADR 0230); `bus upgrade`, because the bus is never rolled by the mesh, and
|
||||
`upgrade release-backlog`, because after a failed walk the next opens only on a person's word
|
||||
`upgrade release-backlog`, because after a failed rollout the next opens only on a person's word
|
||||
(ADR 0236); and `secret rotate` when its cause is a leak (`leaked-in-logs`) — a person judges what was
|
||||
disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that
|
||||
prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A
|
||||
@@ -604,7 +619,7 @@ the other machines follow. "Reported applied" is not enough.
|
||||
|
||||
**As built** (ADR 0236):
|
||||
|
||||
- **Every walk is gated, not only the core's.** A module is judged by its own health: it reported
|
||||
- **Every rollout is gated, not only the core's.** A module is judged by its own health: it reported
|
||||
applied, no witness put it back, no condition was raised since its send about the machine or about the
|
||||
module there, and its tools are served on that machine where it has tools. Three healthy judgings at
|
||||
least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of
|
||||
@@ -627,7 +642,7 @@ the other machines follow. "Reported applied" is not enough.
|
||||
- **No build reaches a machine without a gate**: a gated send carries and judges everything waiting on
|
||||
its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused
|
||||
or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and
|
||||
manifest is no move. What waits is moved by a **walk**, one machine at a time, the control
|
||||
manifest is no move. What waits is moved by a **rollout**, one machine at a time, the control
|
||||
node last, each judged; one that fails holds the next until `upgrade release-backlog --why`.
|
||||
- **A wait for a person passes, and each module of a send keeps its own pass**
|
||||
(ADR 0254): a module whose only fault is a new login the operator owes
|
||||
@@ -787,10 +802,10 @@ has a week of entries; the durations are recorded for every machine.
|
||||
|
||||
| Repository | Delivers |
|
||||
|---|---|
|
||||
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
|
||||
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
|
||||
| `mesh-host` | the heartbeat carries its interval; D1's validator published as a package the controller imports |
|
||||
| mesh-tools | the tool runner's heartbeat (S11) |
|
||||
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) |
|
||||
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5); the `nats` module's watch of its server's closed connections, publishing `bus.slow-consumer` and `bus.connection-closed` (S9, ADR 0357) |
|
||||
| mesh-lab | R9: each signal suppressed in turn |
|
||||
|
||||
**Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a
|
||||
@@ -923,7 +938,7 @@ and "the commit is the build at hand"), on branches, not yet merged:
|
||||
| hq | its own `merge-check.sh`, running `records.py`, `index.py` and `cycle.py` |
|
||||
|
||||
**Not yet:** the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the
|
||||
state machine replacing the walks' states, with its table, its test, its conditions and healer H2;
|
||||
state machine replacing the rollouts' states, with its table, its test, its conditions and healer H2;
|
||||
the status's link to the kept plan; the notes under `refs/notes/mesh-plan`; check builds in a scratch
|
||||
namespace (the check builds no module today, so there is none to keep apart yet); the statuses made
|
||||
required — prepared as calls of the forge module's tool, applied by the operator.
|
||||
|
||||
@@ -327,36 +327,36 @@ Presence facts are events on the bus. The router keeps them as **current state o
|
||||
machine and per intake kind, overwritten — never as a history, and never in a message's words. A
|
||||
consumer that wants a timeline of the operator's day is a decision of its own.
|
||||
|
||||
### While a delivery walks
|
||||
### While a delivery rollouts
|
||||
|
||||
*2026-10-12, ADR 0308; the build is tracked by issue 532.*
|
||||
|
||||
A delivery restarts what it moves, and those restarts raise conditions that clear once it is done; a router
|
||||
that restarted used to say again what it had been told to keep quiet (issue 531). So the router holds what a
|
||||
walk touches and tells the walk's result once.
|
||||
rollout touches and tells the rollout's result once.
|
||||
|
||||
- **A walk names what it touches.** The controller's walk events say, when a walk starts and again when it
|
||||
- **A rollout names what it touches.** The controller's rollout events say, when a rollout starts and again when it
|
||||
ends with its outcome (delivered, failed or stopped), the modules it moves and the machines it sends them to.
|
||||
What it touches is those modules on those machines, and every module a moved tool runner or node-engine
|
||||
carries there.
|
||||
- **The router holds what an open walk touches.** A message whose subject is a touched module or machine is
|
||||
- **The router holds what an open rollout touches.** A message whose subject is a touched module or machine is
|
||||
held instead of sent, kept with when it was held, and `history` says it was held. A message several open
|
||||
walks touch is held until the last of them ends or its bound passes.
|
||||
rollouts touch is held until the last of them ends or its bound passes.
|
||||
- **Some messages are never held:** an ask (§6), a secret given or replaced, a message about anything no open
|
||||
walk touches, and a walk's own failure.
|
||||
- **A walk's end is told once:** one message with its outcome, the modules and machines it reached and how
|
||||
rollout touches, and a rollout's own failure.
|
||||
- **A rollout's end is told once:** one message with its outcome, the modules and machines it reached and how
|
||||
long it took, then each held message whose condition is still open, one line each. One whose condition
|
||||
cleared during the walk is only counted. A walk delivered with nothing held and nothing open is one line, or
|
||||
cleared during the rollout is only counted. A rollout delivered with nothing held and nothing open is one line, or
|
||||
nothing on a channel set to say only what needs the operator.
|
||||
- **A hold has a bound:** twice the largest delivery budget in force among the walk's deliveries
|
||||
- **A hold has a bound:** twice the largest delivery budget in force among the rollout's deliveries
|
||||
([to-be 47](47-delivery-from-commit-to-delivered.md)): 20 minutes for a core module, 10 for leaf modules at
|
||||
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a walk none of whose
|
||||
deliveries has a budget. Past it, every held message still open is sent with a line saying the walk still
|
||||
runs and since when, and the cleared ones are counted. A walk that hangs never silences the mesh.
|
||||
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a rollout none of whose
|
||||
deliveries has a budget. Past it, every held message still open is sent with a line saying the rollout still
|
||||
runs and since when, and the cleared ones are counted. A rollout that hangs never silences the mesh.
|
||||
- **A restart says nothing it has said:** a router that restarts says again only what is open and was never
|
||||
answered, never a silenced message or one withheld for its content.
|
||||
|
||||
*How it is checked:* the controller's tests of the walk events, and the router's tests of holding, never
|
||||
*How it is checked:* the controller's tests of the rollout events, and the router's tests of holding, never
|
||||
holding, the end message, the bound and the restart, as ADR 0308 lists them; live, the next delivery of the
|
||||
tool runner gives the operator one message at its end, and the channel's `history` shows what was held and
|
||||
released.
|
||||
@@ -467,13 +467,13 @@ the factors are the router's state.
|
||||
### An approved settings change takes effect (ADR 0344)
|
||||
|
||||
*2026-10-13, issue 641.* Approving a settings proposal is the act that makes it take effect. The
|
||||
serving controller sets the layer and, in the same act, opens a **settings walk**: every machine
|
||||
serving controller sets the layer and, in the same act, opens a **settings rollout**: every machine
|
||||
that runs the module and that the layer reaches, the control node last, each sent and its module
|
||||
judged at the first-node gate before the next. A settings walk sends the layer and no build: every
|
||||
judged at the first-node gate before the next. A settings rollout sends the layer and no build: every
|
||||
module on the machine stays at the build it was last sent, so a build held for a person, or waiting
|
||||
for a first-node gate, stays where it is. A send that would move a running module's data stops the
|
||||
walk and names the `push --move` that sends it. The walk says its start and end like every walk, so
|
||||
its end is told as every walk's is. A failure stops the walk and leaves the layer as approved; the
|
||||
rollout and names the `push --move` that sends it. The rollout says its start and end like every rollout, so
|
||||
its end is told as every rollout's is. A failure stops the rollout and leaves the layer as approved; the
|
||||
previous layer is in the settings history. The ask says all this before the Approve, and no longer
|
||||
names a push. A layer set by hand still waits for a push.
|
||||
|
||||
@@ -481,9 +481,9 @@ A delivered file is not yet an applied one: a module whose program reads its set
|
||||
starts applies them itself (a service's `restart-on`, or a tool of its own). Betterbird is such a
|
||||
module; issue 643 makes it restart Betterbird at a quiet moment.
|
||||
|
||||
How it is checked: the controller's tests open the walk on the warrant, send one machine at a time
|
||||
How it is checked: the controller's tests open the rollout on the warrant, send one machine at a time
|
||||
with every build kept and the module judged, stop before a data move and on a failure without
|
||||
holding other walks back, and read the ask's words.
|
||||
holding other rollouts back, and read the ask's words.
|
||||
|
||||
### Proofs and tiers
|
||||
|
||||
|
||||
@@ -2,62 +2,122 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-controller, mesh-catalog modules/mesh-delivery]
|
||||
updated: 2026-10-13
|
||||
updated: 2026-10-18
|
||||
decisions:
|
||||
- "ADR 0426"
|
||||
- "ADR 0421"
|
||||
- "ADR 0420"
|
||||
- "ADR 0419"
|
||||
- "ADR 0418"
|
||||
- "ADR 0408"
|
||||
- "ADR 0407"
|
||||
- "ADR 0406"
|
||||
- "ADR 0405"
|
||||
- "ADR 0404"
|
||||
- "ADR 0403"
|
||||
- "ADR 0402"
|
||||
- "ADR 0401"
|
||||
- "ADR 0398"
|
||||
- "ADR 0395"
|
||||
- "ADR 0382"
|
||||
- "ADR 0377"
|
||||
- "ADR 0364"
|
||||
- "ADR 0362"
|
||||
- "ADR 0360"
|
||||
- "ADR 0350"
|
||||
- "ADR 0323"
|
||||
- "ADR 0313"
|
||||
- "ADR 0306"
|
||||
- "ADR 0304"
|
||||
- "ADR 0299"
|
||||
- "ADR 0298"
|
||||
- "ADR 0282"
|
||||
- "ADR 0276"
|
||||
- "ADR 0270"
|
||||
- "ADR 0267"
|
||||
- "ADR 0250"
|
||||
- "ADR 0249"
|
||||
- "ADR 0239"
|
||||
- "ADR 0238"
|
||||
- "ADR 0236"
|
||||
- "ADR 0233"
|
||||
- "ADR 0214"
|
||||
- "ADR 0201"
|
||||
---
|
||||
|
||||
# 47 — Delivery: from a commit to delivered
|
||||
|
||||
**A delivery is one commit in one repository, from the moment its pull request's head is announced to the
|
||||
moment every machine of its deploy plan runs it, or to the moment it fails, is superseded or is stopped. A
|
||||
delivery group is two or more deliveries that share a head branch name, delivered as one unit in order. Both
|
||||
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
|
||||
(ADR 0239.)
|
||||
**A change reaches the machines in one of two ways, both owned by mesh-delivery: a hotfix (one pull request,
|
||||
delivered at its merge) or a feature (one branch name across its repositories: in two or more, one delivery
|
||||
group merged in one pass on the operator's one Approve and delivered by one delivery plan; in one, a lone
|
||||
delivery). A design's milestones are features delivered one after another. The trunk is what runs; there is no
|
||||
"release branch"** (ADR 0401, ADR 0418 §1, ADR 0421 §2).
|
||||
|
||||
**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a
|
||||
rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk
|
||||
at a time. Every delivery the walk answers ends with it
|
||||
(ADR 0276).
|
||||
**Every step starts on an event, and the state lives on the bus.** The forge's events, the controller's
|
||||
events and the nodes' reports move a delivery; the bus's own rules keep the queues, the merge window and the
|
||||
order of a plan's tiers, which only the module graph gives (ADR 0419); and one chain, from module to commit to build to rollout to assignment, is kept in the bus's
|
||||
key-value store as its only source (ADR 0406, ADR 0407, ADR 0408). A running delivery never touches merging:
|
||||
every merge is recorded at once for the next delivery plan of what it touches.
|
||||
|
||||
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
|
||||
of source and artifacts decides, and nothing here changes that. This design says who answers for one
|
||||
commit's journey and in what order the steps are allowed.
|
||||
commit's journey, in what order the steps are allowed, and where each fact of it is kept. It was rewritten on
|
||||
2026-10-16 from the decisions of research 055, which replaced the "release branches", the toolchain image, the
|
||||
controller's delivery tables and the timers that polled them.
|
||||
|
||||
## The parts
|
||||
|
||||
```
|
||||
forge's holder ──pull.updated / .merged / .closed──► mesh-delivery ◄──checked / plan-moved── controller
|
||||
(statuses, the view, (deliveries, groups, (planner, gate,
|
||||
the notes) the table, its state) registration,
|
||||
▲ │ │ sending, walk)
|
||||
└── its tools: note, view, status ◄───────────────┘ └── its verbs: delivery-plan, -order,
|
||||
-check, deliver, delivery-stop,
|
||||
delivery-walks
|
||||
forge (system webhook) ──forge.pull-request.* / merged / status──► the bus ◄──built / checked / plan-moved── controller
|
||||
│ ▲ (planner, order,
|
||||
│ │ registration,
|
||||
▼ │ sending)
|
||||
mesh-delivery │
|
||||
(deliveries, groups, a key per │
|
||||
pull request, the state table) ▼
|
||||
node-engine on every node
|
||||
builders (build seat) ◄── work queues: delivery builds, approve-asked checks, other checks ───────────────── (reports; backups first)
|
||||
```
|
||||
|
||||
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
|
||||
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
|
||||
rule, a gated send, the rollback and the walk of one trunk commit's tiers across machines. It still answers
|
||||
every pull request's head with its check itself, so a check never depends on mesh-delivery.
|
||||
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
|
||||
group, the state table and their state, and nothing else.
|
||||
- **The forge's holder** sets a commit's statuses, keeps the delivery's view as one comment on the pull
|
||||
request, and appends to the commit's note under `refs/notes/mesh-plan`.
|
||||
- **The build seat** builds and checks, asked by the controller as before.
|
||||
- **mesh-delivery** (the seat `mesh-delivery`) owns hotfixes, features and milestones as deliveries and delivery
|
||||
groups, decides each step on events, keeps one key per pull request, and asks the controller for each act
|
||||
and the nodes' backup seats for the backups a delivery needs. It never sends to a machine, writes a
|
||||
declaration or registers a build (ADR 0239 §25).
|
||||
- **The controller** keeps planning, ordering, registration, sending and judging. It writes the chain's
|
||||
links it owns — builds, rollouts with their tiers, assignments — and the merge window's
|
||||
keys, all on the bus, by compare-and-set, while it holds the lease.
|
||||
- **The builders**, the holders of the build seat, build and check with toolchains installed on their node, as
|
||||
the build seat's own account, and take their jobs from the bus's work queues, delivery builds first
|
||||
(ADR 0405, ADR 0406).
|
||||
- **The forge module** (`gitea`) publishes every forge event on the bus through the forge's system webhook
|
||||
(issue 729): pull requests updated, synchronized, merged and closed, and commit statuses; no label or comment
|
||||
moves a delivery, and mesh-delivery subscribes to none (ADR 0421 §11). It sets statuses, keeps the delivery's view as one comment on the pull request, and appends
|
||||
to the commit's note under `refs/notes/mesh-plan`.
|
||||
- **Each node's backup seat** (`node-backup`) backs up the data a delivery plan moves there before the plan
|
||||
sends anything (ADR 0404).
|
||||
|
||||
## Two ways a change reaches the machines
|
||||
|
||||
*ADR 0401.*
|
||||
|
||||
| Way | What it is | How it is delivered |
|
||||
|---|---|---|
|
||||
| **hotfix** | one pull request in one repository | at its merge, as a lone delivery; a lone merge is cut at once (ADR 0282 §2, as ADR 0304 §1 changed it) |
|
||||
| **feature** | one branch name across its repositories; in two or more, its pull requests are one delivery group; in one, a single pull request | a group is merged in one pass once its composed check passed at every member's head and the operator gave the group's one Approve (ADR 0418 §4, ADR 0421 §2); its merges are one batch and one delivery plan, each module built once and sent once (ADR 0418 §1); in one repository, as a lone delivery like a hotfix |
|
||||
|
||||
- **The trunk is what runs.** Every delivery is cut from a repository's trunk. A merge into any other branch
|
||||
starts no delivery, and the controller says so for that merge (issue 889). Holding merged work back, on a
|
||||
branch beside the trunk or per module, was dropped: from 2026-10-13 23:52 to 2026-10-16 09:32 feature 57's
|
||||
branches held the controller for 55.7 h, and 28 of 68 other merges touched a held module (research 055,
|
||||
finding 3).
|
||||
- **A design's milestones are features, one after another.** Each milestone is delivered as a feature when its
|
||||
own group is ready; the next starts when the previous one's delivery plan is done (playbook 04).
|
||||
- **A milestone is functional, its delivery shaped technically.** A milestone is named by what the operator or
|
||||
the mesh can do once it is delivered; how it ships is shaped by the delivery plan its changes would have, asked of
|
||||
the controller's planner through mesh-delivery's `what-if` (ADR 0239 §3): the modules it moves and their fan-out.
|
||||
- **What a merge waits for.** After its checks, a merge waits for one act of a person: the operator's Approve,
|
||||
asked as soon as both required checks passed at the head (ADR 0421 §1, §3). For a group the Approve is asked
|
||||
once, when every member's checks and the group's composed check passed, and covers every member's head
|
||||
(ADR 0421 §2); it is the review ADR 0382 §6 asks (ADR 0421 §9). No ready mark, label, comment, forge review or
|
||||
recheck stands before it. The operator's Approve is asked for every merge into main, a hotfix's too
|
||||
(ADR 0398 §3; the operator, 2026-10-16 23:40).
|
||||
|
||||
## A delivery
|
||||
|
||||
@@ -66,12 +126,21 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
|
||||
- the pull request (number, base, head branch, description), or none for a commit that reached the trunk
|
||||
without one;
|
||||
- its **delivery plan**: build plan, deploy plan and verdict, as the controller's planner computed them for
|
||||
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
|
||||
its diffset (ADR 0238 §2, as ADR 0323 §1 changed it);
|
||||
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
|
||||
**machine steps** while delivering;
|
||||
- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch
|
||||
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
|
||||
window and the batch*).
|
||||
- its **Approve**: the ask, the operator's answer with its warrant, and the review record the pull request was
|
||||
opened with (ADR 0421 §5, §8);
|
||||
- once merged, **the commit it landed on the trunk as**, and the delivery plan that delivers it: the plan of the
|
||||
batch that answers its merge, which may carry a later commit of its repository that contains it. Every member
|
||||
of a delivery group is delivered by the same plan, and its delivery ends with that plan's result
|
||||
(ADR 0418 §1).
|
||||
|
||||
**One key per pull request.** Beside the delivery, mesh-delivery keeps one key per open pull request on the
|
||||
bus: its head, the trunk commit its current check is asked against, and its verdict, written by
|
||||
compare-and-set (ADR 0406 §4). It writes a new head when the forge says one, and a new trunk commit only when it
|
||||
asks the pull request's check again (ADR 0395 §1). A running check watches that key and stops when the head or
|
||||
the trunk commit in it moves.
|
||||
|
||||
## The state table
|
||||
|
||||
@@ -84,372 +153,422 @@ allows.
|
||||
|---|---|---|---|
|
||||
| proposed | a head was announced and is being checked | 1 hour from when its check is asked; none while its verdict is out of date and no check is asked | nothing: the check's own watchdog (S6) speaks |
|
||||
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
|
||||
| ready | it passed; it waits to be merged | none | — |
|
||||
| ready | it passed; its Approve is asked (a group member's once the group is ready), and it waits for it and the merge | none | — |
|
||||
| rejected | it failed its check | none | — |
|
||||
| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started |
|
||||
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
||||
| delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says |
|
||||
| published | merged on its trunk, recorded in a batch; it waits for its plan to start | 30 minutes | `superseded` when a plan that does not answer its merge took over its plan; `go` when its plan started |
|
||||
| held | it waits for a person: a failed backup (ADR 0404 §4), a policy of record; it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
||||
| delivering | its plan runs: backups, then builds asked and sent tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the plan's key says |
|
||||
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
|
||||
|
||||
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
|
||||
at the moment the build seat clones it, the **trunk commit**, and the verdict names both. When another merge
|
||||
moves the trunk, a `ready` delivery goes back to `proposed` on the observed event `trunk moved`, with both
|
||||
commits in the transition's reason; a `rejected` one stays `rejected`, its statuses saying the verdict is out
|
||||
of date. Nothing is asked for it: a recheck, a new head or the merge tool asks the check again. The row from
|
||||
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
|
||||
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
|
||||
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own walk.
|
||||
when the builder clones it, and the verdict names both. When another merge moves the trunk, a `ready`
|
||||
delivery goes back to `proposed` on the event `trunk moved`, and mesh-delivery asks the check of every active,
|
||||
cleanly merging open pull request into that trunk at once (ADR 0395, *A move of the trunk asks every check
|
||||
again* below). An Approve already asked stays; the merge waits for the fresh check, and the Approve is not asked
|
||||
again for a trunk move (ADR 0421 §1, §3). The row from `proposed` to `checked` takes only a verdict that names the trunk's head.
|
||||
|
||||
What this leaves as it is, said so nobody reads more into it:
|
||||
**What needs no person ends by itself** (ADR 0270): a merge that moves nothing is `delivered` at its merge; a
|
||||
merge a later delivered merge contains is `covered`, by the forge's word that the later merge commit contains
|
||||
it; a merge whose plan said its modules unchanged is `delivered` when that plan ends (issue 611); a stop whose
|
||||
plan ended without building its modules is read as covered once a later merge covers it (issues 474, 480).
|
||||
|
||||
- **mesh-delivery's trunk head is the last merge it heard** into that trunk while the delivery was open. A push
|
||||
to the trunk that is not a merge it hears does not move it; the forge's holder, which reads the branch, and
|
||||
the merge tool, which compares commits when asked, still catch it.
|
||||
- **A verdict that names no trunk commit**, from a build seat older than ADR 0323, counts in mesh-delivery as it
|
||||
did: such a delivery stays `ready`. The forge reads the same verdict out of date once its trunk moves, and the
|
||||
merge tool refuses it and asks the check. The two agree once the head is judged again.
|
||||
- **A verdict no build judges** (a change that touches no module, in a repository with no `merge-check.sh` or
|
||||
not the mesh's) names the base branch's head the forge announced with the head as its trunk commit.
|
||||
- **In a busy repository the merge tool can refuse, ask, and refuse again**: another merge may move the trunk
|
||||
while the asked check runs, and the new verdict is then out of date too. Each refusal asks once more; the
|
||||
merge goes through once a check finishes before the next merge.
|
||||
- **A check that was asked and never answers** is listed by `stalled` an hour after the ask, and is asked again
|
||||
with `mesh-delivery.recheck`, which the merge tool's refusal names.
|
||||
**History is retired, not held** (ADR 0250). A delivery made from the forge's word at its merge, heard fifteen
|
||||
minutes or more after it, with no plan of it open, goes to `retired` at adoption. **Nothing of a published
|
||||
delivery is registered before its plan starts.**
|
||||
|
||||
The transitions are those of ADR 0239 decision 2, ADR 0270 and ADR 0323. A delivery that is
|
||||
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
|
||||
delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by
|
||||
`stalled` at once. Nothing waits silently.
|
||||
**The machine steps.** While a delivery is `delivering`, each machine its plan reaches has a row with the
|
||||
module and the build: `backed-up`, `sent`, `applied` or `failed`, and `silent` when no report came within
|
||||
ApplySilentAfter (fifteen minutes). They are read from the plan's key, written by the controller from the
|
||||
nodes' reports, never inferred from time.
|
||||
|
||||
**What needs no person ends by itself** (ADR 0270):
|
||||
## Order without a person
|
||||
|
||||
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
|
||||
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
|
||||
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
|
||||
running it by the walk of a later merge into the same trunk, delivered on its own passing check, and the
|
||||
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
|
||||
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
|
||||
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
|
||||
machine only is covered too: its walk records the empty rest as sent once its first machine passes
|
||||
(ADR 0270, as corrected 2026-10-12);
|
||||
- *a merge whose walk said its modules unchanged* moves nothing too, and is `delivered` when that walk ends,
|
||||
whichever commit the walk carried (issue 611). A delivery plan made while another walk built a module reads
|
||||
that module whole and names it; the walk, cut later, reads the build source that build recorded (ADR 0267) and
|
||||
may find none of the changed files in it. The walk's record then says so in `delivery.unchanged`: each such
|
||||
module with why, "its build source holds none of the changed files", and not in its modules, which keep
|
||||
meaning what the walk builds. A delivery whose every module the walk built or said unchanged is delivered,
|
||||
saying which matched what runs. One with a module the walk neither built nor said unchanged is read as below.
|
||||
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
|
||||
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
|
||||
- *a stop whose walk ended without building its modules* (issue 474) stays `stopped`, a final state, and
|
||||
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
|
||||
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
|
||||
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
|
||||
*ADR 0419, ADR 0418 §2.*
|
||||
|
||||
**History is retired, not held** (ADR 0250).
|
||||
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk
|
||||
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
|
||||
to `retired` at adoption, before any check, hold or walk, and its pull request is not told. A held delivery that
|
||||
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its walk
|
||||
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
|
||||
build registered before its turn would be carried by somebody else's send.
|
||||
A delivery group has no order among its members, and nobody writes one (ADR 0419 §1). Inside a plan only the
|
||||
module graph orders: the planner orders the build and deploy tiers by edges between modules, never between
|
||||
repositories or members (ADR 0418 §2, ADR 0419 §3). Modules with no edge between them share a tier.
|
||||
|
||||
**The machine steps.** While a delivery is `delivering`, each machine its walk reaches has a row with the
|
||||
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
|
||||
readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the
|
||||
controller's `plan-moved` event, never inferred from time.
|
||||
| Edge | A module goes before | Read from |
|
||||
|---|---|---|
|
||||
| *built by* and *stands on* | the modules built by it or standing on it | the module graph |
|
||||
| *SDK before its users* | every module built on the SDK, `mesh-sdk` | the module graph |
|
||||
| *verb before caller* | a module whose change adds to its invoked verbs a verb this module's change adds to a seat's served verbs | the manifests |
|
||||
| *version skew* | (the controller, when the plan moves it) every module whose manifest the plan changes | the change |
|
||||
|
||||
- **Engine before controller is the one default.** When a plan moves both the node-engine and the controller and
|
||||
no edge orders them, the node-engine's tier comes first; an edge either way wins over it (ADR 0419 §4).
|
||||
- **Each tier says the edge that placed it.** The plan, `delivery-order`, `show` and the plan's note name, for
|
||||
each module placed after another, the edge and its rule, and where the default was overruled, the edge that
|
||||
overruled it (ADR 0419 §5).
|
||||
- **A cycle is refused before anything merges.** Edges that form a cycle fail the group's composed check, or a
|
||||
lone pull request's check, naming every module and edge in it. The way out is in the change itself (a verb
|
||||
promised optional before it is required, ADR 0246), never an order typed by a person (ADR 0419 §6).
|
||||
- **No `after:` line is read.** mesh-delivery reads no `after:` line, no refusal asks for one, and a description
|
||||
that holds one is told once on its pull request that the line is not read (ADR 0419 §2).
|
||||
|
||||
**The SDK by version.** A Go module names the SDK by its released version and fetches it from the mesh's forge,
|
||||
checked against its recorded checksum; vendored copies go. Inside a group, a member may name the SDK member's
|
||||
head: its merge check resolves that commit from the forge and builds against it, and the head stays fetchable
|
||||
after the group merges because the forge keeps every pull request's head. The catalogue's check accepts an SDK
|
||||
commit on the SDK's trunk or the head of an SDK pull request the forge holds, open or merged (ADR 0402 §2 to §4).
|
||||
|
||||
## Checks
|
||||
|
||||
### One check per pull request, kept by the bus
|
||||
|
||||
*ADR 0323, ADR 0350, ADR 0395, ADR 0406.*
|
||||
|
||||
- **Three work queues, taken in order.** Delivery builds go on a work-queue stream of their own; checks of pull
|
||||
requests whose Approve is asked or given on a second (ADR 0421 §10); every other check on a third. A builder takes from the first, then the
|
||||
second, then the third. Each check stream has one subject per pull request and keeps only the newest request:
|
||||
a newer request replaces one still waiting, by the stream's rule. The fresh check of such a pull request, asked
|
||||
on the merge path, is published on the second stream; a request left behind on the third for a head and trunk commit already
|
||||
judged or running is acknowledged and dropped when taken, by reading the pull request's key. Nothing pauses the
|
||||
worker or deletes and republishes a request (issue 558).
|
||||
- **A running check stops itself.** It watches its pull request's key and stops when the head or the trunk
|
||||
commit in it moves, without asking the forge (issue 726). An ended check delivers no verdict, so none of its
|
||||
builds is reused.
|
||||
- **A failed builder loses nothing.** A job a builder took and never acknowledged goes to another builder
|
||||
after its wait.
|
||||
- **A change is built once** (ADR 0350). A check that passes also builds what its merge would move into the
|
||||
artifact store's scratch namespace under a reuse key, and the plan of the merge commit copies the build of the
|
||||
same key, named by the check's verdict, instead of building.
|
||||
|
||||
### A move of the trunk asks every check again
|
||||
|
||||
*ADR 0395.*
|
||||
|
||||
- A trunk move asks the check of every active open pull request into that trunk that merges cleanly, against the
|
||||
new head; a conflict is told once on the pull request, not checked.
|
||||
- A pull request with no push for 7 days and no Approve asked or given is left out; its statuses say out of date
|
||||
and that a push or a recheck asks its check (ADR 0421 §10).
|
||||
- For a group, a move asks each member's own check, and the composed check runs again through those checks; the
|
||||
group's own merges, made in one pass on its Approve, ask its members nothing again (ADR 0418 §4).
|
||||
- Its cost is at most one check per active, cleanly merging pull request per trunk move; issue 944 measures it.
|
||||
|
||||
### A delivery group's checks
|
||||
|
||||
*ADR 0239 §14, ADR 0323 §6, ADR 0382, ADR 0403, ADR 0418 §4, ADR 0421 §2, §9.*
|
||||
|
||||
- **The composed check is each member's gate.** The group's composed check composes every machine with all
|
||||
members' heads, each merged into its own trunk's head first, judged by the controller built from the group's
|
||||
controller member when it has one (ADR 0382 §3). Its verdict sets `mesh/delivery-group` and each member's
|
||||
`mesh/merge-gate`, and names every member it read. No gate runs for a member alone.
|
||||
- **Each member's own repository check stays required.** Each member's `mesh/repo-check` runs its repository's
|
||||
suite with the siblings read beside it (ADR 0382 §2).
|
||||
- **No composed verdict, no gate.** A member's gate is pending while no composed verdict at the members' current
|
||||
heads exists. A member's new head, close or base move puts the composed verdict out of date, and with it every
|
||||
member's gate. When mesh-delivery cannot say the groups, the pull request is checked alone and its verdict
|
||||
says so (ADR 0382 §1).
|
||||
- **A group merges in one pass, with no order.** Once the composed check passed at every member's head and the
|
||||
operator's one Approve covers those heads, mesh-delivery merges every member in one pass, no member waiting for
|
||||
another's merge or delivery (ADR 0418 §4). The Approve is the review ADR 0382 §6 asks: the merge tool reads its
|
||||
warrant for those heads in place of an approving forge review (ADR 0421 §9). A member the forge refuses is
|
||||
merged again on its fresh check while the group's batch stays open for it.
|
||||
|
||||
## The Approve and the merge
|
||||
|
||||
*ADR 0421, ADR 0398 §3, ADR 0406 §1.*
|
||||
|
||||
- **Asked as soon as the checks pass.** When a pull request's delivery reaches `ready` (both `mesh/merge-gate` and
|
||||
`mesh/repo-check` passed at its head against its trunk's head), mesh-delivery asks the operator's Approve of its
|
||||
merge into main at once, on a channel that proves who answered (ADR 0274; Telegram today). The ask names the
|
||||
pull request, its issues, the reviewer's verdict, the delivery plan and the trunk commit the checks judged. A
|
||||
verdict that names a trunk commit the trunk had already left raises no ask: mesh-delivery asks that head's check
|
||||
against the trunk's head, and its passing verdict raises the ask (ADR 0421 §1).
|
||||
- **A group is asked once.** One ask names every member and its head, once every member is `ready` and the
|
||||
composed check passed at those heads; one Approve covers them all (ADR 0421 §2).
|
||||
- **The reviewer's verdict is attached, not a gate.** The review finishes before the push (playbook 07), and the
|
||||
session that opens a pull request gives its review record: who reviewed it, the commit the review approved and
|
||||
the verdict. The ask carries it; with none, or one naming another commit, the ask says so in its first line and
|
||||
the operator decides (ADR 0421 §5).
|
||||
- **The merge.** mesh-delivery merges through the forge's merge tool at the heads the Approve names, only while
|
||||
they are still the heads and both checks passed against the trunk's head. When the trunk moved, the merge tool
|
||||
asks the fresh check, and the event of its passing verdict asks the merge again; no timer asks it, and no person
|
||||
rechecks (ADR 0421 §3, §12).
|
||||
- **A push or a Decline ends the Approve.** A new head ends its ask or its Approve, for a group every member's, and
|
||||
is said once on the pull request; the new head is asked again once its checks pass. A Decline merges nothing and
|
||||
is said once (ADR 0421 §6).
|
||||
- **A failure is said, never merged past.** An approved head whose check failed is said once on the pull request
|
||||
and listed by `stalled`; so is a merge the merge tool kept refusing for an hour (ADR 0421 §7).
|
||||
- **Recorded.** The ask, the answer with its warrant and the review record are kept on the delivery, shown by
|
||||
`show` and said in mesh-delivery's log; the merge's transition names the Approve (ADR 0421 §8).
|
||||
|
||||
## The merge window and the batch
|
||||
|
||||
*2026-10-10, by ADR 0276.*
|
||||
*ADR 0276, ADR 0282, ADR 0299, ADR 0304, ADR 0407.*
|
||||
|
||||
- **The window.** The controller puts every merge into a branch a module follows into the open **batch**.
|
||||
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
|
||||
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
|
||||
merges and its times are kept in the controller's store and survive a restart.
|
||||
- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for
|
||||
every commit in it, tiered along the module graph across repositories, with at most one commit per
|
||||
repository, the latest of its branch. The walk names every merge it answers: its own commits, and the
|
||||
earlier merges of a repository that its later commit contains.
|
||||
- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started
|
||||
is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new
|
||||
walk, which answers every merge the folded one answered.
|
||||
- **The controller's own path.** A merge that moves a module whose walk waits for nobody's word (the controller,
|
||||
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
|
||||
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
|
||||
walk like any other. An own-path walk folds no waiting catalogue walk; that walk's merges are batched again
|
||||
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
|
||||
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is
|
||||
answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends
|
||||
`covered` at once.
|
||||
- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for
|
||||
an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own
|
||||
commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own
|
||||
commit, the newest first, one walk each, until one is delivered. A walk stopped stops every delivery whose
|
||||
merge it walks again nowhere (below).
|
||||
- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's
|
||||
tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is
|
||||
at its turn.
|
||||
- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a
|
||||
folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting
|
||||
walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
|
||||
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
|
||||
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
|
||||
and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as
|
||||
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
|
||||
delivery it holds.
|
||||
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch
|
||||
waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk
|
||||
answers within ten minutes is held with the true reason (the open walks of its repository and that none
|
||||
names its merge, or that the controller holds none) and listed by `stalled` at once.
|
||||
- **Every merge is recorded at once.** The forge's merged event records a merge into the open batch of what it
|
||||
touches, or opens one, by whichever controller holds the plans. A merge is never refused, held or dropped,
|
||||
and needs no person's release (issue 921). The bus's durable consumer keeps every merge event until it is
|
||||
recorded; no catch-up timer looks for missed merges.
|
||||
- **The window is two keys.** An open batch's window is a key whose time to live is `merge-window` (90 s),
|
||||
refreshed by each merge into the batch, beside a key set once when the batch opens whose time to live is
|
||||
`merge-window-at-most` (10 minutes). The first to expire closes the batch, heard by a watcher; no cutter
|
||||
polls. A stop that records merges again into the open batch closes its window too (ADR 0299 §3).
|
||||
- **A group's merges are one batch.** A batch holding a merge of a delivery group with a member not yet merged
|
||||
does not close when either key expires; once the group's last member merged, the window runs again from that
|
||||
merge. A group whose last member has not merged ten minutes (`merge-window-at-most`) after its first raises a
|
||||
condition naming the member and why, and nothing of the group is delivered while it waits (ADR 0418 §3).
|
||||
- **A lone merge is cut at once**: no batch it shares a repository, module, graph edge or delivery group member
|
||||
with is assembling (ADR 0282 §2, ADR 0304 §1, ADR 0306).
|
||||
- **The plan.** A closed batch becomes one delivery plan: every commit in it, tiered along the module graph
|
||||
across repositories, at most one commit per repository, the latest of its trunk, each module it moves built once
|
||||
and sent once to each node that runs it. It answers every merge in the batch, its own commits and the earlier
|
||||
merges a later commit contains (ADR 0276 §2, as ADR 0299 §7 changed it; ADR 0418 §1). No member of a group has
|
||||
a turn: only the module graph orders its tiers (ADR 0418 §2).
|
||||
- **A plan fixes its commits when it starts.** A plan starts at its first send, after mesh-delivery's word, given once for a
|
||||
group's plan when every member is merged (ADR 0418 §2), and
|
||||
from then nothing that merges changes it. A plan still waiting for its word is folded into the next batch
|
||||
(ADR 0276 §4).
|
||||
- **Plans that share something run one after another.** A batch that shares a repository, a module or a graph
|
||||
edge with an open plan waits for it, and starts with every merge recorded since when that plan ends. A batch
|
||||
that shares none runs beside the open ones, so a hotfix is never held by an unrelated feature's delivery.
|
||||
- **The ends.** A plan done ends every delivery it answers: `delivered` for its own commits, `covered` for an
|
||||
earlier merge it carried by a later commit. A plan that failed fails its own commits' deliveries; each earlier
|
||||
merge it answered is then delivered on its own commit, the newest first, until one is delivered (ADR 0276 §3). A
|
||||
group's plan fails as one: no member is stopped because another failed, what earlier tiers delivered stays, and
|
||||
the fix is a new change (ADR 0418 §5).
|
||||
- **Shown.** `plans` lists the open batch first, with its window's time left and its merges, and a batch waiting
|
||||
behind an open plan names that plan; the controller says the batch as `plan-moved` in the state `assembling`.
|
||||
|
||||
## A backup before the send
|
||||
|
||||
## A stop on a shared walk
|
||||
*ADR 0214 §7, ADR 0233, ADR 0404.*
|
||||
|
||||
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
|
||||
When a delivery plan is ready to send, each node it reaches backs up, through its `node-backup` seat, every item
|
||||
the plan's moved modules keep there whose protection is a backup (a copy or a dump). An item protected by
|
||||
`redundancy`, one that says `backup: none`, and a cache are never asked, whatever their class. A node with no backup holder backs up
|
||||
nothing, and the plan says so for that node.
|
||||
|
||||
One walk carries every merge of its batch, so ending a walk for one delivery ends it for every other
|
||||
delivery on it. A stop therefore ends only the change it is for:
|
||||
- **The send waits for every backup.** The controller sends nothing of the plan to any node until every backup
|
||||
it asked has answered done.
|
||||
- **A failed backup holds the delivery.** A backup that fails, or does not answer within ten minutes, holds the
|
||||
delivery with the node, the module and the backup's own words as the reason, and raises a condition for the
|
||||
operator. A person's `release` asks the backups again, and the send goes on when they pass.
|
||||
- **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home
|
||||
server, about 165 s (research 055, finding 5). A plan moving no backed-up item costs nothing more.
|
||||
|
||||
## The send and the rollout
|
||||
|
||||
*ADR 0360, ADR 0362, ADR 0377, ADR 0406 §5.*
|
||||
|
||||
- **Every machine at once.** A plan's builds go to every machine that runs them in one step, tier by tier; a tier
|
||||
is asked once the tier before it is built and sent (ADR 0360 §1, §6). No machine goes first and nothing waits
|
||||
for a health reading. A module of policy `record` that would recreate a container or move data waits for a
|
||||
person (ADR 0377).
|
||||
- **Every move is said.** The controller says every move of a plan as `plan-moved`, a move a command makes among
|
||||
them, so mesh-delivery needs no periodic reconcile; it reads `delivery-walks` once when it starts.
|
||||
- **A rollout advances on events.** The `built` and sent events of a tier ask the next; the nodes' reports end
|
||||
the rollout (ADR 0360 §7, with ADR 0362's provisioning report). Each send puts a key whose time to live is
|
||||
ApplySilentAfter (fifteen minutes); its expiry names the machine silent. There is no rollout ticker.
|
||||
- **A failed apply is loud and fixed forward** (ADR 0360 §8); the node-engine's witness still puts back a
|
||||
controller or tool runner that does not come up (ADR 0360 §9).
|
||||
|
||||
## A stop on a shared plan
|
||||
|
||||
*ADR 0299.*
|
||||
|
||||
One plan carries every merge of its batch, so a stop ends only the change it is for:
|
||||
|
||||
- **No stop by member.** A group's plan fails as one (ADR 0418 §5), so mesh-delivery asks no stop of a member
|
||||
after a failed one. The mechanism stands for a person's stop with the way `again`: the controller ends the plan
|
||||
and records every other merge it answered again in the open batch of its kind, whose window the stop closes.
|
||||
- **What is left out with it.** A later merge of the stopped merge's repository, and a merge that moves a
|
||||
module a left-out merge moves, would deliver the stopped change; each is left out too, its delivery `stopped`,
|
||||
naming why.
|
||||
- **A person's stop.** `stop` of a delivery whose plan carries others first answers what it would take down, and
|
||||
acts on `way`: `again` (the others delivered again without it) or `walk` (the whole plan stopped).
|
||||
- **What it does not do.** What the stopped plan already sent stays on the machines it reached; putting it back
|
||||
is issue 509's.
|
||||
|
||||
- **A group's stop.** When a member after a failed one is stopped and its walk is open, mesh-delivery asks
|
||||
`delivery-stop` for that walk with the stopped members' merges to leave out, every member the same pass
|
||||
stops on that walk in one ask. The controller ends the walk, said as stopped, and walks every other merge
|
||||
it answered again.
|
||||
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
|
||||
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
|
||||
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
|
||||
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
|
||||
walked again.
|
||||
- **Where the others go.** They join the open batch of their kind, where merges heard while the walk ran
|
||||
already wait, so an older commit never walks after a newer one; only with none open do they form a new
|
||||
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
|
||||
before it is cut, are cut without the rest of their window. Its walk withholds the left-out merges' modules
|
||||
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
|
||||
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
|
||||
merge moved are written in one transaction.
|
||||
- **The walk's record.** The stopped walk names the batch that walks its merges again, every merge it left
|
||||
out, and each module it already sent that nothing walks again, with the machines it reached. A delivery
|
||||
whose merge is walked again goes from `delivering` back to `published` (the row `rewalk`), or stays
|
||||
`published`, and follows the fresh walk once it is cut. A delivery whose merge was left out, and every
|
||||
delivery on a walk stopped whole, ends `stopped`, naming the walk and the stop.
|
||||
- **A stopped merge never ships by a walk.** mesh-delivery asks every open walk or batch that names a stopped
|
||||
delivery's merge to leave it out, and never gives such a walk its word. When the controller refuses, it
|
||||
stops the whole walk.
|
||||
- **A person's stop.** `stop` of a delivery whose open walk carries other deliveries does nothing at first: it
|
||||
answers with each of them, which of them contain this merge and end either way, and two ways given with
|
||||
`way`: `again` (the walk ended and the others walked again without this merge) or `walk` (the whole walk
|
||||
stopped, every delivery on it with it). A delivery alone on its walk is stopped at once. A delivery is
|
||||
marked stopped only once its walk is; a stop the controller refuses leaves it as it was.
|
||||
- **What it does not do.** What the stopped walk already sent stays on the machines it reached, and its
|
||||
asked builds still register, so a later send may carry them further; the stop names the machines, and
|
||||
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
|
||||
stopped merge's modules, contains the stopped change and delivers it when walked; the stop names each it
|
||||
can see, and holding a merged change back for good is a revert's job.
|
||||
## The delivery budget
|
||||
|
||||
*Decided 2026-10-10 by ADR 0282.
|
||||
Where it differs, this section replaces the window for every merge and the one walk at a time above, slice by
|
||||
slice as each slice below is built; until a slice is built, the text above is what runs.*
|
||||
*ADR 0282, ADR 0304 §4, ADR 0313, ADR 0360 §12, ADR 0362 §5.*
|
||||
|
||||
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
|
||||
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
|
||||
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
|
||||
shown and held to no budget. Measured on 51 walks on 2026-10-09 and 10, a leaf walk took a median of 6.0 minutes
|
||||
(7 of 20 within five), a core walk of one tier 5.6 (worst 8.8), and a controller walk dragging two catalogue tiers
|
||||
14.2 (issue 338).
|
||||
A delivery's **delivery time** runs from its merge until every machine its plan sent has reported the send
|
||||
applied, and for a provisioned module also placed. A **leaf** delivery's budget is three minutes, a **core**
|
||||
delivery's ten. A delivery held for a person, a failed backup among them, is shown and held to no budget. Each
|
||||
phase is kept — the window, the wait behind a plan that shares something, the backups, the build, the send, the
|
||||
reports — and a delivery over its budget raises `delivery.<class>.over-budget`, naming its longest phase. The
|
||||
**fix time** (ADR 0313) runs from an issue's first move to `fixing` to its last fix delivered, and contains the
|
||||
delivery time.
|
||||
|
||||
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
|
||||
above runs from the merge; the operator's own target is wider, the **fix time**, which runs from an issue's
|
||||
first move to `fixing` to the moment its last fix is delivered. Each issue's class carries a **fix budget** too
|
||||
(five minutes leaf, ten core, the delivery budget's own figures), and the tracker splits a fix time into four
|
||||
phases — implementing, review, checks, and merge to running (the delivery time above) — so a fix over its
|
||||
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
|
||||
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
|
||||
## Builds without a toolchain image
|
||||
|
||||
**What changes in the walk.**
|
||||
*ADR 0267, ADR 0350, ADR 0405.*
|
||||
|
||||
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
|
||||
- **Independent walks run at once**: no shared repository, module or module-graph edge with any open walk. One
|
||||
node judges one walk at a time; a send to a node under another walk's judgement waits for its verdict; no leaf
|
||||
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
|
||||
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
|
||||
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
|
||||
- **A tier's first nodes are judged at once.**
|
||||
- **The bus's grants arrive before the gate** (ADR 0298): when a walk's builds change the bus's user list (a new
|
||||
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
|
||||
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
|
||||
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
|
||||
report; the plan shows an expected landing time from the last seven days' medians.
|
||||
- **One build module per language** (Go; Node and TypeScript) installs its toolchain as a declared package on
|
||||
every node holding the build seat, pinned to one version that is the same on every builder. The toolchain
|
||||
image and its module retire.
|
||||
- **Every builder runs the same toolchain, and says so.** Each builder names its toolchain versions with every
|
||||
build outcome, and the controller's self-check raises a condition while two builders differ.
|
||||
- **Builds and tests run directly**, as the build seat's own account on its node, in no container. A
|
||||
repository's `# mesh-check-toolchain` line selects which installed toolchain runs its check.
|
||||
- **A judge has an account of its own.** A judge built from a sibling's merge (ADR 0382 §3) is built and run as a
|
||||
second account of the Go build module, whose home, configuration and cache the build seat's account cannot write,
|
||||
so a check's script cannot change it.
|
||||
- **Every dependency comes by the version its lock file names**, checked against its checksum: the mesh's own
|
||||
code from the mesh's forge, its own packages from its package registry, any other from its public source.
|
||||
- **A builder fetches from the forge over SSH with a key of its own**, as the build seat's account, under a forge
|
||||
account for that builder that reads the mesh's repositories and pushes nothing.
|
||||
- **A build's reuse key** holds the installed toolchain's version and the build module's recipe version in place
|
||||
of the toolchain image's digest, so a toolchain update rebuilds only the modules of that language.
|
||||
|
||||
*Walks beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
|
||||
a change shares).* Building the walks at once found what those bullets left open, and ADR 0306 decides it:
|
||||
## One chain, on the bus alone
|
||||
|
||||
- **A walk's builds are its own, never another's backlog.** A send composes a module whose build another open
|
||||
walk owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
|
||||
judged only by its own walk, so a failed walk puts back only its own builds. What waits on a node with no open
|
||||
walk to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
|
||||
for builds an open walk owns).
|
||||
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
|
||||
merge window whether or not a batch is assembling, so a group's members walk in one batch, in their order.
|
||||
- **Related means everything the walks would move.** A module in common is any module a walk plans to build or
|
||||
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
|
||||
every change. Two related walks are neither cut beside each other nor judged beside each other on one node.
|
||||
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
|
||||
warning naming the walk and the core walks that set it aside.
|
||||
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
|
||||
unhealthy network or engine) fails every judgement open on it, leaf and core, each walk putting back only its
|
||||
own builds.
|
||||
- **A rollback beside another walk's judgement** behaves as it does today; whether it should wait is tracked by
|
||||
issue 528.
|
||||
*ADR 0408.*
|
||||
|
||||
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of walk A left out
|
||||
of walk B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
|
||||
overlapping dependents keeping two walks apart; the warning on the third set-aside; a failed apply failing every
|
||||
open judgement on its node.
|
||||
Module, commit, build, published package or artifact, rollout and assignment are one chain, each link
|
||||
answerable from the next. A **rollout** here is the controller's record of one batch's delivery; a delivery's own
|
||||
delivery plan (*A delivery* above) stays the delivery's, and names the rollout that delivers it.
|
||||
|
||||
**Who builds what.**
|
||||
| Link | Names | Written by | Bucket |
|
||||
|---|---|---|---|
|
||||
| build | its module, repository and commit, its reuse key, what it published: an artifact by digest or a package release by name and version | the controller | `mesh-controller_builds` |
|
||||
| rollout | the commit it fixed per repository, its tiers, the builds it carried, the machine steps, and every assignment it changed with the build before and after; its states and transitions | the controller | `mesh-controller_rollouts` |
|
||||
| assignment | the module and node, the build, artifact or package and commit it runs, and the rollout that put it there | the controller | `mesh-controller_assignments` |
|
||||
|
||||
| Part | Owner | What it adds |
|
||||
|---|---|---|
|
||||
| the walk's phases and readings | mesh-controller | each phase's start and end and each reading in the walk and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
|
||||
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
|
||||
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
|
||||
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
|
||||
| walks at once | mesh-controller | the independence test, one judgement per node, a core walk's ground, a repository's next batch |
|
||||
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
|
||||
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
|
||||
Beside the chain, on the bus too: deliveries and groups (mesh-delivery's `deliveries` and `groups`, ADR 0239 §37),
|
||||
one key per open pull request (mesh-delivery), and each open batch with its merges and its window's two keys (the
|
||||
controller, ADR 0407).
|
||||
|
||||
**Slices**, each one pull request per repository, each with its acceptance sentence:
|
||||
|
||||
| Slice | Delivers | Done when |
|
||||
|---|---|---|
|
||||
| 1 — the measure | the walk's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
|
||||
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its walk's phases |
|
||||
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its walk |
|
||||
| 4 — walks at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
|
||||
| 5 — the landing time | the expected landing time | `plans` shows it for an open walk, and the week's deliveries land within a minute of it at the median |
|
||||
|
||||
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
|
||||
only once it is built.
|
||||
|
||||
## A delivery group
|
||||
|
||||
- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull
|
||||
requests. Two or more make a group. A group that has started delivering takes no new member.
|
||||
- **Order**: `after: <repository>` lines in a member's description, and the controller's `delivery-order`
|
||||
over the graph (ADR 0239 decision 4). The plan shows the order and why each pair is ordered: *declared*,
|
||||
*built by*, *version skew*, *engine before controller*, *by name*.
|
||||
- **Precedence** (ADR 0249): when rules order one pair both ways, the higher one decides. *Declared* comes
|
||||
first, then *built by* and *version skew*, then *engine before controller*. The pair shows the rule that
|
||||
won and the rules it won over. Rules of one rank both ways are a contradiction. The group is `rejected`,
|
||||
and the reason names both members, both rules and the `after:` line that would settle it.
|
||||
- **Check**: the members' heads composed together, one gate over every machine with all their definitions,
|
||||
each head merged into its own trunk's head first and named with that trunk commit (ADR 0323). It is asked
|
||||
by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on
|
||||
every member's head as `mesh/delivery-group`. A member rebased or pushed is a new head with its own check:
|
||||
it takes its old head's place in the group, even once the group is delivering, and the composed check runs
|
||||
again for the new heads instead of holding the group. A merge into a member's trunk that is not one of the
|
||||
group's members puts the composed check out of date: the group reads `checking` with the reason, and the
|
||||
composed check runs again once a member is judged again. Checked by mesh-delivery's
|
||||
`TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember` and `TestARebasedMemberIsComposedAgainNotHeld`.
|
||||
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
|
||||
A member published while its group's composed check is asked and not answered waits there for the answer;
|
||||
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
|
||||
heads that merged and its own walk is still open (ADR 0270).
|
||||
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
|
||||
walk while the walk's other merges are walked again (ADR 0299). The members before it stay delivered.
|
||||
Its own builds were put back at its gate.
|
||||
|
||||
A group's state is derived from its members and its composed check, and is shown, never stored apart from
|
||||
them.
|
||||
- **The bus is the only source.** Each bucket is written by the controller that holds the lease, by
|
||||
compare-and-set, and the controller reads its own buckets to decide; a rollout resumes from its key after a
|
||||
restart. No table of the controller's store holds builds, rollouts, batches or assignments, so no record mirror,
|
||||
audit or healer exists for them; the nodes, settings, declared-config and predefined-keys mirrors stay as
|
||||
ADR 0289 and ADR 0356 have them. A module reads a bucket by naming it in `reads`; mesh-config reads the
|
||||
assignments bucket.
|
||||
- **No key of the chain is removed.** Each bucket keeps up to 64 earlier revisions of each key as its history. A
|
||||
value is never cut to a summary and stays under the bus's largest message. The bus's storage is declared as data
|
||||
protected by a backup, so it is backed up nightly and before a delivery moves the bus.
|
||||
- **One read answers each question.** For any module on any node, one read of its assignment names the commit,
|
||||
the build, the artifact or package and the rollout that put it there; one read of a rollout lists every
|
||||
assignment it changed.
|
||||
- **The tables go by a numbered migration**: builds, rollouts and assignments are written once into the buckets,
|
||||
waiting build requests onto the work queues and an open batch into its keys; then the tables are dropped from the
|
||||
controller's store and its consolidated schema.
|
||||
- **The artifact store and the package registry keep the bytes**; the chain names them (ADR 0251).
|
||||
|
||||
## What every transition does
|
||||
|
||||
In this order, each by its owner:
|
||||
|
||||
1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one
|
||||
key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later
|
||||
each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends
|
||||
it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250);
|
||||
1. **kept**: mesh-delivery puts the delivery in its state by compare-and-set before anything else; what it owes
|
||||
outside is kept with it and retried from ten seconds doubling to an hour, at most sixty times (ADR 0250);
|
||||
2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state
|
||||
changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address;
|
||||
3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's
|
||||
note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its
|
||||
end a line of what was executed there. The forge's holder writes it in its own repository as the forge's
|
||||
own account, and a line already there is not appended again;
|
||||
4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine
|
||||
steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and
|
||||
`mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull
|
||||
request, `mesh/merge-gate` among them.
|
||||
changes), with no secret or address;
|
||||
3. **noted**: a line on the commit's note, appended by the forge module as the forge's own account;
|
||||
4. **reflected**: the delivery's view on the pull request, `mesh/delivery` on the head and `mesh/delivery-group`
|
||||
on every member's head.
|
||||
|
||||
A doubled step is refused by compare-and-set and by the bus's message de-duplication, which also replaces the
|
||||
forge module's files of what it announced (ADR 0406 §6).
|
||||
|
||||
## The verbs
|
||||
|
||||
**mesh-delivery's seat** (served by its holder; the first five read):
|
||||
**mesh-delivery's seat**:
|
||||
|
||||
| Verb | Answers or does |
|
||||
|---|---|
|
||||
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
|
||||
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the walk |
|
||||
| `groups` | every group with its members in order and its derived state |
|
||||
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
|
||||
| `deliveries` | every delivery not final, the final ones of the last day, and every one `stalled` still lists |
|
||||
| `show` | one delivery or group whole: plan, the edge that placed each tier, the Approve, transitions, machine steps, the chain's links |
|
||||
| `groups` | every group with its members and its derived state |
|
||||
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner |
|
||||
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds |
|
||||
| `recheck` | a rejected or ready delivery back to proposed, with why |
|
||||
| `release` | a held delivery to delivering, with why: a person's word |
|
||||
| `stop` | any delivery not final to stopped, with why; its walk is ended through the controller. On a walk that carries other deliveries it first answers what ending the walk would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
|
||||
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
|
||||
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take |
|
||||
| `recheck` | an exception verb, no step of the flow: a check asked again for a proposed delivery nothing else asks again, an inactive one or a failed verdict gone out of date (ADR 0395 §6, ADR 0421 §12) |
|
||||
| `release` | a held delivery goes on, with why; for a failed backup it asks the backups again |
|
||||
| `stop` | a delivery to stopped, with why; on a shared plan it first answers what it would take down (ADR 0299) |
|
||||
| `close` | H2's only verb: the transition the table names for a stalled delivery |
|
||||
| `retire-history` | held history to retired (ADR 0250) |
|
||||
| `times` | each class's budget, the week's median and worst delivery time, and the deliveries over budget |
|
||||
|
||||
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
|
||||
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
|
||||
takes the merges to leave out and walks the walk's other merges again without them, ADR 0299),
|
||||
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
|
||||
is recorded as a hand act. The controller says every walk it keeps as `plan-moved`, from a verb's command
|
||||
too.
|
||||
|
||||
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
|
||||
also announces `pull.closed`, and a merge's head and the statuses the forge holds on it.
|
||||
|
||||
## The store
|
||||
|
||||
Key-value state on the bus, declared by the module (ADR 0201): `deliveries` and `groups`. One writer, the
|
||||
seat's holder, whose code changes state from a single goroutine. A final delivery is kept for thirty days and
|
||||
then removed. Its note on the commit keeps its record after that. The bus's own snapshot (ADR 0235) backs it
|
||||
up.
|
||||
**The controller's verbs for it**: `delivery-plan`, `delivery-order` (the edges that order a plan's tiers,
|
||||
ADR 0419 §2), `delivery-check`, `deliver`,
|
||||
`delivery-stop`, `delivery-walks`, and `plans` (the open batch, the open plans, and a what-if for a repository
|
||||
and its files). A person has `plans go <plan> --why`.
|
||||
|
||||
## The bootstrap
|
||||
|
||||
- A walk that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
||||
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
|
||||
- Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the
|
||||
controller starts it as it always did.
|
||||
- A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises
|
||||
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and
|
||||
`plans go <walk> --why` as the way on.
|
||||
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
|
||||
- A plan that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
||||
controller's own, started by the merge; mesh-delivery records it (ADR 0239 §31). Such a merge shares one batch
|
||||
and one plan with the other merges of its delivery group, and that plan is started by the controller when the
|
||||
group's batch closes, not by its own merge (ADR 0418 §6).
|
||||
- Any other plan waits for `deliver` while the seat has a holder on record; with none, the controller starts it.
|
||||
- A replaced controller or mesh-delivery carries on from the bus: the keys, the queues and the durable
|
||||
consumers hold every step, so nothing is lost or repeated (ADR 0406 §6).
|
||||
|
||||
## The switch
|
||||
## Acceptance tests
|
||||
|
||||
1. The controller's change (the seat, its verbs, `plan-moved`, the wait for a holder's word, `plans go`) is
|
||||
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
|
||||
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
|
||||
and published.
|
||||
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Walks that start
|
||||
from then on wait for its word. Walks already open finish on their own and are adopted as deliveries in
|
||||
`delivering`.
|
||||
4. To step back, unassign it. Waiting walks are then started by the controller at its next pass.
|
||||
What the build proves before this design is implemented (research 055's direction):
|
||||
|
||||
## Phases
|
||||
1. A feature across the controller and the catalogue with a new seat verb is delivered as one group, with no
|
||||
`after:` line, one composed check, one Approve and one delivery plan: its merges are one batch, each module is
|
||||
built once and sent once, its commits are carried by that one plan, and the plan names the edge that placed
|
||||
each tier.
|
||||
2. An SDK change and a Go module using it are delivered as one group.
|
||||
3. A data module is backed up before a delivery moves it, and a failed backup holds the delivery.
|
||||
4. A build on any builder uses the installed toolchain with no container, and a toolchain update rebuilds only
|
||||
modules of that language when its version changes.
|
||||
5. Two merges into one repository within a minute leave each open pull request with one queued check, by the
|
||||
stream's rule.
|
||||
6. The operator's Approve is asked within seconds of the event of a pull request's checks passing, with no ready
|
||||
mark, label or comment written by anyone.
|
||||
7. A controller replaced in the middle of a rollout resumes it with no step lost or repeated.
|
||||
8. The merge window closes by its key's expiry.
|
||||
9. A hotfix to a module the running feature's delivery does not touch reaches the machines in minutes, beside it.
|
||||
10. For any module on any node, one read of its assignment names the commit, build, artifact or package and the
|
||||
rollout that put it there, and one read of that rollout lists every assignment it changed.
|
||||
11. Two merges into one repository while a delivery of that repository runs are both delivered by the next plan,
|
||||
with no person's release and nothing held in mesh-delivery.
|
||||
12. No table in the controller's store holds builds, rollouts or assignments.
|
||||
|
||||
| Phase | Delivers | Done when |
|
||||
|---|---|---|
|
||||
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
|
||||
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a walk never let go is raised by S16 |
|
||||
| C — the walk itself | the per-tier walk moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
|
||||
## What retires
|
||||
|
||||
- The "release branches" and the controller's judge of them (ADR 0358, superseded by ADR 0401).
|
||||
- The toolchain image and its module (ADR 0405).
|
||||
- The window cutter, the merge catch-up timer, the rollout ticker, mesh-delivery's tick, reconcile and
|
||||
ready-mark poll, the forge's poll as a trigger, the queue reordering, the forge-asking cancel, and the forge
|
||||
module's announcement files (ADR 0406, ADR 0407).
|
||||
- The ready mark: the label `mesh/ready`, the `ready at` comment, `mark-ready`, the forge module's
|
||||
`ready-authors` setting, and mesh-delivery's subscriptions to labeled, unlabeled and commented events
|
||||
(ADR 0421 §4, §11); a recheck by hand as a step of the flow (ADR 0421 §12).
|
||||
- A group's order: `after:` lines, the ranked rules and the order of the repositories' names (ADR 0419, which
|
||||
supersedes ADR 0249).
|
||||
- Delivery member by member, a stop of a member after a failed one, and the own path's separate batch for a
|
||||
group's merges (ADR 0418 §1, §5, §6).
|
||||
- The controller's tables of builds, build requests, rollouts and their tiers, batches and merges, and
|
||||
assignments; the record mirrors, probe D17's audit and healer H6 for those kinds (ADR 0408).
|
||||
|
||||
## Milestones
|
||||
|
||||
Confirmed by the operator on 2026-10-17 (playbook 04). Each is one release of playbook 07; the next starts
|
||||
when the previous one's delivery is done.
|
||||
|
||||
| # | Outcome | Acceptance tests | Repositories | Issues |
|
||||
|---|---|---|---|---|
|
||||
| 1 | A merge during a running delivery is never dropped and needs no person's release; the merge window lives on the bus; a rollout that shares nothing with an open one runs beside it | 8, 9, 11 | mesh-controller | 973 (with the rest of 525) |
|
||||
| 2 | A ready mark is acted on within seconds, and each pull request has at most one waiting check | 5, 6, 7 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 974, 975, 976 |
|
||||
| 2b | build-agent and route-proxy live whole in mesh-controller (ADR 0420, accepted 2026-10-17): the controller accepts a reviewed move, mesh-delivery announces each merge with its Approve, and the two modules move; built as two releases before the rule that a milestone is one release | — | mesh-controller, mesh-catalog (mesh-delivery, build-agent, route-proxy) | release 1: 1025; release 2: 1019 |
|
||||
| 3 | A feature with a new seat verb across the controller and the catalogue is one group, one batch, one delivery plan and one Approve: no order lines, no ready mark, each module built and sent once | 1, 6 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 977, 978, 1022, 895 (with 1046 folded into it: unit and integration tests split, each level within its bound); 605 (1021 closed as its duplicate) is resolved by this build (ADR 0419 §7) |
|
||||
| 4 | A delivery backs up the data it moves first, and a failed backup holds it and is said | 3 | mesh-controller, mesh-catalog (mesh-delivery, the backup seat's holder) | 980, split into 1065, 1066, 1067, 1068 |
|
||||
| 3 fix | A group approved together merges in one pass and goes out in one delivery plan, also when its members' tests read each other's repositories | 1 | mesh-catalog (mesh-delivery, gitea) | 1082, 1049, 1047, 1083, 1079 (1084 closed as its duplicate) |
|
||||
| 5 | One read of a module on a node names the commit, build and rollout that put it there | 10, 12 | mesh-controller, mesh-catalog (mesh-config, nats) | 981 |
|
||||
| 5b | The controller makes no build or deploy plan: building and delivering a change are done outside it, driven by events on the bus, and the controller receives only what each node should run | — (its decision names them) | mesh-controller, mesh-catalog (mesh-delivery) | 1117 |
|
||||
| 6 | Builds use installed toolchains with no container, builders fetch over SSH with their own key, and an SDK change ships in one group with its users | 2, 4 | mesh-catalog (build modules, build-agent, ssh-client, gitea), mesh-controller, mesh-sdk | 979, 971 |
|
||||
|
||||
The issues are linked `depends-on` in this order; 976 depends on 975, and 979 also on 971. Milestone 3's first live group (milestone 4's, on 2026-10-17) held one Approve but merged in two passes and went out in two delivery plans: a member's tests read the other member's repository, so the first merge made the second's verdict out of date, and a closed group never asked its composed verdict again (issues 1049, 1082). The operator put a fix of milestone 3 after milestone 4 and before milestone 5 on 2026-10-18; acceptance test 1 is proved again by it. The operator put milestone 5b after milestone 5 and before milestone 6 on 2026-10-18, as an issue rather than a research effort; where the planning goes is settled by a decision record before it is built (issue 1117), and milestone 6's build changes are read against it. Milestone 2b was
|
||||
put before milestone 3 by the operator on 2026-10-17. Milestone 3 builds ADR 0418, ADR 0419 and ADR 0421
|
||||
(issues 977, 978 and 1022), and splits mesh-controller's unit tests from its integration tests (issue 895, the
|
||||
operator on 2026-10-17), so the unit level holds its bound of ADR 0391; the integration level's sixty seconds
|
||||
is issue 1055. Acceptance test 6 spoke of a ready mark when milestone 2 was delivered; restated for
|
||||
the Approve, it is proved again by milestone 3.
|
||||
|
||||
## What is not decided here
|
||||
|
||||
- A web view of deliveries beyond the pull request's comment.
|
||||
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
|
||||
module's protection tool.
|
||||
- Moving the per-tier walk out of the controller (Phase C).
|
||||
- Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's
|
||||
protection tool.
|
||||
- How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog, mesh-sdk]
|
||||
updated: 2026-10-11
|
||||
updated: 2026-10-14
|
||||
decisions:
|
||||
- "ADR 0292"
|
||||
- "ADR 0283"
|
||||
@@ -13,6 +13,7 @@ decisions:
|
||||
- "ADR 0236"
|
||||
- "ADR 0227"
|
||||
- "ADR 0256"
|
||||
- "ADR 0368"
|
||||
---
|
||||
|
||||
# 48 — A module says how it is healthy
|
||||
@@ -200,7 +201,7 @@ build alike, so any build of the module is excused while the wait checks out.
|
||||
**Each module of a send keeps its own pass** (ADR 0254). A send is still judged as one, with one verdict.
|
||||
The gate also counts, per module, the judgings in a row that found it healthy or waiting for a person;
|
||||
when the send fails, a module whose count reached the passes asked keeps a pass, said "on its own", and is
|
||||
neither put back nor left as a move no gate has seen for every other walk to wait on. A module found broken
|
||||
neither put back nor left as a move no gate has seen for every other rollout to wait on. A module found broken
|
||||
is marked failed and put back at once, and the send's verdict waits until the modules beside it have their
|
||||
own, or the bound, so every module of a failed send leaves it with a verdict: its own pass, or put back.
|
||||
|
||||
@@ -246,6 +247,11 @@ The runtime restarts what exits, as now. The node-engine never restarts, recreat
|
||||
being unhealthy. The condition reaches the operator through the conversation (to-be 46), or a healer once a
|
||||
record gives one that act.
|
||||
|
||||
One module repairs itself, by its own record: the dnsmasq module's watch restarts the resolver's unit when
|
||||
it stops answering, at most once every two minutes and three times an hour, and says through its declared
|
||||
health whether the restart helped (ADR 0368). The node-engine still restarts nothing; any other restart on
|
||||
health needs a record of its own.
|
||||
|
||||
## 8. A declaration is proved before it is trusted
|
||||
|
||||
- **`module check`** refuses a `health` field naming an endpoint the module does not declare under
|
||||
@@ -520,7 +526,8 @@ controller.
|
||||
|
||||
## What is not decided here
|
||||
|
||||
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231).
|
||||
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231). The resolver's own
|
||||
restart is decided, for that module alone, by ADR 0368.
|
||||
- Health of scheduled work, and of what a module's events do (issue 276).
|
||||
- A slow starter that needs more than five minutes to be ready — a change to the bound, recorded, when one
|
||||
exists.
|
||||
|
||||
@@ -31,7 +31,7 @@ shape — what each is for, where its records live, and how they depend on one a
|
||||
| **Placement** | to decide, send and apply what each node runs, and say why | the controller's `inventory` database |
|
||||
| **Provisioning** | to resolve which module serves a provision for which consumer, and wire the two | the controller |
|
||||
| **Identity and access** | to issue, hold, rotate and check credentials and grants | the controller's `identity` and `licences` databases, and the vault |
|
||||
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and walk |
|
||||
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and rollout |
|
||||
| **Health and repair** | to notice what is wrong, say it once, repair what may be repaired unattended, and record what was done by hand | the controller's condition store |
|
||||
| **Data** | to know every item of data a module holds, how precious it is, and keep it recoverable | the manifests' data declarations, and every node's `node-backup` seat |
|
||||
| **Connectivity** | to make every node and module reachable by name where it should be, and unreachable where it should not | the controller |
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-10-10
|
||||
status: in-progress
|
||||
code: [mesh-controller]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0289"
|
||||
- "ADR 0201"
|
||||
@@ -12,7 +12,9 @@ decisions:
|
||||
|
||||
# 54 — The controller's records on the bus
|
||||
|
||||
**What each node runs, how far each walk has got, what each build is doing and what each setting holds
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's record mirrors (internal/recordmirror, cmd/mesh-controller/record_mirrors.go, internal/inventory's record mirrors), the mechanism specified here. Not yet checked mirror by mirror; not broken down into milestones.
|
||||
|
||||
**What each node runs, how far each rollout has got, what each build is doing and what each setting holds
|
||||
can be watched on the bus, and what is watched never says something the controller's store does not**
|
||||
(ADR 0289,
|
||||
from gap G2 of research 036).
|
||||
@@ -51,7 +53,7 @@ do not have: a check that the bucket says what the store says.
|
||||
Each value is built from a named list of fields. A field not on the list is never in the value. A value
|
||||
holds only what is built from its record: no epoch, no time of writing and no time of the last look,
|
||||
because any lease holder must build the same bytes from the same record, and the audit must too. Times
|
||||
that belong to the record itself, such as when a walk was opened or a resource's state began, are fields
|
||||
that belong to the record itself, such as when a rollout was opened or a resource's state began, are fields
|
||||
like any other. The epoch and the time of the write travel in the event.
|
||||
|
||||
### nodes — `mesh-controller_nodes`, key: the node's name
|
||||
@@ -74,15 +76,15 @@ like any other. The epoch and the time of the write travel in the event.
|
||||
| the build the node was last sent for it and the build its node reports running | — |
|
||||
| the seats it holds there, and the provisions it is bound to | the bindings' credentials, which are secrets |
|
||||
|
||||
### walks — `mesh-controller_walks`, key: the walk's id
|
||||
### rollouts — `mesh-controller_walks`, key: the rollout's id
|
||||
|
||||
| Holds | Leaves out, and why |
|
||||
|---|---|
|
||||
| the repository and commit, or the batch of merges it walks (ADR 0276) | — |
|
||||
| the repository and commit, or the batch of merges it rollouts (ADR 0276) | — |
|
||||
| its state (building, rolling, done, failed), its current tier, and each tier's modules and nodes with each step's state | the build and apply logs, which the `plans` and `builds` verbs answer |
|
||||
| when it was opened, when it last moved, when it ended, and its note | — |
|
||||
|
||||
A walk is removed from the mirror 14 days after it ended.
|
||||
A rollout is removed from the mirror 14 days after it ended.
|
||||
|
||||
### builds — `mesh-controller_builds`, key: the build's id
|
||||
|
||||
@@ -137,8 +139,8 @@ called).
|
||||
caller today): a numbered migration of `inventory` and the consolidated schema, in the same change.
|
||||
- **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key.
|
||||
Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server
|
||||
has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do
|
||||
not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14
|
||||
has offered since release 2.11 and the bucket enables), so the markers of removed rollouts and builds do
|
||||
not pile up in a bucket that has no age. The controller's one-minute tick removes rollouts and builds 14
|
||||
days after they ended.
|
||||
- **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry
|
||||
in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place.
|
||||
@@ -169,7 +171,7 @@ counts in §6 shows the gap.
|
||||
**D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run
|
||||
does this for each mirror:
|
||||
|
||||
1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended
|
||||
1. It reads every record the mirror should hold from the store (for rollouts and builds, those open or ended
|
||||
within 14 days) and builds each value as the writer would.
|
||||
2. It lists the bucket's keys and reads their values.
|
||||
3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A
|
||||
@@ -225,10 +227,10 @@ remain open there.
|
||||
| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing |
|
||||
| Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises |
|
||||
| A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second |
|
||||
| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) |
|
||||
| A value is under its cap, or cut and marked | the controller's test, which builds the largest rollout the mesh could make (every module on every node) |
|
||||
| Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket |
|
||||
| A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event |
|
||||
| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
|
||||
| Ended rollouts and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the rollouts bucket's keys including deleted ones at most its live keys plus a day's removals |
|
||||
| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
|
||||
| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
|
||||
| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised |
|
||||
@@ -236,7 +238,7 @@ remain open there.
|
||||
|
||||
## What it leaves out
|
||||
|
||||
- **History.** A mirror is current state. What happened beyond the 14 days a walk or build is kept, and
|
||||
- **History.** A mirror is current state. What happened beyond the 14 days a rollout or build is kept, and
|
||||
beyond EVENTS' week, is gap G8 of research 036, which remains open there.
|
||||
- **People reaching the mirrors**, through a WebSocket and an auth callout, is gap G1, open in the same
|
||||
effort.
|
||||
|
||||
@@ -1,15 +1,19 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-10-12
|
||||
status: in-progress
|
||||
code: [mesh-catalog, hq]
|
||||
updated: 2026-10-18
|
||||
decisions:
|
||||
- "ADR 0290"
|
||||
- "ADR 0321"
|
||||
- "ADR 0423"
|
||||
- "ADR 0424"
|
||||
---
|
||||
|
||||
# 55 — The knowledge base
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built and live since 2026-10-11: the knowledge base is the module mesh-knowledge in mesh-catalog, holding every decision record and research effort (issues 425 and 426, resolved and delivered). Left: the later slices, issues 428 and 429.
|
||||
|
||||
**The mesh keeps its decision records and research efforts whole in a module of its own, `mesh-knowledge`, one
|
||||
instance on the control-node, holding the seat `knowledge-base`: each record's number, status, topic, typed links,
|
||||
its clauses with their own status, and its reasoning, with every version kept. A record is proposed, checked when it
|
||||
@@ -26,7 +30,8 @@ gets, in one call under 4 KiB, the clauses in force, the records they come from,
|
||||
read the reasoning behind each.
|
||||
|
||||
This document designs the first slice in detail, tracked by issue 425. The slices after it are in
|
||||
[§ 9](#9-the-slices-after-the-first).
|
||||
[§ 9](#9-the-slices-after-the-first), which also designs the third: designs and the glossary's entries become records,
|
||||
and the knowledge base works out each design's state (ADR 0423, ADR 0424).
|
||||
|
||||
## 1. The parts
|
||||
|
||||
@@ -39,9 +44,12 @@ This document designs the first slice in detail, tracked by issue 425. The slice
|
||||
├ verbs ask, get, holds, links, history, reserve, propose, import, return
|
||||
│ served by the control-node's tool runner
|
||||
├ events proposed, reviewed, accepted, changed, imported, drafted, confirmed, returned
|
||||
└ consumes the router's warrants: the operator's Approve for a record that needs it (§ 5)
|
||||
├ consumes the router's warrants: the operator's Approve for a record that needs it (§ 5)
|
||||
└ consumes the delivery module's delivery-done announcement: a design's state (§ 9)
|
||||
|
||||
mesh-issues (control-node) the counters for decision and research numbers (§ 6)
|
||||
mesh-issues (control-node) the counters for decision and research numbers (§ 6);
|
||||
the issues' design links, read for a design's state (§ 9)
|
||||
mesh-delivery (its seat) announces each delivery done, on the bus (§ 9)
|
||||
claude-code (same node) claude_code_judge: review a proposal (§ 5), draft and confirm clauses (§ 7)
|
||||
records (any node) records_read at a commit: the import's source (§ 8)
|
||||
messenger (control-node) the router: the operator's ask and its warrant
|
||||
@@ -55,9 +63,11 @@ the records module, as the issue tracker's import does.
|
||||
|
||||
## 2. What one record holds
|
||||
|
||||
- `number` and `kind` (decision or research), its key together; `title`, `topic`, `status`, `date`, `deciders`;
|
||||
- `number` and `kind` (decision or research; with the third slice also design and glossary entry, § 9), its key
|
||||
together; `title`, `topic`, `status`, `date`, `deciders`;
|
||||
- its **reasoning**, in sections: for a decision, context, the options weighed, consequences, how it is checked,
|
||||
references; for a research effort, its overview and each of its documents by name; markdown, as written;
|
||||
references; for a research effort, its overview and each of its documents by name; for a design, its `to-be`
|
||||
section and, where something of it runs, its `as-is` section (§ 9); markdown, as written;
|
||||
- for a decision, its **clauses** (§ 3);
|
||||
- its **links**, typed, each stored once and read from both ends;
|
||||
- its **versions**: every change is a new version with who made it, when, and why in one line; `history` lists them
|
||||
@@ -130,7 +140,7 @@ rules that judge the whole record set run (an accepted decision cited by nothing
|
||||
undo the acceptance. It is a fault of the records, which their authors repair, not of the module: it is told to the
|
||||
operator once, as a warning naming the rule and the records, and cleared once the rule holds again. The module's
|
||||
health names it and stays healthy, because health says only whether the module works, and a finding never holds a
|
||||
walk at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
|
||||
rollout at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
|
||||
reviewed).
|
||||
|
||||
The rule `cited` is also judged ahead, in the review, before the judgement: a decision that names no home and that no
|
||||
@@ -190,11 +200,89 @@ field by field and section by section, and the result is written on issue 426.
|
||||
|---|---|---|---|
|
||||
| 1 | this document: the module, the import, clauses, answers, `propose`, its own review and acceptance | the acceptance sentence above, live, and a record proposed, reviewed and accepted without a pull request | issue 425 |
|
||||
| 2 | the decision records and research efforts leave this repository; path mentions become "ADR NNNN" and "research NNN"; playbooks 01 and 02 changed | this repository holds no decision record, and `get` answers every one | issue 426 |
|
||||
| 3 | designs, playbooks and glossary entries as kinds | `holds` answers for a design's decisions and a glossary word | issue 428 |
|
||||
| 3 | designs and the glossary's entries as records, moved in one delivery, and each design's state worked out by the knowledge base (§ 9.1 to § 9.3; ADR 0423, ADR 0424) | `get` answers design 47 with its sections and links, a word's entry answers from the knowledge base, and a design's state follows its issues' deliveries without anyone writing it | issue 428 |
|
||||
| 4 | meeting notes, drawings and blog posts | not decided; needs its own decision | issue 429 |
|
||||
|
||||
Numbers from the issue tracker, which `reserve` needs, are the tracker's own slice, tracked by issue 424. This
|
||||
repository keeps no copy of the records, generated or otherwise (issue 435, answered by the operator).
|
||||
repository keeps no copy of the records, generated or otherwise (issue 435, answered by the operator). Playbooks
|
||||
stay in this repository as files: they are how work is done here, not records of what was decided (research 061).
|
||||
|
||||
### 9.1 Designs and glossary entries as records
|
||||
|
||||
**A design** is a record of the kind `design`, named "design NN" and keeping its number (ADR 0423). Its text is its
|
||||
sections: the `to-be` section, what is being built, and the `as-is` section, what runs. Each of the as-is documents
|
||||
becomes the as-is section of the design it describes, never a record of its own: the import matches 3 of the 18 by
|
||||
name and 15 by topic, and shows the operator the list before it runs. Its owning repositories are its field `code`;
|
||||
its decisions, and every design it names, are `cites` links. It holds no state written by anyone (§ 9.2). A new
|
||||
design takes its number from `reserve` and is written with `propose`, and is reviewed as every record is (§ 5).
|
||||
|
||||
**A glossary entry** is a record of the kind `glossary entry`, one per entry of the glossary page, kept whole: its
|
||||
head word or words (an entry may name several with one meaning, as research effort, graduation, hand-off and playbook
|
||||
do), its domain, its meaning, its *Not:* line with each retired word's scope, and its identifier line. Each row of
|
||||
the page's homonym table becomes an entry of its own, naming the qualified form each domain uses. The entries replace
|
||||
the page as the authority on words (ADR 0423, changing ADR 0244).
|
||||
|
||||
**The readers change with them.** Everything that read the page or the design folders reads the records instead:
|
||||
|
||||
| Reader | Reads, after the move |
|
||||
|---|---|
|
||||
| the knowledge base's own rule on a retired word (§ 4) | the *Not:* and identifier lines of the entries, no longer the page through the records module; it judges a design when it is proposed, as it judges a decision |
|
||||
| hq's word check | the entries' *Not:* and identifier lines, and its uniqueness check (a head word heading two entries, or also retired) over the entries; it covers what stays in this repository, and designs leave its coverage |
|
||||
| the catalogue's repository check | the retired words of no scope or *(tools)* from the entries; its copied list is retired, so hq has no copy to compare |
|
||||
| hq's cycle check | the design records, for every design naming its decisions, not files under `03-DESIGN/` |
|
||||
|
||||
Each of the four fails, saying so, when the knowledge base does not answer, and never passes on nothing.
|
||||
|
||||
**Links become links between records, and the old paths go.** At the import every file link inside a design becomes
|
||||
a link between records, which the rule on a link to nothing then checks (§ 4). The links from documents that stay in
|
||||
this repository are rewritten to name the design ("design 47"). No page stays at an old path: afterwards this
|
||||
repository holds no file under `03-DESIGN/` and no `00-META/glossary.md`.
|
||||
|
||||
**It moves in one delivery**, as the records did: the import, the two kinds, the four readers, the rewritten links,
|
||||
the playbooks' steps that write a design, and the instruction text sessions read, which names the glossary's place.
|
||||
A reader left behind would pass on nothing, so none moves alone.
|
||||
|
||||
### 9.2 A design's state
|
||||
|
||||
**The knowledge base says a design's state, and nobody writes it** (ADR 0424): proposed, designed, in progress or
|
||||
implemented, kept matching what runs. No person, session or document sets it, and `propose` refuses a design that
|
||||
carries one.
|
||||
|
||||
It reads two things. The issues are those the issue tracker holds with a `design` link to the design. Whether one
|
||||
is done comes from its deliveries: an issue is done when it is resolved and the delivery its fix went out in reached
|
||||
`delivered`, whether a hotfix alone or a delivery group. The delivery module announces each delivery done on the
|
||||
bus, and on each announcement the knowledge base works the state of every design that issue links out again.
|
||||
|
||||
```
|
||||
issue tracker ── issues with a design link ──┐
|
||||
├──▶ knowledge base ── works out ──▶ the design's state
|
||||
mesh-delivery ── "delivery done", bus ───────┘ (served by get, ask)
|
||||
```
|
||||
|
||||
**The rules.**
|
||||
|
||||
| Case | State |
|
||||
|---|---|
|
||||
| a decision the design names is not accepted | proposed |
|
||||
| no linked issue done, or no linked issue at all | designed |
|
||||
| some linked issues done | in progress |
|
||||
| every linked issue done | implemented |
|
||||
|
||||
Three kinds of issue count apart: an issue closed won't fix is left out; an issue resolved with nothing to deliver,
|
||||
a fix of documents only, is done when resolved; and a new issue linked to an implemented design makes it in progress
|
||||
again, until that issue is done. Milestones are not read: a milestone is a group of a design's issues delivered
|
||||
together, and the state depends only on the issues and their deliveries.
|
||||
|
||||
**Imported designs say what they lack.** A design imported with no linked issue shows `no issues linked yet` beside
|
||||
designed, until an issue is linked to it; the import names each such design by number. Linking their issues is
|
||||
tracked by issue 428.
|
||||
|
||||
### 9.3 Its place beside design 47
|
||||
|
||||
The delivery-done announcement this slice reads is design 47's
|
||||
([47 — Delivery](47-delivery-from-commit-to-delivered.md)): its hotfixes and delivery groups, and each delivery's
|
||||
end at `delivered`. This slice is an extension after design 47's last milestone. It changes none of design 47's
|
||||
milestones and nothing of how a delivery runs; it only listens for the announcement.
|
||||
|
||||
## 10. How it is checked
|
||||
|
||||
@@ -212,6 +300,26 @@ repository keeps no copy of the records, generated or otherwise (issue 435, answ
|
||||
| A record written in this repository meanwhile writes its clauses | `records.py`, check `clauses`, on every hq pull request, until the records left this repository (issue 426); since then `records.py`, check `no-records`, refuses any record written here |
|
||||
| Live | the acceptance sentence, asked from a session outside this repository |
|
||||
|
||||
**The third slice's acceptance tests** (§ 9.1 to § 9.3; research 062's direction), what its build proves before it
|
||||
is done:
|
||||
|
||||
1. After the import, `knowledge-base.get` of design 47 returns its sections and its links, and every link from a
|
||||
design resolves to a record: none is refused by the rule on a link to nothing.
|
||||
2. This repository's main holds no file under `03-DESIGN/` and no `00-META/glossary.md`, and its link check passes
|
||||
with the 76 links from documents that stay here rewritten.
|
||||
3. hq's cycle check and word check read the knowledge base, and fail, saying so, when it does not answer.
|
||||
4. A proposal holding a retired word is refused after the move, the word read from the entries; the entries are
|
||||
175 and hold 183 head words and 29 retired words, 0 unparsed, both counts checked.
|
||||
5. In the module's tests, each state rule of § 9.2 gives its state from fixture issues; live, an issue linked to a
|
||||
design whose hotfix is delivered turns the design implemented within a minute of the delivery being done, and
|
||||
a new issue linked to it turns it in progress.
|
||||
6. The import reports every design with no linked issue as `no issues linked yet`, by number.
|
||||
7. The instruction text sessions read names the glossary's new place.
|
||||
8. Every entry of the glossary's Config section is filed under the Config domain, and the domains the entries name
|
||||
are the twelve of ADR 0354.
|
||||
9. Tests 1 to 8 pass together after the one delivery, run against main and the live knowledge base, with nothing
|
||||
in this repository, the catalogue or the instruction text pointing at a removed path.
|
||||
|
||||
## What it leaves out
|
||||
|
||||
Semantic or vector search; a browser view; line comments on a proposal (the judgement's findings name the section and
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-10-13
|
||||
status: in-progress
|
||||
code: [mesh-catalog, mesh-controller]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0316"
|
||||
- "ADR 0317"
|
||||
@@ -12,6 +12,8 @@ decisions:
|
||||
|
||||
# 56 — The mail client, set up by the mesh
|
||||
|
||||
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: issue 586, this design's core, is resolved and delivered (hq#336, mesh-catalog#262, mesh-controller#286), and mesh-catalog's betterbird module has every piece named here. Not yet proven live end to end, nor given an as-is: issue 1076.
|
||||
|
||||
**On every node that holds a display server, the `betterbird` module installs Betterbird from a pinned
|
||||
upstream release, ships the Owl add-on with it through Betterbird's enterprise policy, and declares the
|
||||
operator's mail accounts without their credentials. A password or token never passes through the mesh: each
|
||||
|
||||
@@ -0,0 +1,363 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-controller, mesh-host, mesh-sdk, mesh-catalog]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0348"
|
||||
- "ADR 0349"
|
||||
- "ADR 0201"
|
||||
- "ADR 0113"
|
||||
- "ADR 0158"
|
||||
- "ADR 0360"
|
||||
- "ADR 0362"
|
||||
---
|
||||
|
||||
# 57. Building config and provisioning
|
||||
|
||||
**The work of ADR 0348 (config is a domain of its own) and ADR 0349 (provisioning), broken into
|
||||
packages small enough that each ends at something that can be seen to work, in the order their
|
||||
dependencies allow.** The two decision records are the authority on *what* is built; this document holds
|
||||
only the packages, their order, their sizes and their proofs, and is wrong the moment it disagrees with
|
||||
them. It is the shape [design 38](38-building-the-operators-machine.md) gave the operator's machine.
|
||||
|
||||
## How this is built
|
||||
|
||||
**Each package is one issue in the issue tracker, and one pull request per repository it touches.** The
|
||||
issue carries the package's number here in its title, links the packages it waits for with `depends-on`,
|
||||
and links ADR 0348 or ADR 0349. At most five packages are worked at once, each by its own agent in its
|
||||
own worktree; a package starts once every package it depends on is merged. Every pull request is
|
||||
reviewed independently at its head and merged only with its checks green.
|
||||
|
||||
**It ships as one release.** The operator's decision of 2026-10-14: one delivery, not one per package.
|
||||
Each of mesh-controller, `mesh-host`, mesh-sdk, mesh-catalog and hq has the branch
|
||||
`feat/57-config-and-provisioning`, and every package's pull request merges into that branch, never into
|
||||
main. When the last package is merged, the whole is rehearsed in the lab: the rollout, the moves of data in
|
||||
their order, and every proof that is run live. Then each repository's branch reaches main in one pull
|
||||
request, and the five, sharing their branch's name, are one delivery group (ADR 0239 §2), judged as one
|
||||
future state. The five are merged within one merge window, so they are one batch and one rollout (ADR 0276
|
||||
§1, §2), in an order declared with `after:` lines in each pull request (ADR 0239 §11): mesh-sdk, then
|
||||
`mesh-host`, then mesh-controller, then mesh-catalog, then hq. The release holds every package but T6, R1 and R2:
|
||||
the media catalogue and the applications live in repositories of their own and move after it, and only
|
||||
then can the old forms be refused (R1) and the record say what runs (R2).
|
||||
|
||||
**The moves of data run after the delivery, in this order, each a verb that says what it did:**
|
||||
placement out of the settings layers (F2), the settings layers into mesh-config (K5), the secrets into
|
||||
the vault, nothing rotated (V3), then the providers one by one (B2, B3). Each is rehearsed in the lab
|
||||
first, and each is checked by reading back through the module that owns the data before anything
|
||||
points at it.
|
||||
|
||||
**What a release costs.** A proof that is run live (dunst on a workstation, a secret reaching its file, a
|
||||
database consumer connecting) is first seen in the lab rehearsal, and a fault found then holds the
|
||||
release, not one package. A merge's builds go to every machine at once (ADR 0360), so the release has no
|
||||
first node to hold it: after the pull requests' checks, the lab rehearsal is the only live check before it
|
||||
reaches every node. A machine that fails to apply it, or whose provisioner reports a module's files failed
|
||||
(ADR 0362), is named in the rollout's end and raises apply-failing, and is fixed forward with a new pull
|
||||
request into main; one that says nothing within fifteen minutes is named silent and gets its builds from
|
||||
the backlog rollout once it is heard from (ADR 0360 §10).
|
||||
|
||||
**Nothing breaks on the way.** ADR 0349 clause 17: each file is filled the old way or the new way, never
|
||||
both, by the form of its manifest, and the old forms stay accepted until the last module has moved.
|
||||
Inside the release this still holds: the media catalogue and the applications kept in their own
|
||||
repositories move after it, so the old forms stay accepted until they have (R1).
|
||||
|
||||
**Where it stands, 2026-10-17** (the Milestones section below). The packages' branches were merged into
|
||||
main of mesh-controller, the node-engine (`mesh-host`) and mesh-catalog on 2026-10-16. T6 moved after them, as
|
||||
planned above, and is resolved. B3 was left out as well, so B3, R1 and R2 are what is left.
|
||||
|
||||
## What exists already, measured
|
||||
|
||||
Counted 2026-10-14 at mesh-controller `1601cb1`, the node-engine (`mesh-host`) `1a26778`, mesh-sdk `a2d2af0`, mesh-catalog
|
||||
`ebf7f37`.
|
||||
|
||||
| Piece | Today | Becomes |
|
||||
|---|---|---|
|
||||
| the controller's composer | `catalogue.Resolution.compose` fills every namespace but `${secret:}` in one loop (`declaration.go` ~1000–1085); the namespaces and their fields are tabled in `fillers` | fills placement only, for a module that has moved |
|
||||
| settings | tables `settings`, `settings_history`; the `settings` verb (`modules.go` 407–760); proposals and asks (`proposals.go`, 975 lines); trusted files; `takes-settings` merging; about 4 100 lines and 2 500 of tests | the `mesh-config` module and its buckets; the verb `config` |
|
||||
| placement kept as settings | `places`, `accesses` and the kept networks are read from settings layers | facts of an assignment, the controller's own |
|
||||
| secrets | the controller mints, seals and accepts every secret (`internal/secrets`, `inventory/secrets.go` 1 125 lines; tables `secret`, `module_secret`, `secret_ask`, `operator_key`); the vault (`mesh-vault`, TypeScript) is read-only and never asked | the vault, in Go, holding and minting; handing out on request and reply |
|
||||
| bindings | grant files and binding files composed by the controller (`declaration.go` 750–818); `${bound:}` 117 uses | each provider answers its consumers; the vault mints the pair's credential |
|
||||
| contributions | `${contribution:}` (4 collectors), `${shell:}` (3), `${environment:}` (1); 22 contributors | collections, filled by the provisioner client |
|
||||
| file text | 225 file resources in 88 modules, all inline (about 193 KB); 12 modules keep a second copy in 34 files with a test that the two agree (ADR 0349's context counted eleven; the twelfth is lemurs) | template files under `files/`, packed by the module's build |
|
||||
| packing a directory | the builder's `archive` artifact packs a tree and publishes it by digest; the node-engine fetches and unpacks it | the same mechanism carries a module's templates |
|
||||
| the node-engine's file writing | `applyFile`, `applyBlock`, `applyInto`, create-once, drift written back, `Keep` backups, restart-on and reload-on, container reads | the same code, reached also through a narrow local write verb |
|
||||
| the node-engine's local verbs | one socket (`cli.sock`), peer-checked by uid, forwarding to the controller; no local write | a second, narrow socket for the provisioner client |
|
||||
| bus identities | one principal per node for the tool runner, one per module with a process of its own | the provisioner client gets one of its own |
|
||||
| the SDK | KV, watches, asks, caller proof; no sealing helper; the TypeScript `provisioner` is a database-provider harness | a seal box, provider helpers, and readers for config, bound values and secrets |
|
||||
|
||||
**Two measurements decide the order.** The builder can already pack a module's directory, and the
|
||||
controller already fills every placeholder at registration time from text; so template files can ship
|
||||
first, the controller reading the text back from the packed `files/`, and nothing on a node changes. And
|
||||
the node-engine's writing code needs no change in kind, only a second way in; so the provisioner client
|
||||
reuses it rather than writing as root itself.
|
||||
|
||||
## The order the work allows
|
||||
|
||||
```
|
||||
package waits for package waits for
|
||||
F1 — P1 —
|
||||
F2 — P2 P1
|
||||
F3 — P3 F3
|
||||
F4 — P4 P3
|
||||
T1 — P5 P2, P4, T1, K3
|
||||
T2 T1 P6 P5
|
||||
T3 T4 T5 T2 P7 P6
|
||||
T6 T1 V1 F1
|
||||
K1 T1 (same file) V2 V1
|
||||
K2 K1 V3 V2
|
||||
K3 F3, K2 V4 V3, P5
|
||||
K4 K3 V5 V3, V4
|
||||
K5 F2, K4 B1 P5, V5
|
||||
K6 K1 B2 B1
|
||||
K7 K5, P6 B3 B2
|
||||
L1 P6 L2 L1
|
||||
S1 P6 R1 T5, T6, K6, K7, P7, V4, B3, L2, S1
|
||||
R2 R1
|
||||
```
|
||||
The issue tracker carries these as `depends-on` links (thirteen issues also link what they wait for
|
||||
through another package), so the packages ready to start are those whose linked packages are all merged
|
||||
into the release's branches. An issue's "Waits for: … delivered" means merged into the release's
|
||||
branches: nothing is delivered before the release. The first wave touches different repositories and different files:
|
||||
**T1, F1, F3, P1 and F4**. K1 waits for T1 because both change `internal/catalogue/manifest.go`; P3
|
||||
waits for F3 because both change `internal/broker`; F2 joins as soon as a slot is free.
|
||||
|
||||
## 0 — Foundations
|
||||
|
||||
**F1 (issue 650) — A seal box in the SDK.** *mesh-sdk. Half a day.* The ADR 0206 box (X25519, HKDF-SHA256,
|
||||
AES-256-GCM) moves from the licence manager into `mesh-sdk/go/seal`, with a TypeScript twin, and the
|
||||
licence manager uses it. **Proof:** a byte-for-byte test against the licence manager's existing sealed
|
||||
values; the licence manager's tests pass unchanged.
|
||||
|
||||
**F2 (issue 651) — Placement leaves the settings layers.** *mesh-controller. Two days.* `places`, `accesses` and the
|
||||
kept networks become facts of an assignment, held by the controller beside the assignment, with a
|
||||
migration moving each existing layer's keys; `Settle` stops letting a setting replace a provider's
|
||||
served keys. **Proof:** composition of every node before and after the migration is byte-identical;
|
||||
the settings layers hold no placement key afterwards.
|
||||
|
||||
**F3 (issue 652) — The assignments and the modules' config declarations on the bus.** *mesh-controller. Two days.*
|
||||
The controller keeps the bucket of assignments design 54 decides (which module on which node), and a
|
||||
bucket of each registered module's declared config values, written when they change, with one event per
|
||||
write. mesh-config needs both to resolve. **Proof:** a test against a real bus: an assignment and a
|
||||
registration each change exactly one key and raise one event.
|
||||
|
||||
**F4 (issue 653) — The words.** *hq. Half a day.* The glossary gains *config value*, *predefined key*, *provision*
|
||||
and *provisioner client*, *collection*, and the ownership words *owned*, *seed* and *managed*. The word "setting" goes on *config
|
||||
value*'s *Not:* line only in R2, once the 67 documents still using it are reworded; until then the
|
||||
glossary says it is the older word. **Proof:** `words.py` passes.
|
||||
|
||||
## 1 — Template files (ADR 0349 clause 2)
|
||||
|
||||
**T1 (issue 654) — A file resource may name a template.** *mesh-controller. Three days; the largest package of the
|
||||
stage.* A file resource takes `"template": "files/<name>"` in place of `content`; the module's build packs
|
||||
`files/` into an archive artifact of its own (the existing `archive` packing); registration and `Resolve`
|
||||
read each template's text back from the packed tree into `content`, so composition and the node-engine
|
||||
are unchanged; `module check` and the merge gate read templates from the repository tree beside the
|
||||
manifest; a file naming both `template` and `content` is refused. **Proof:** composition tests: a module
|
||||
with its text in `files/` composes byte-identically to the same module with it inline; the merge gate
|
||||
judges a change to a template alone.
|
||||
|
||||
**T2 (issue 655) — The twelve modules with copies.** *mesh-catalog. A day.* adwaita, clipmenu, dunst, feh,
|
||||
gnome-keyring, i3, i3status-rust, lemurs, picom, rofi, screen-lock and xdg move to their templates; the
|
||||
34 inline copies and the `sameAsSource` tests go. **Proof:** each module's composed declaration on its
|
||||
nodes is unchanged (the merge gate's plan says no file changes).
|
||||
|
||||
**T3 (issue 656) — The desktop modules.** *mesh-catalog. A day.* Every remaining module assigned only to
|
||||
workstations. **Proof:** as T2.
|
||||
|
||||
**T4 (issue 657) — The servers' modules.** *mesh-catalog. Two days.* Every remaining module with a build. **Proof:**
|
||||
as T2.
|
||||
|
||||
**T5 (issue 658) — The ten modules without a build.** *mesh-catalog. A day.* ca-trust, de-spiegel, invoicing,
|
||||
local-model-consumer, matrix, node-env, only-office, sshd, step-ca and supabase gain an archive build for
|
||||
their templates. **Proof:** as T2; afterwards no manifest in the catalogue carries file text.
|
||||
|
||||
**T6 (issue 659) — The media catalogue and the applications.** *mesh-media-catalog and each application's
|
||||
repository. Two days.* **Proof:** as T2, per repository.
|
||||
|
||||
## 2 — Config (ADR 0348)
|
||||
|
||||
**K1 (issue 660) — A module declares its config.** *mesh-controller. Two days.* The manifest takes `config` (type,
|
||||
description, validation, trusted, `predefined`, `default` or `required`), `takes-config` and
|
||||
`${config:<key>}`, with `settings`, `takes-settings` and `${setting:}` read as aliases; registration
|
||||
refuses everything ADR 0348's "How it is checked" lists for registration: a value without type or
|
||||
description; a value with neither a default nor `required`; a default failing its validation; an
|
||||
object or list not trusted; a predefined key not on the list or of another type (with K2); an untrusted
|
||||
value falling back to a trusted key; a value neither in a template nor read at runtime; a placeholder
|
||||
naming an undeclared value; a `reload-on` or `restart-on` naming an undeclared file; a moved module's
|
||||
mergeable file not naming its keys; a mesh word or `puid`/`pgid` in `takes-config`. **Proof:** the registration refusal tests, one per
|
||||
rule; every manifest in the catalogue still registers.
|
||||
|
||||
**K2 (issue 661) — The list of predefined keys.** *mesh-controller and mesh-catalog. A day.* `predefined.json` at the
|
||||
catalogue's root, with `font-size` and `agent-managed-settings` first; the build validates it and every
|
||||
manifest naming a key; the merge check refuses removing a key still named or holding a value; the newest
|
||||
trunk build's list is published. **Proof:** the merge check refuses a planted removal; an older build's
|
||||
list does not replace a newer one.
|
||||
|
||||
**K3 (issue 662) — The mesh-config module.** *mesh-catalog. Four days.* A Go module holding the seat `mesh-config`:
|
||||
the buckets `mesh-config_values`, `_resolved`, `_predefined`, `_proposals`; the four sources resolved from
|
||||
F3's assignments and declarations; the verb `config` (set, clear, list, history, narrowed by module and
|
||||
node); validation; the hand-act log; `config-changed`; only its account writes its buckets. Terminal-only
|
||||
for trusted values until K4. **Proof:** its tests resolve each source in turn; a test against a real bus
|
||||
refuses a put from another account.
|
||||
|
||||
**K4 (issue 663) — Trusted values are proposed and approved.** *mesh-catalog. Two days.* mesh-config asks under its
|
||||
own name, with its own grant and cap of open asks, binds each option to the level, key and value, records
|
||||
the approver beside the value, and checks the proposal on Approve (ADR 0277's rules, ADR 0348 clause 8).
|
||||
**Proof:** a proposal test end to end with a stand-in channel; an Approve against a changed value is
|
||||
refused.
|
||||
|
||||
**K5 (issue 664) — The values move, and `settings` forwards.** *mesh-controller and mesh-catalog. Two days.* Each
|
||||
existing layer moves into mesh-config (a node layer to `module.<node>.<module>`; a whole-mesh layer to
|
||||
mesh config, a manifest default or each node, as ADR 0348's consequences say; claude-code's managed
|
||||
settings to `agent-managed-settings`), recorded with why; for a module not yet handed to the provisioner
|
||||
the controller reads `mesh-config_resolved` where it read its layers (ADR 0349 clause 17's old way,
|
||||
an exception to ADR 0348 §9 that R1 removes), and `settings` answers by forwarding to `config`. **Proof:** composition of
|
||||
every node before and after is byte-identical; `settings` and `config` list the same values.
|
||||
|
||||
**K6 (issue 665) — The catalogue says config.** *mesh-catalog. A day.* The 15 modules using `${setting:}`, the 9
|
||||
declaring settings and the 4 with `takes-settings` move to the new names and the full declaration.
|
||||
**Proof:** the catalogue's check finds no old name in a moved module.
|
||||
|
||||
**K7 (issue 666) — The operator's acceptance on dunst.** *live. An hour.* With every level cleared the first
|
||||
workstation's dunstrc says 10; a mesh font size, then that node's font size, then dunst's own value on
|
||||
that node each appear in its dunstrc within the call's answer, and the history shows each change with who and why. Needs P6.
|
||||
|
||||
## 3 — The provisioner (ADR 0349 clauses 3–5, 11, 13)
|
||||
|
||||
**P1 (issue 667) — The node-engine knows who owns a placed file.** *`mesh-host`. A day.* `declaration.File` gains
|
||||
`owner-at-runtime` (`owned`, `seed`, `managed`), mapped onto today's paths (drift written back,
|
||||
create-once, `into`), and the declaration gains the list of files the provisioner fills. **Proof:** the
|
||||
apply tests: each ownership behaves as its old form did.
|
||||
|
||||
**P2 (issue 668) — The narrow local write verb.** *`mesh-host`. Two days.* A second unix socket accepts a write only from
|
||||
the provisioner client's account (`SO_PEERCRED`) and only for a path the declaration names as filled,
|
||||
writes through `applyFile`, `applyBlock` or `applyInto` with `Keep`, and fires restart-on, reload-on and
|
||||
container reads as an apply does. **Proof:** tests: an undeclared path and another uid are refused; a
|
||||
write under `reload-on` reloads its program once.
|
||||
|
||||
**P3 (issue 669) — The provisioner client's own bus account.** *mesh-controller. A day.* A principal of its own per
|
||||
node for `node-provisioner`, granted to read its node's keys in each domain's bucket, publish
|
||||
`prov.applied.<node>.>` and ask the vault; nothing else. **Proof:** the grants test, and a test against a
|
||||
real bus refusing a read of another node's keys.
|
||||
|
||||
**P4 (issue 670) — The provisioner's server.** *mesh-catalog. Two days; after P3.* A Go module holding `mesh-provisioner` once:
|
||||
the registry of which domain answers which kind, the buckets `_registry`, `_needs`, `_applied` and
|
||||
`_collect`. **Proof:** its tests; the buckets listed live.
|
||||
|
||||
**P5 (issue 671) — The provisioner's client.** *mesh-catalog. Four days.* A Go service on every node, outside the tool
|
||||
runner: fetches a module's templates by digest, records needs, watches each owner's keys for its node,
|
||||
fills only when every value is there, writes through P2, reports `prov.applied`, writes an `owned` file
|
||||
back on drift, removes a module's files and needs when it leaves, fills from current values when its node
|
||||
comes back. Its first owners are placement (from the declaration) and config (from
|
||||
`mesh-config_resolved`). **Proof:** the client's test list in ADR 0349, against a real bus.
|
||||
|
||||
**P6 (issue 672) — The controller hands a module to the provisioner, and the rollout waits for it.** *mesh-controller.
|
||||
Two days.* For a module marked moved, the declaration carries placement, the templates' digest and the
|
||||
filled-files list, and the controller runs no other filling pass for it; a rollout counts the module applied
|
||||
on a machine only after the node-engine's report and that node's `prov.applied` for the new build, and a
|
||||
failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and rollout
|
||||
tests; dunst on the first workstation filled by the client, live.
|
||||
|
||||
**P7 (issue 673) — The node-engine stops writing placed files from the declaration.** *`mesh-host`. A day.* For a moved
|
||||
module the engine writes its filled files only through P2, and removes as an orphan only what the verb
|
||||
placed. **Proof:** the apply tests; no placed file written twice in a day's journal on the
|
||||
first workstation.
|
||||
|
||||
## 4 — Secrets (ADR 0349 clauses 6–11, 16; issue 647)
|
||||
|
||||
**V1 (issue 674) — The vault in Go, holding secrets.** *mesh-catalog. Four days.* `mesh-vault` is rewritten in Go:
|
||||
its own store, a copy of each secret sealed to the operator key, `get` on request and reply sealed to the
|
||||
asker's key (F1's box; the node-engine's key format is settled here and written down), `secret-changed`
|
||||
without a value, no secret in any bucket. **Proof:** the vault's test list in ADR 0349.
|
||||
|
||||
**V2 (issue 675) — The vault's verbs.** *mesh-catalog and mesh-controller. Three days.* `secret accept`, `secret-ask`
|
||||
(with ADR 0277 §24–§27's record and bounds in the vault's store), `issued` (ADR 0305, the provider's tool
|
||||
answering the vault alone) and `rotate` become the vault's; the controller's verbs forward. **Proof:** the
|
||||
`secret-ask` and issuing provider tests in ADR 0349.
|
||||
|
||||
**V3 (issue 676) — The secrets move to the vault.** *mesh-controller and mesh-catalog. Two days.* Every row of
|
||||
`secret`, `module_secret`, `secret_ask` and the operator key's public half are copied into the vault as they are,
|
||||
nothing rotated, and read back through the vault before anything points at it. **Proof:** for every
|
||||
secret, the vault's sealed answer opens to the same value the controller's does.
|
||||
|
||||
**V4 (issue 647) — A node's secrets come from the vault.** *mesh-catalog, mesh-controller, `mesh-host`.
|
||||
Three days.* The client fills `${secret:}` from the vault's `get`; for a moved module the controller seals
|
||||
nothing; the core's credentials travel sealed by the vault to their node and the node-engine opens them.
|
||||
**Proof:** a secret rotated in the vault reaches its file without a new declaration, live.
|
||||
|
||||
**V5 (issue 677) — The vault mints pair credentials.** *mesh-catalog. Two days.* One value per provider assignment and
|
||||
provision, remade for all holders when the vault sees a consumer's need recorded or deleted (ADR 0158
|
||||
§4–§6). A need recorded for a bind the vault already holds from V3 adopts that value and remakes
|
||||
nothing, so the move itself rotates no credential. **Proof:** the provider test in ADR 0349, with a stand-in provider.
|
||||
|
||||
## 5 — Bindings (ADR 0349 clause 12)
|
||||
|
||||
**B1 (issue 678) — A provider's helper in the SDK.** *mesh-sdk. Two days.* Go: watch the needs for a kind, write one
|
||||
entry per consumer, refuse a derived value for a multi-holder consumer, answer from the vault's pair
|
||||
credential. The TypeScript `provisioner` harness is renamed so the word means one thing. **Proof:** its
|
||||
tests with a stand-in bus.
|
||||
|
||||
**B2 (issue 679) — PostgreSQL answers its consumers.** *mesh-catalog and mesh-controller. Three days.* The first
|
||||
provider moves; its grant files and binding files are retired. **Proof:** every database consumer on
|
||||
every node connects after the move with the credential it had, adopted by the vault as V5 says, live.
|
||||
|
||||
**B3 (issue 680) — The other providers.** *mesh-catalog. A week, one pull request per provider.* **Proof:** as B2, per
|
||||
provider; afterwards the controller composes no grant file.
|
||||
|
||||
## 6 — Collections (ADR 0349 clause 13)
|
||||
|
||||
**L1 (issue 684) — A collection is declared and filled by the client.** *mesh-controller and mesh-catalog. Three days.*
|
||||
A collector requires a collection; a contributor provides an item from a template; the client fills the
|
||||
collector's template from `_collect`. `${contribution:}`, `${shell:}` and `${environment:}` read as
|
||||
collections. **Proof:** zsh's file holds every placed contributor's item and loses one whose contributor
|
||||
leaves.
|
||||
|
||||
**L2 (issue 681) — The collectors and contributors move.** *mesh-catalog. Two days.* zsh first, then the bar, the
|
||||
display session, the hotkeys, xorg, power and node-env, with their 22 contributors. **Proof:** each
|
||||
collector's composed file is unchanged at the move.
|
||||
|
||||
## 7 — A module's code asks at runtime (ADR 0349 clause 14)
|
||||
|
||||
**S1 (issue 685) — The SDK's readers.** *mesh-sdk and mesh-catalog. Two days.* `config`, `bound` and their watches,
|
||||
answered by the node's client for what the manifest declares. `secrets` refuses until the module runs
|
||||
under a bus account of its own, which waits on issue 569 and its decision. **Proof:** the SDK's test in
|
||||
ADR 0349; claude-code reads its managed settings through it.
|
||||
|
||||
## 8 — The old forms go (ADR 0349 clause 17)
|
||||
|
||||
**R1 (issue 682) — Refused and removed.** *mesh-controller, `mesh-host`, mesh-catalog. Three days.* Once every module of
|
||||
every catalogue has moved: the catalogue's check refuses inline text, `settings`, `${setting:}` and the
|
||||
old contribution forms; the controller's filling passes but placement, its settings, secret and binding
|
||||
code and tables (a new migration) go; the node-engine keeps opening sealed values for the core's files
|
||||
only; the controller stops reading `mesh-config_resolved`. **Proof:** the controller's test that its
|
||||
code opens no secret outside genesis and reads no config value; the absence of the
|
||||
removed code.
|
||||
|
||||
**R2 (issue 683) — The record says what runs.** *hq. Two days.* The as-is designs are written, designs 27 and 32 are
|
||||
amended, the 67 documents that still say "setting" in running prose are reworded so the glossary can
|
||||
retire the word (F4 left it off the *Not:* line for them), design 49 lists the Config domain, ADR 0348's and 0349's built status is said, and this document's status becomes `implemented`.
|
||||
|
||||
## Milestones
|
||||
|
||||
The packages above were built as one delivery, not milestone by milestone; the milestones group them by what
|
||||
each makes true. Whether a milestone's issues are delivered is read from the issue tracker (playbook 07).
|
||||
|
||||
| # | Outcome | Packages | Issues |
|
||||
|---|---|---|---|
|
||||
| 1 | Config, provisioning, secrets, bindings, collections and runtime asks run the new way; the old forms are still accepted | every package but T6, B3, R1, R2 | 647, 650 to 679 but 659, 681, 684, 685 |
|
||||
| 2 | The media catalogue and the applications carry their file text as templates | T6 | 659 |
|
||||
| 3 | The other providers answer their consumers; the old forms are refused; the record says what runs | B3, R1, R2 | 680, 682, 683 |
|
||||
|
||||
## What is deliberately not here
|
||||
|
||||
- **Secrets in a module's own code** wait on issue 569 (a module's tools run as the tool runner's
|
||||
account) and its decision; S1 ships without them.
|
||||
- **Release separated from delivery** stays issue 644's, and **the bus's replicas** issue 645's.
|
||||
- **Renaming "walk"** is issue 646's.
|
||||
|
||||
## How this list is kept true
|
||||
|
||||
Each package's proof that runs in its tests is run when it is merged into the release's branch, and its
|
||||
line here gains the date and the commit; a proof that runs live is run in the lab rehearsal and again
|
||||
after the release's delivery. A package whose proof fails is not reworded; the failure is recorded under it and the package
|
||||
stays open. The issue tracker says which packages are worked now; this document says only the order.
|
||||
@@ -0,0 +1,322 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: [mesh-catalog, mesh-media-catalog, mesh-sdk, mesh-tools, mesh-controller, mesh-host, mesh-lab]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0383"
|
||||
- "ADR 0384"
|
||||
- "ADR 0385"
|
||||
- "ADR 0386"
|
||||
- "ADR 0387"
|
||||
- "ADR 0388"
|
||||
- "ADR 0389"
|
||||
- "ADR 0390"
|
||||
- "ADR 0411"
|
||||
- "ADR 0412"
|
||||
- "ADR 0413"
|
||||
- "ADR 0414"
|
||||
---
|
||||
|
||||
# 58 — Agent work as the mesh's own
|
||||
|
||||
**An agent's work belongs to the mesh, not to one vendor's program or one machine. A thread is kept whole on the
|
||||
bus by a thread seat; a thread handler on an always-on node runs the agent harness headless for it, in a sandbox
|
||||
of its own, with the thread's own branch on the forge as its workspace; the agent works through a workspace toolkit
|
||||
and named tools, never a general command runner. Every person, agent and module is an identity held by one identity
|
||||
seat, with a role made of grants; the bus checks each login by its signature, and the tool runner checks each call
|
||||
against the role, tool by tool, by the effect every tool carries. An identity is also its logins: the mesh makes each
|
||||
person's and agent's machine account on every machine it is present on, gives each its own bus password, and runs a
|
||||
module for an identity only where the module is set to; everything else runs as the operator.**
|
||||
(ADR 0383 to ADR 0390, from research 053; ADR 0411 to ADR 0414, from research 057.)
|
||||
|
||||
**Acceptance.** The operator starts a thread from the terminal client, an agent on an always-on node works it in a
|
||||
sandbox, asks a question the operator answers in place, and the thread moves to another node and carries on with
|
||||
nothing lost; the agent's calls outside its role are refused, saying which grant is missing.
|
||||
|
||||
## 1. The parts
|
||||
|
||||
```
|
||||
any node always-on nodes the control-node
|
||||
───────── ─────────────── ────────────────
|
||||
front ends nox-thread-handler nox-threads (seat thread)
|
||||
├ terminal client (first) ├ takes a thread's lease ├ a restricted stream per thread
|
||||
├ web page (then) ├ runs the agent harness headless ├ the object store for large content
|
||||
└ Telegram: questions, │ inside a microsandbox └ leases in the bus's key-value store
|
||||
approvals ├ answers permission requests
|
||||
│ │ through the mesh's asks identity seat
|
||||
│ └ commits + pushes the workspace ├ persons, agents, modules, roles
|
||||
│ branch every turn │ in the bus's key-value store
|
||||
│ nox-workspace (node seat workspace) ├ signs logins (one key per role)
|
||||
│ └ read, glob, search, edit, write, └ revokes over the bus, reads back
|
||||
│ git, named tools; no command runner
|
||||
│ claude-code
|
||||
│ ├ mirrors a session into its thread
|
||||
│ ├ rebuilds a session on pick-up
|
||||
│ └ writes the login and licence token
|
||||
│ into the sandbox
|
||||
└──────────────── every call over the bus, carrying its thread and,
|
||||
for a module acting for an agent, the agent's token
|
||||
```
|
||||
|
||||
## 2. Threads (ADR 0383)
|
||||
|
||||
A **thread** is one piece of agent work, kept as a stream on the bus in the mesh's own format. The stream is the
|
||||
source of truth: the conversation, the agent's questions and their answers, tool calls and results, usage, and
|
||||
every change of state. Large content goes to the object store and is named from the stream. The stream is
|
||||
restricted (ADR 0376): only modules granted it read it. A thread is never deleted; the trail is kept for good.
|
||||
|
||||
A thread is **open**, **assigned** (to a node and an agent harness), **released** or **stopped**. Its seat's verbs create,
|
||||
assign, release, stop, move it, and take it over with a reason. The assignment is a **lease** in the bus's
|
||||
key-value store, changed only by a compare-and-set on its revision. Its holder renews it every 20 s. A lease not
|
||||
renewed for 60 s is said to the operator; nothing takes it over by itself.
|
||||
|
||||
## 3. Running a thread (ADR 0384)
|
||||
|
||||
The thread handler on an always-on node takes the lease and starts the agent harness (Claude Code today) headless, one
|
||||
turn at a time. It answers the agent harness's permission requests through the mesh's asks, so a front end answers them in
|
||||
place. At the end of every turn it commits and pushes the workspace, which is the thread's own branch on the forge;
|
||||
a node that picks the thread up fetches it, so a move loses at most the turn in progress. Before a shutdown it
|
||||
releases at a turn boundary; on a stop it ends the process group, since a tool's child process outlives the agent harness.
|
||||
|
||||
The claude-code module mirrors the session's stream into the thread and, on pick-up, rebuilds a session from the
|
||||
thread. Usage per thread comes from each run's result and rate-limit messages on the stream, not the transcript, and the
|
||||
licence manager records it with the thread as its session.
|
||||
|
||||
The front ends are clients of a thread doing the same work: the terminal client first, then the web page, and
|
||||
Telegram for questions and approvals. Their first scope is the conversation as it streams, the agent's questions and
|
||||
permission requests answered in place, sending a message and stopping a turn. The rest is tracked by issue 850.
|
||||
|
||||
## 4. The workspace toolkit (ADR 0385)
|
||||
|
||||
The node seat `workspace`, held by `nox-workspace` on every thread-handler node and served through the mesh's MCP
|
||||
server, works in the thread's workspace only, by relative path: read, glob, regex search, edit, write, git, and named
|
||||
tools such as running the repository's tests and its merge check. It is built early. There is no general command
|
||||
runner in it, and no module offers one to an agent: a command an agent needs becomes a named tool in the module that
|
||||
owns it. Claude Code's own built-in tools, its command tool included, are turned off once the toolkit is in.
|
||||
|
||||
## 5. Identities and roles (ADR 0386)
|
||||
|
||||
The identity seat, held once for the mesh, owns every **identity** (a person, an agent or a module) and every
|
||||
**role**, in the bus's key-value store. An identity has a name, a kind, a role and whether it is switched on. They
|
||||
change only through the seat's verbs, each recorded with who asked; a module or front end whose role grants the
|
||||
seat's reads may watch the store live, and it holds no secret. Only the
|
||||
operator creates a person or an agent, or changes its role. A module's identity is asked for by the controller when
|
||||
the module is assigned to a node, named by machine and module, with a role made from what its manifest declares it
|
||||
calls.
|
||||
|
||||
A role holds **grants**. A grant is an effect everywhere, an effect on one seat, or one named tool, and may be
|
||||
limited to named machines. Two fences check them: the bus, coarsely, by seat and module, from the role's signing
|
||||
key; and the tool runner, exactly, tool by tool, in the SDK. Every call carries its thread; a module acting for an
|
||||
agent carries a token the identity seat signed for that agent, and the tool runner then checks the agent's role. A
|
||||
refusal names the missing grant; every call is recorded with its identity and thread. A session on an operator's
|
||||
machine calls as the operator's person identity or as an agent, never through the machine's tool-runner login.
|
||||
|
||||
### 5.1 An identity on the machines (ADR 0411, ADR 0414)
|
||||
|
||||
A person or agent identity also holds a **login name**, the **machines** it is present on and, for a person, the
|
||||
login name kept on a machine where it differs. The seat starts with two: the operator's person identity and `nox`, the one
|
||||
agent; every piece of agent work is a thread of `nox` (§2), and a second agent identity is made only when a rule needs
|
||||
another role. The agent is `nox` everywhere: its login, its forge account (the forge account `nox-agent` renamed,
|
||||
keys and team kept), Keycloak, mail and the bus. A login or service account the mesh makes for the operator from now on carries the operator's identity name;
|
||||
the operator's existing logins and forge account are kept as they are. On a machine where agents ran as the
|
||||
operator's login, that login stays the operator's and `nox` gets a login of its own; nox's bus account is `agent.nox`. `nox` is also the
|
||||
operator's terminal command (`mesh-cli`), and prose says which is meant.
|
||||
|
||||
An identity's login on one machine is its **machine account**. The controller composes the machine account of every
|
||||
identity present on a machine into that machine's declaration, and the node-engine applies it with its `user`
|
||||
resource. A login that exists is adopted under its name, uid and home; the mesh never renames or deletes one. An
|
||||
agent's machine account has its own home and group, no password, no root group and no sudo rule, and the node-engine
|
||||
judges on every look that it cannot become root without a person, by the ways ADR 0266 §12 to §17 list. No module declares a login for a
|
||||
person or an agent: claude-code's `agent-account` resource goes, and the catalogue's check refuses a manifest that
|
||||
declares one. The node record's operator account and agent account are read from the identities, no longer stated
|
||||
at the controller's terminal or in the mesh-config snapshot; the machine facts `account`, `agent-account` and their
|
||||
homes keep their names. A machine names an agent account only where the identity seat marks an agent as its agent
|
||||
account: the control node, marked `nox`, where the guard and the licence stay as they are. The other machines are
|
||||
marked with none, so the guard stays off there and the operator's sessions keep the mesh's tools, while the login `nox` serves the
|
||||
modules run for `nox`. Where a person or an agent is present, its kept logins and the agent account mark change only
|
||||
through a verb of the identity seat that only the operator calls, on an Approve; an assignment never sets the mark,
|
||||
because an agent may call the verb that changes an assignment (ADR 0266 §4). The verb, and the controller at composition,
|
||||
refuse an agent whose login on a machine would be `root`, a person's login there or a well-known service account
|
||||
(ADR 0266 §5). The self-check judges every agent login,
|
||||
marked or not. The seed: the operator on every machine, with the operator's existing login kept where it differs from the
|
||||
identity's name; `nox` on every machine, the agent account on the control node.
|
||||
|
||||
```
|
||||
identity seat controller each machine (node-engine)
|
||||
───────────── ────────── ──────────────────────────
|
||||
person operator, machines, composes, per machine: the operator's login (adopted)
|
||||
the login kept on each the machine accounts present nox, root: never (adopted on the control node,
|
||||
agent nox, its machines the facts account, agent-account made elsewhere)
|
||||
each identity's bus password ──► its secret, owned by its login, 0600
|
||||
```
|
||||
|
||||
### 5.2 Run for an identity (ADR 0412)
|
||||
|
||||
Every module runs as the operator account of its machine, as today, with no setting; running for the operator is that
|
||||
default run. A manifest may declare that its module can **run for** an agent identity, at assignment, at start or both: serve one identity with that identity's machine account, home and bus password.
|
||||
An assignment of such a module may name the identity it runs for, and the module may let the one who starts it
|
||||
choose: claude-code started for the operator is the default run with the operator's configuration, and started for `nox`
|
||||
it runs for `nox` with the agent's home. A start
|
||||
goes down only: a module is started for an identity by a process already running as that identity's login, or by the
|
||||
operator's own login for an agent; an agent never starts one for a person, and the module refuses it, naming who asked
|
||||
and for whom. A module may be assigned to one machine once per identity it runs for there, each its own process under
|
||||
its own login, and the controller keys an assignment by machine, module and identity. The first two to declare it are
|
||||
claude-code and ssh-client. On a machine the identity seat marks with an agent account, claude-code's assignment is
|
||||
run for that agent (`nox` on the control node), so its licence and home scope stay in the agent's home, and the controller refuses
|
||||
one there that is not. ssh-client keeps its default assignment for
|
||||
the operator wherever `nox` is present and gains a second one run for `nox`, which makes the agent's forge key in the
|
||||
agent's home and closes issue 989; gitea's provider registers that key on the forge account `nox`. A module run for an identity calls as that
|
||||
identity and gets no module identity of its own (this narrows ADR 0386 §4, which holds for every other assignment). A module with an account of its
|
||||
own (`runs-as`, ADR 0259 §8) cannot declare it, and a run for an identity with no machine account on that machine is
|
||||
refused at composition and at start. ADR 0266's guard
|
||||
stands: on the control node, a session started for the operator gets the operator's configuration and its tool calls are still
|
||||
refused.
|
||||
|
||||
### 5.3 A bus password per identity (ADR 0413)
|
||||
|
||||
Each person and agent identity has a bus account of its own (`person.<name>` for the operator, `agent.nox`), whose memberships are made
|
||||
from its role, and its password is a secret owned by its machine account at mode 0600 on each machine it is present
|
||||
on, which no other account but root can read. A person's password is placed only on a machine that names an agent
|
||||
account of its own judged unable to become root, by a verdict at most fifteen minutes old; on the other machines, where agents can still run as the operator,
|
||||
it is not placed, and the controller says so. A module run for an identity reaches the bus with that password, can do
|
||||
only what the identity's role allows, and every call is recorded as that identity's. Nox's role never holds an approval
|
||||
grant, a trusted party's grant or a dangerous tool named in it (ADR 0387 §4 does not apply to it), and the identity seat
|
||||
refuses one; each dangerous act of nox's is asked one by one. The break-glass agent (§6) keeps its own rules and gets
|
||||
no login and no bus password from this. This changes ADR 0259 §8, which kept every bus credential out of an agent's account; the rest of §8 stands.
|
||||
The password lives in today's list of bus accounts and is replaced by a signed login for the same role at the bus
|
||||
switch (§7, §8).
|
||||
|
||||
### 5.4 Traced to GENESIS
|
||||
|
||||
Checked on 2026-10-17 (playbook 02, step 1); no conflict was found.
|
||||
|
||||
- [Mission](../../00-META/mission.md): agents "hold identity", and there is one kind of participant, some of them
|
||||
human; persons and agents are held alike, with the same fields, and differ in what their role allows, which the
|
||||
mission calls modality. "The mesh brokers credentials": the bus password per identity and the service accounts made
|
||||
from it. "The mesh owns the truth": logins are derived from identities, no longer stated per node. "Dogfood": logins
|
||||
go through the node-engine's existing resource.
|
||||
- [Context](../../00-META/context.md): "Identity is mesh-internal", with no directory and no SSO, so the identity seat
|
||||
holds it and Keycloak only receives accounts; "Nodes are personal" and the mesh is a guest, so existing logins are
|
||||
adopted, never renamed; human agents are few, often one, so the seat holds two identities and one agent.
|
||||
- [Effect](../../00-META/effect.md): a human agent's environment is part of the mesh, its login included.
|
||||
|
||||
## 6. Effects, asks and break-glass (ADR 0387)
|
||||
|
||||
Every seat verb and module tool is one permission and carries one **effect**, set in its module's manifest:
|
||||
**reads**, **acts** or **dangerous**. The catalogue's check refuses an unmarked tool, once every module already in
|
||||
the catalogue is marked. A dangerous tool is granted only by name, and naming one in a role goes to the operator as
|
||||
an ask first. Every other dangerous act is an ask (ADR 0259), bound to that act and carried out on the warrant.
|
||||
|
||||
The **break-glass** agent holds every tool and never asks. It is sealed until the operator opens it, or a condition
|
||||
the operator has named as opening it is raised (none is named at the start); it stays open until the operator closes
|
||||
it, and every act it takes is said to the operator at once. A tool served without an effect, from a module built
|
||||
before its mark, counts as dangerous.
|
||||
|
||||
## 7. Signed logins on the bus (ADR 0388)
|
||||
|
||||
The bus checks each login by its signature and reads no list. The root key is sealed in the vault and used only to
|
||||
replace the identity seat's signing keys, on the operator's Approve. The seat signs with one scoped signing key per
|
||||
role, so every login stays small and each identity gets its own inbox from the role's template. An agent's login
|
||||
lives minutes, a module's and a node's hours; each is renewed by a reconnect forced before expiry, by the handler or
|
||||
the node-engine.
|
||||
|
||||
A switched-off identity is revoked over the bus. The seat reads back every update, sends updates at least a second
|
||||
apart, and resends the current state to every server id it has not seen on the system account, which every joined
|
||||
server carries. This changes ADR 0106's rule that the bus is configured only through its settings file.
|
||||
|
||||
## 8. The switch (ADR 0389)
|
||||
|
||||
A new bus with signed logins runs beside today's, joined as a leaf node carrying the system account. Identities move
|
||||
one at a time, each checked before the next. Streams, key-value buckets and the object store move last, each by a
|
||||
copy of its whole configuration: copy, wait for equal counts, stop its writers, then one update hands the addresses
|
||||
over. Today's bus is switched off when nothing is left on it.
|
||||
|
||||
## 9. Sandboxes (ADR 0390)
|
||||
|
||||
Each session runs in a microsandbox of its own, its version pinned and declared in a module. The workspace is
|
||||
mounted; the claude-code module writes in the agent's login and the licence token without its refresh part, and
|
||||
replaces the token as the licence manager refreshes it. Connections are allowed only to the vendor's interface, by
|
||||
address range, and to the bus. A node whose KVM modules are not loaded gets them from the sandbox module.
|
||||
|
||||
## 10. The build's acceptance tests
|
||||
|
||||
The build is not done until each of these has been shown, and the milestones of the work breakdown (playbook 04)
|
||||
carry them:
|
||||
|
||||
1. A licence token replaced under a running session keeps that session working.
|
||||
2. The sandbox module loads KVM on the home server, and a sandbox runs there.
|
||||
3. Every existing module's tools carry an effect, and the catalogue's check refuses an unmarked one.
|
||||
4. The switch, run on the mesh's own modules in the lab from the bed test kept on the lab's branch
|
||||
`research/053-signed-bus-logins`, loses nothing and leaves every identity on the new bus.
|
||||
5. The acceptance sentence at the top, end to end.
|
||||
|
||||
From research 057 (ADR 0411 to ADR 0414):
|
||||
|
||||
6. The identity seat lists the operator and `nox` with their machines, kept logins and the control node's agent account mark; a
|
||||
create or a presence change by any caller but the operator's Approve is refused.
|
||||
7. On every other machine a login `nox` exists; on the control node the login `nox` is adopted with its uid and home unchanged;
|
||||
on every machine the node-engine judges `nox` unable to become root; a login `nox` given a sudo rule in a test is left
|
||||
unaltered and the self-check raises `agent-can-become-root` for it, on a machine that marks no agent account too;
|
||||
the operator's existing logins are
|
||||
unchanged.
|
||||
8. No manifest in the catalogue declares a login for a person or an agent; the catalogue's check refuses one.
|
||||
9. For every machine, the declaration before and after differs, for the modules not set to run for an identity, only
|
||||
in the machine accounts' `user` resources (now the controller's, and `nox` added on the other machines); claude-code on the control node, run for `nox`, places the same
|
||||
files in the same home; on the other machines the guard stays off and the operator's sessions keep the mesh's tools.
|
||||
10. A session started for `nox` on the control node gets the agent's home configuration and one started for the operator the
|
||||
operator's, with no tool call through; a session as `nox` that starts claude-code for the operator is refused; with
|
||||
ssh-client run for `nox`, nox's key is on the forge account `nox` and the session pushes a branch over SSH as `nox`.
|
||||
11. Nox's bus password is owned by `nox` at mode 0600 on every machine nox is present on; a call with it outside
|
||||
nox's role is refused; a grant test finds no approval grant, trusted party's grant or dangerous tool in nox's
|
||||
role; the operator's password is placed only on a machine with an agent account judged root-free within fifteen
|
||||
minutes, and the controller says so for every other.
|
||||
12. The forge account `nox-agent` is renamed `nox`, keeps every machine's key, stays in the agents' write team, and
|
||||
every `forge-account` entry names `nox`.
|
||||
|
||||
## 11. Milestones
|
||||
|
||||
Each milestone is one feature step (playbook 07), usable as soon as its delivery plan is done, and starts when
|
||||
the previous one's is. What it replaces is in the as-is documents
|
||||
[15 — The agent and its licences](../00-as-is/15-the-agent-and-its-licences.md) and
|
||||
[16 — Logins on the bus](../00-as-is/16-logins-on-the-bus.md), and for milestones 11 to 14
|
||||
[17 — Accounts on a machine](../00-as-is/17-accounts-on-a-machine.md). The issues are opened once the operator confirms
|
||||
the milestones, and their numbers written here.
|
||||
|
||||
**Confirmed by the operator on 2026-10-16.** One issue per milestone: 1 is issue 878, 2 is 879, 3 is 880, 4 is
|
||||
881, 5 is 882, 6 is 883, 7 is 884, 8 is 885, 9 is 886, 10 is 887. A milestone's issue is split into one issue per
|
||||
repository part when its feature step starts (playbook 07): several of the parts are modules that do not exist yet,
|
||||
and the issue tracker takes only registered owners. The building waits until the delivery under way of
|
||||
[57 — Building config and provisioning](57-building-config-and-provisioning.md) is done; the status here stays
|
||||
`designed` until milestone 1 starts.
|
||||
|
||||
| # | Outcome, in one sentence | Proves | Repositories and their parts | Depends on |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Every tool on the mesh says whether it reads, acts or is dangerous, a tool's description shows it, and the catalogue refuses a module that leaves one unmarked | test 3 | mesh-sdk: the effect in a tool's declaration, an unmarked tool served as dangerous; mesh-controller: the manifest field; mesh-catalog and mesh-media-catalog: every module marked, the check | — |
|
||||
| 2 | A session works on a repository's workspace through the mesh's tools alone: read, glob, search, edit, write, git, its tests and its merge check, with no command runner | — | mesh-catalog: the module `nox-workspace` on the node seat `workspace`, on the always-on nodes; mesh-controller: the seat | 1 |
|
||||
| 3 | A thread created through the thread seat runs headless on an always-on node, its conversation readable through the seat, its usage counted per thread, and a move to another node loses at most the turn in progress | — | mesh-catalog: `nox-threads` (seat `thread`, restricted streams, leases), `nox-thread-handler` (seat `thread-handler`), claude-code (mirror and rebuild), claude-licence-manager (usage per thread) | 2 |
|
||||
| 4 | The operator starts, follows, answers and stops a thread from a terminal client, and answers its questions on Telegram | — | mesh-catalog: the terminal client, a Go program of `nox-threads`; claude-code: questions and permission requests through the asks | 3 |
|
||||
| 5 | Every session runs in a sandbox of its own that reaches only the vendor and the bus, and the home server runs one too | tests 1 and 2 | mesh-catalog: the sandbox module (microsandbox pinned, KVM loaded where off); `nox-thread-handler`: runs sessions in it; claude-code: writes the login and the licence token without its refresh part | 3 |
|
||||
| 6 | Every person, agent and module is an identity with a role, and an agent's call outside its role is refused, saying which grant is missing | — | mesh-catalog: the identity seat's module; mesh-sdk: the tool runner's exact check, calls carrying thread and agent token; mesh-controller: an identity asked for each assignment; mesh-tools: sessions call as a person or an agent, never through the tool runner's login | 1 |
|
||||
| 7 | A dangerous act not named in a role comes to the operator as an ask, and the sealed break-glass agent opens only on the operator's word or a named condition | test 5 | mesh-sdk: dangerous calls asked; identity module: dangerous tools granted by name on an ask, break-glass and its telling | 6, 5, 4 |
|
||||
| 8 | The bus checks signed logins on a second bus beside today's, the identity seat signs and revokes them, and the switch, run in the lab on the mesh's own modules, loses nothing | test 4 | identity module: signing keys, renewal, revocation read back and resent to new servers; the node-engine (`mesh-host`): node and module logins renewed; mesh-controller: the second bus as a leaf with the system account; mesh-lab: the bed test from the branch `research/053-signed-bus-logins` | 6 |
|
||||
| 9 | Every identity is on the new bus, its data moved by copy, and today's bus is switched off | — | mesh-controller and the identity module: one identity at a time, then streams, buckets and the object store; mesh-catalog: the old bus's module retired | 8 |
|
||||
| 10 | The operator follows and answers threads from the web page as from the terminal client, logged in as a person identity | — | mesh-catalog: the web page, a client of the thread seat over the bus's WebSocket listener | 9, 4 |
|
||||
|
||||
The front ends' later parts, the inference seat and the other agent harnesses are tracked by issue 850; the
|
||||
seat-sense "role" in older designs by issue 872.
|
||||
|
||||
**Milestones 11 to 14, from research 057, confirmed by the operator on 2026-10-17.** One issue per repository
|
||||
part, opened at the confirmation, so a fixing agent per repository can start when its step does. Milestone 11
|
||||
waits for nothing; 12 waits for milestone 6.
|
||||
|
||||
| # | Outcome, in one sentence | Proves | Repositories and their parts (issues) | Depends on |
|
||||
|---|---|---|---|---|
|
||||
| 11 | The agent's forge account is `nox`, with its keys and its place in the agents' write team kept | test 12 | mesh-catalog gitea: the rename, keys registered on `nox`, every `forge-account` entry (993) | — |
|
||||
| 12 | The mesh makes the operator's logins and nox's on every machine from the identities, and judges nox unable to become root everywhere | tests 6 to 9 | identity module: logins, machines, kept logins, the agent account mark, the operator's verb (994); the node-engine (`mesh-host`): existing logins adopted, every agent login judged (995); mesh-controller: machine accounts composed, node accounts read from the identities, the terminal lines retired (996); mesh-catalog: claude-code declares no login, the catalogue's check refuses one (997) | 6 |
|
||||
| 13 | The operator and nox each reach the bus with their own password, held to their role; the operator's only where no agent runs as the operator | test 11 | mesh-controller: the operator's and nox's bus accounts, secrets at 0600, the fifteen-minute verdict (999); identity module: nox's role refuses approvals and dangerous tools (998); until milestone 7 builds the asks, a dangerous act of nox's is refused, not asked | 12 |
|
||||
| 14 | The operator starts a claude-code session for nox or for themself, and nox pushes over SSH with its own key | test 10 | mesh-controller: run for, assignments keyed by identity, the refusals (1001); mesh-sdk: calls as the identity, a start goes down only (1002); claude-code: started for nox or for the operator (1003); ssh-client and gitea: nox's own key, closing issue 989 (1004) | 11, 13 |
|
||||
|
||||
The agent's and the operator's accounts in Keycloak and mail (ADR 0414 §2 and §3) follow milestone 13 and are
|
||||
tracked by issue 1000, which needs a design section first. A second agent identity is made only when a rule
|
||||
needs another role (ADR 0414 §1); no milestone makes one.
|
||||
@@ -47,15 +47,17 @@ document is written and this one's status becomes `implemented`.
|
||||
| [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | ADR 0173, ADR 0182, ADR 0205 |
|
||||
| [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | ADR 0227, ADR 0224, ADR 0218 |
|
||||
| [`46-the-conversation-with-the-operator.md`](46-the-conversation-with-the-operator.md) | **Designed.** The mesh tells and asks its operator over channels that are holders of two kinded benches, `channel` and `intake`, declaring capabilities from a fixed vocabulary; the router orders them by work context and never lowers the bar; an answer that performs an action is checked and performed by the controller, on a TOTP code or a verified Telegram sender, never on a desk click alone; operator messages addressed or answered by the mesh's responder, untrusted input kept as data, references for what words may not carry, and a recoverable factor; Telegram first, in six phases | ADR 0234, ADR 0227 |
|
||||
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the walk, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
|
||||
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the rollout, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
|
||||
| [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of eleven domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | ADR 0244, ADR 0006, ADR 0008, ADR 0302 |
|
||||
| [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | ADR 0247, ADR 0223 |
|
||||
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | ADR 0251, ADR 0189 |
|
||||
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | ADR 0263, ADR 0051, ADR 0092 |
|
||||
| [`53-the-issue-tracker.md`](53-the-issue-tracker.md) | **Designed.** The mesh keeps its issues in one module, `issues`, on the control-node: every number from one counter advanced by compare-and-set, the cycle's issue rules refused at the write, an event per change; an issue lives whole in the tracker, hq keeping no issue content (ADR 0291) | ADR 0275, ADR 0291 |
|
||||
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, walks, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
|
||||
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, rollouts, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
|
||||
| [`55-the-knowledge-base.md`](55-the-knowledge-base.md) | **Designed.** The mesh keeps its decision records and research efforts whole in one module, `mesh-knowledge`, on the control-node: number, status, topic, typed links both ways, each decision's clauses with their own status, and the reasoning, with every version; a record is proposed, checked at the write, reviewed by the knowledge base's own judgement and accepted there, with the operator's Approve when it needs it; agents ask what holds in one bounded call | ADR 0290 |
|
||||
| [`56-the-mail-client-set-up-by-the-mesh.md`](56-the-mail-client-set-up-by-the-mesh.md) | **Designed.** On every node with a display server, the `betterbird` module installs Betterbird from a pinned upstream release mirrored into the artifact store, ships the Owl add-on pinned and installed by Betterbird's enterprise policy, and declares the operator's mail accounts in a trusted setting without credentials; each account signs in once at a person's first use, and one the profile already holds is adopted by its address | ADR 0316, ADR 0317 |
|
||||
| [`57-building-config-and-provisioning.md`](57-building-config-and-provisioning.md) | **Designed.** The work of ADR 0348 and ADR 0349 as 37 packages, each an issue with its proof: foundations, template files, config in its own module, the provisioner and its client, secrets in the vault, bindings from providers, collections, a module's runtime readers, and removing the old forms; at most five worked at once | ADR 0348, ADR 0349 |
|
||||
| [`58-agent-work-as-the-meshs-own.md`](58-agent-work-as-the-meshs-own.md) | **Designed.** Agent work as the mesh's own: a thread seat keeps every thread whole on the bus; a thread handler runs the harness headless in a sandbox per session, the workspace on the thread's own forge branch; a workspace toolkit and named tools, no general command runner; one identity seat for persons, agents and modules, roles of grants, every tool an effect, signed bus logins, and the switch to them one identity at a time | ADR 0383, ADR 0384, ADR 0385, ADR 0386, ADR 0387, ADR 0388, ADR 0389, ADR 0390 |
|
||||
|
||||
## Not yet written
|
||||
|
||||
|
||||
+1
-1
@@ -28,7 +28,7 @@ for _, n := range nodes {
|
||||
```
|
||||
|
||||
So a provider on the same machine is never a `Provider` in `world.Offered`, never becomes a
|
||||
`Needed`, and the credential loop — which walks `resolved.Needs` — has nothing to walk. Every step
|
||||
`Needed`, and the credential loop — which traverses `resolved.Needs` — has nothing to traverse. Every step
|
||||
is individually reasonable and the sum is a silent gap.
|
||||
|
||||
## Why it was not noticed
|
||||
|
||||
@@ -92,7 +92,7 @@ node that looks like a real one.
|
||||
|
||||
## Fixed
|
||||
|
||||
Needs fan out per consuming module in one place, after the resolution walk. The credential's key
|
||||
Needs fan out per consuming module in one place, after the resolution rollout. The credential's key
|
||||
gains the consuming module, the grant file is named after both halves of the consumer, needs are
|
||||
matched by provision *and* module, and the provisioners name the role and the access key after the
|
||||
module rather than the machine. The refusal is gone because there is nothing left to refuse.
|
||||
|
||||
+1
-1
@@ -87,7 +87,7 @@ too new to have anybody watching it.
|
||||
## Fixed
|
||||
|
||||
**The registry module names its image and is added, not built.** The lab's artifact-store test now
|
||||
walks the only path open to a real first mesh — the manifest goes in directly, no builder and no
|
||||
rollouts the only path open to a real first mesh — the manifest goes in directly, no builder and no
|
||||
store involved — and passes.
|
||||
|
||||
**And the cycle is refused where it is written.** A manifest that provides `artifact-store` and
|
||||
|
||||
@@ -45,7 +45,7 @@ field it does not know — whole. That was
|
||||
mesh now records which host each machine reports and `status` names every machine running an older one
|
||||
than another does.
|
||||
|
||||
So the flag day is visible before it is walked into, which it was not when this was filed. It does not
|
||||
So the flag day is visible before the mesh runs into it, which it was not when this was filed. It does not
|
||||
make the field free: **the oldest host in the mesh is still what the mesh may send**, and one machine of
|
||||
four is behind today. A sequence that an old host refuses takes that machine out of the mesh's reach
|
||||
entirely — worse than the replay it prevents, which has never been observed.
|
||||
|
||||
@@ -18,7 +18,7 @@ half-understood declaration off a machine, and exactly what makes this expensive
|
||||
|
||||
[Issue 087](../087-the-controller-cannot-tell-a-host-is-too-old/00-report.md) is resolved: the mesh now
|
||||
records the host version each machine reports and `status` names every machine running an older host
|
||||
than another does. The flag day is visible before it is walked into, which it was not on 2026-09-23.
|
||||
than another does. The flag day is visible before the mesh runs into it, which it was not on 2026-09-23.
|
||||
|
||||
That makes the cost measurable rather than hypothetical, and the measurement is the reason this is not
|
||||
being built today.
|
||||
|
||||
@@ -72,7 +72,7 @@ what the sidecar is has nowhere in the design layer to look, which is how this w
|
||||
resource type no decision records and contradicting an accepted one. The rule is "no design
|
||||
without a decision"; the check is "no design without *a* decision." An unenforced rule is
|
||||
indistinguishable from a wrong one, and these two documents are what that gap looks like when
|
||||
something walks through it.
|
||||
something rollouts through it.
|
||||
|
||||
## Open questions
|
||||
|
||||
|
||||
@@ -92,7 +92,7 @@ built from a repository elsewhere, or a mesh whose forge module is down — and
|
||||
event performs, so the two cannot disagree about what "moved" means.
|
||||
|
||||
**Bases first needed edges, and there were none.** The order this report asked for was written and
|
||||
walked a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
|
||||
traversed a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
|
||||
digest was never in the file the builder read edges from. A build now reports what it was handed, the
|
||||
control plane records it by artifact path, and the order is read from each module's newest build.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ Every node's resolution is read first, and the names are attributed across them
|
||||
terminus serves the name**: among the providers a name reaches, the one that is not itself published
|
||||
under a labelled name through another provider. An identity provider is routed through the proxy and
|
||||
so is a consumer of names, not their end; the proxy contributes no label to anyone and is. The rule
|
||||
knows nothing of what "route" means — it reads the graph the modules declared — and it walks nodes,
|
||||
knows nothing of what "route" means — it reads the graph the modules declared — and it rollouts nodes,
|
||||
requirements and contributions in order, so one mesh yields one region. Every shape of contribution
|
||||
is counted, so a module routed under several names has every one of them resolved.
|
||||
|
||||
|
||||
+1
-1
@@ -73,5 +73,5 @@ give.
|
||||
|
||||
All three faults are closed on the controller's main: one ask in flight (8d4e940), a merge rebuilds what
|
||||
stands on it (7e701c0), and a merge produces a tiered plan the mesh keeps, read with `plans` (d2ed6d5,
|
||||
76f2756, ADR 0162). Plans were since carried into walks and deliveries (ADR 0236, ADR 0239); the order a
|
||||
76f2756, ADR 0162). Plans were since carried into rollouts and deliveries (ADR 0236, ADR 0239); the order a
|
||||
person kept is the order the tiers give.
|
||||
|
||||
+1
-1
@@ -86,6 +86,6 @@ machine. That is the path the console exists to replace, so it is a workaround a
|
||||
Both pull requests named above are merged and live. Read through the console on 2026-10-08:
|
||||
`mesh_describe mesh-controller.plan` declares `node` (required) and `files`; `mesh-controller.push`
|
||||
declares `node`, `behind`, `cause` and `why`, and its description says that a push naming no machine
|
||||
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` walks every verb it serves:
|
||||
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` exercises every verb it serves:
|
||||
an argument a verb does not declare is refused, and every flag of a verb's command is in its schema
|
||||
or accounted for.
|
||||
|
||||
+1
-1
@@ -46,7 +46,7 @@ Pull requests: novox/mesh-tools#14 and novox/mesh-controller#71 — open, not me
|
||||
first line of its answer, that it is a push of the whole mesh and which machines it is sending, and
|
||||
its last line names the machines told. `plan` gains `files`, `push` gains `behind` (the whole mesh
|
||||
said outright, refused beside `node`), and the listing verbs a `limit`.
|
||||
- **Checked by tests that walk every verb the controller announces:** for every combination of a
|
||||
- **Checked by tests that exercise every verb the controller announces:** for every combination of a
|
||||
verb's declared arguments, the verb either refuses or every argument given changes the command it
|
||||
runs; every verb refuses an argument it does not declare, and `node` where it takes none; and every
|
||||
flag of the command a verb runs — read from the command's own source — is an argument of the verb's
|
||||
|
||||
@@ -26,7 +26,7 @@ list alone. The flush has no such narrowing.
|
||||
|
||||
## What it costs
|
||||
|
||||
`record` is the policy for a change that must be walked through the mesh by hand. A named push cannot
|
||||
`record` is the policy for a change that must be rolled out through the mesh by hand. A named push cannot
|
||||
do that, so the policy does not hold the change back from anything but the merge. The command's name
|
||||
says one machine, and the command's output is the only place that says otherwise.
|
||||
|
||||
|
||||
@@ -61,7 +61,7 @@ The mesh-delivery tests `checkask_test.go` cover:
|
||||
- An announcement that carries a decided gate status uses it.
|
||||
- H2 re-asks once, a second re-ask is refused by the table, and a late verdict still settles the delivery.
|
||||
|
||||
The table tests walk the new row, and the pairs the table does not hold are refused by name. Live: once
|
||||
The table tests exercise the new row, and the pairs the table does not hold are refused by name. Live: once
|
||||
the fix is rolled out, the stalled delivery has its check asked at the owner's first tick. The controller
|
||||
journal then shows its verdict, and `show` moves from proposed to checked to ready without a person
|
||||
acting.
|
||||
|
||||
@@ -39,7 +39,7 @@ The mesh's own record shows it:
|
||||
- **What each machine reports.** `postgres.server`, `mongodb.server` and nine `mailu.*` containers
|
||||
have run only since 18:57 on the control node, and `postgres.server` since 18:56 on the anchor.
|
||||
|
||||
No walk held back for later was involved. The sends were the plan's own sends to its first machines.
|
||||
No rollout held back for later was involved. The sends were the plan's own sends to its first machines.
|
||||
|
||||
## Why it happened
|
||||
|
||||
@@ -53,7 +53,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
|
||||
push". So the gate did not judge the recorded builds, the refusal of ungated sends did not see
|
||||
them, and the gated send carried them. `record` held only against the end of a person's push to
|
||||
another machine (issue 259's `heldBack`) and against the bus (the bus step). It did not hold
|
||||
against a plan's walk, a rollback, a healer or a rotation.
|
||||
against a plan's rollout, a rollback, a healer or a rotation.
|
||||
3. **Nothing tells a person, before or during a send, that it will recreate every container of a
|
||||
module at once.** A change that only alters how containers are declared reads as harmless. A
|
||||
health check, an environment variable or a label each recreates the container all the same,
|
||||
@@ -64,7 +64,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
|
||||
## What is at stake
|
||||
|
||||
`record` is how the mesh keeps the providers, the network path and the irreplaceable data out of
|
||||
unattended walks. Tonight showed that it holds only until some other module on the same machine
|
||||
unattended rollouts. Tonight showed that it holds only until some other module on the same machine
|
||||
rolls out. This was still true after the incident: plex (`record`, irreplaceable data) was rebuilt
|
||||
by a media-catalogue merge at 19:01 and sent nowhere. The next gated send to the anchor, for any
|
||||
module, would recreate it.
|
||||
|
||||
@@ -13,8 +13,8 @@ the tier durations measured for the repository: three times their ninetieth perc
|
||||
than 30 minutes. So the watchdog and the line a person reads used different start times and could use
|
||||
different bounds. The line was the one that was wrong.
|
||||
|
||||
The line for a walk waiting on its delivery's word had the same flaw. It said "for" the time since the
|
||||
last save. The watchdog for that wait (S16) counts from when the walk was opened.
|
||||
The line for a rollout waiting on its delivery's word had the same flaw. It said "for" the time since the
|
||||
last save. The watchdog for that wait (S16) counts from when the rollout was opened.
|
||||
|
||||
Ruled out: a clock difference between the machine running the verb and the store. The count was small
|
||||
and kept growing again from zero, which only a column being rewritten explains.
|
||||
@@ -44,8 +44,8 @@ nothing.
|
||||
## Fix (mesh-controller)
|
||||
|
||||
- The line, the `since` and `late` fields of `status --json`, and the count of late plans all start from
|
||||
when the plan entered its tier. If the delivery let the walk go after that, they start from the go. A
|
||||
walk still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
|
||||
when the plan entered its tier. If the delivery let the rollout go after that, they start from the go. A
|
||||
rollout still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
|
||||
called late. A plan read without the tier stamp falls back to its last save.
|
||||
- LATE uses the bound the watchdog uses: the measured bound for the plan's repository, or the 30-minute
|
||||
floor when nothing has been measured. The watchdog's start time and bound now come from the same two
|
||||
@@ -53,4 +53,4 @@ nothing.
|
||||
- The loop compares the plan before and after each step and saves only when they differ. The fields the
|
||||
store stamps itself are left out of that comparison.
|
||||
- A refused step is logged and saved only when its note is new. The refusal's wording called the gated
|
||||
walk a "release plan", a name the glossary retires; it now says "walk".
|
||||
rollout a "release plan", a name the glossary retires; it now says "walk" (since renamed rollout, issue 646).
|
||||
|
||||
@@ -23,7 +23,7 @@ another, with nothing on the mesh noting the difference.
|
||||
|
||||
ADR 0238 makes the delivery plan the planner's one answer for a pull request's check, the merge gate and
|
||||
the merge. The plan counts a new module's directory as something the merge builds. The merge handler
|
||||
read the same answer (`touchedBy`'s `added`) and dropped it: it only walks modules the catalogue already
|
||||
read the same answer (`touchedBy`'s `added`) and dropped it: it only rollouts modules the catalogue already
|
||||
holds, and a new one is not held until it is built. The code said so outright: "a new module, which a
|
||||
check judges before it merges and a merge does not build". The take-in refuses a build off the trunk
|
||||
with "merge it, and the merge builds it". So one part of the controller promised the build and another
|
||||
|
||||
@@ -29,7 +29,7 @@ already built from it are, and without waiting (`askNewModules`). The build's ta
|
||||
module, as it does for a hand `build`. Nothing is sent: no machine is assigned a new module, and
|
||||
assigning one stays a person's act.
|
||||
|
||||
The new module is outside the merge's plan. A plan walks modules the catalogue holds, tier by tier, and
|
||||
The new module is outside the merge's plan. A plan rollouts modules the catalogue holds, tier by tier, and
|
||||
sends each built one to the machines running it. A new module has no machine and nothing standing on it,
|
||||
and the plan's ask for a module the catalogue does not hold fails the plan.
|
||||
|
||||
|
||||
+9
-9
@@ -7,16 +7,16 @@ opened: 2026-10-08
|
||||
## Symptom
|
||||
|
||||
On 2026-10-08 an agent proving the tunnel join (novox/mesh-lab PR #61) on the laptop could not run the
|
||||
two-node walk, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
|
||||
two-node rollout, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
|
||||
reported by any check before someone tried to run a bed:
|
||||
|
||||
1. **The walk's scenario stocks a packet-filter runtime image that cannot be built.**
|
||||
1. **The rollout's scenario stocks a packet-filter runtime image that cannot be built.**
|
||||
`scripts/build-module-runtime.sh` copied the tool runner's compiled TypeScript and its `node_modules`
|
||||
out of mesh-tools. mesh-tools has had no package at its root since the runtime became `node-tools`, a
|
||||
Go binary that launches every bundle (ADR 0175,
|
||||
ADR 0193),
|
||||
so the script failed at its first step.
|
||||
2. **The walk's builder dialled the old broker.** It was started with the old bus's guest account on
|
||||
2. **The rollout's builder dialled the old broker.** It was started with the old bus's guest account on
|
||||
loopback. The mesh moved its bus to NATS (ADR 0106,
|
||||
ADR 0127,
|
||||
ADR 0131), and the builder
|
||||
@@ -27,8 +27,8 @@ reported by any check before someone tried to run a bed:
|
||||
4. **The replays prover could not run a test in mesh-tools**, because that repository's Go module lives
|
||||
in a subdirectory (`node-tools/`), and the prover ran `go test` only at a repository's root.
|
||||
|
||||
The same day, while trying to run the walk again, two more gaps surfaced: the object store's
|
||||
provisioner image, which the walk also stocks, cannot be built, because the client image it copies
|
||||
The same day, while trying to run the rollout again, two more gaps surfaced: the object store's
|
||||
provisioner image, which the rollout also stocks, cannot be built, because the client image it copies
|
||||
from is gone upstream; and the laptop could not reach the public container registry, which the lab's
|
||||
egress check needs (an outside condition, not the lab's fault).
|
||||
|
||||
@@ -56,9 +56,9 @@ In novox/mesh-lab PR #63, stacked on #61:
|
||||
builds `node-tools` from mesh-tools' Go module, and runs it as the image's entrypoint. The image serves
|
||||
on the node's runtime credential: the Go runtime refuses to serve a module on that module's own
|
||||
credential.
|
||||
- **The walk no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
|
||||
- **The rollout no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
|
||||
names an image (its verbs are a tools bundle the node's runtime launches), so the image was never run.
|
||||
- **The walk's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
|
||||
- **The rollout's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
|
||||
an own `broker` secret, as the catalogue's build agent does. Assigning it to the anchor issues the
|
||||
account ([issue 203](../203-a-fresh-assignment-is-pushed-before-its-credential-exists/00-report.md)).
|
||||
The bus's user list is placed, the push writes the sealed credential, and the builder starts on it.
|
||||
@@ -71,12 +71,12 @@ In novox/mesh-lab PR #63, stacked on #61:
|
||||
Still open:
|
||||
|
||||
- `builds.test.ts` still dials the old broker. Its bed enrols no machine, so the mesh has nowhere to
|
||||
deliver a builder's credential. It needs the walk's enrolment, or it should be retired into the walk.
|
||||
deliver a builder's credential. It needs the rollout's enrolment, or it should be retired into the rollout.
|
||||
- Beds that run a per-module runtime container on the module's own credential (the tools beds for the
|
||||
wiki and the code forge, and the audit bed) run into the runtime's refusal. They need the node's
|
||||
runtime credential instead.
|
||||
- The object store provisioner's image needs a client source that still exists.
|
||||
- The walk has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
|
||||
- The rollout has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
|
||||
public container registry.
|
||||
|
||||
## How it is checked
|
||||
|
||||
+5
-5
@@ -14,25 +14,25 @@ script now does those steps by hand. Verified: the image for the packet filter b
|
||||
answers `initialize` and `tools/list`; `node-tools` in it, started against a real bus with a plain URL,
|
||||
launches the bundle and serves nothing until the mesh issues a membership, which is correct.
|
||||
|
||||
**Why the walk stocked that image at all.** The tunnel branch passed the held images to
|
||||
**Why the rollout stocked that image at all.** The tunnel branch passed the held images to
|
||||
`deriveTheFilterOn`, which turns the catalogue's manifest into the lab's form: build section gone, each
|
||||
container's artifact replaced by a stocked image. The catalogue's nftables manifest has no container,
|
||||
so no image is referenced. Removing the build section is what the bed needed; the stocked image was
|
||||
not.
|
||||
|
||||
**The builder.** mesh-controller's `cmd/mesh-builder` reads `MESH_BROKER_FILE` and nothing else for the
|
||||
bus. Without it, it says it has no credential. The walk set the old broker's variable, which the builder
|
||||
bus. Without it, it says it has no credential. The rollout set the old broker's variable, which the builder
|
||||
no longer reads. A builder's account is a module's account (`builder issue` and `module issue` both mint
|
||||
through the inventory), composed into the bus's user list and sealed to the machine as the module's
|
||||
`broker` own secret. Assigning a module that declares that secret now issues it in the same act (issue
|
||||
203). The catalogue's build agent is that module, but it runs the builder as a container from an
|
||||
artifact the mesh builds, and it needs the artifact store and package registry bound. The walk has
|
||||
artifact the mesh builds, and it needs the artifact store and package registry bound. The rollout has
|
||||
neither before it can build, so it declares a lab module with the same claim and the same secret. In
|
||||
this bed the bus comes from the bundle, so its user list is placed by hand, the step the walk already
|
||||
this bed the bus comes from the bundle, so its user list is placed by hand, the step the rollout already
|
||||
takes at enrolment.
|
||||
|
||||
**The uplink and the prover** were each fixed by the agent that met them (mesh-lab #61 and #62), and are
|
||||
listed here only so the lab's state on that day is in one place.
|
||||
|
||||
**Ruled out.** That the walk failed because of the tunnel changes. The builder and runtime faults are in
|
||||
**Ruled out.** That the rollout failed because of the tunnel changes. The builder and runtime faults are in
|
||||
code that #61 does not touch, and they reproduce on main.
|
||||
|
||||
@@ -21,7 +21,7 @@
|
||||
the planner's sort stalled. Every two-way pair was a cycle, whatever the rules.
|
||||
- The person's order, the controller first, was right for this change. The new node-engine calls a verb
|
||||
the new controller adds, which is ADR 0246's case: a verb is promised before it is required.
|
||||
*Engine before controller* comes from ADR 0236's order for its walks, where the node-engine had to
|
||||
*Engine before controller* comes from ADR 0236's order for its rollouts, where the node-engine had to
|
||||
accept something new first. The rule reads only module names, so it cannot tell the two cases apart. *Built
|
||||
by* is read from the graph.
|
||||
|
||||
|
||||
@@ -59,7 +59,7 @@ and fails only when it cannot read.
|
||||
|
||||
**Merged.** mesh-controller#135 (the build agent judges a pull request by the base branch's
|
||||
`merge-check.sh`) merged on 2026-10-08 at 09:10 UTC, and mesh-lab#65 (R310) at 09:10 UTC. Every later
|
||||
controller delivery carries it; the last, of mesh-controller#170, finished its walk on 2026-10-09 at
|
||||
controller delivery carries it; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at
|
||||
00:11 UTC, and `mesh-controller.status` named no machine behind.
|
||||
|
||||
**The open half is closed.** `gitea_branch_protection_get` for the media catalogue's main, read on
|
||||
|
||||
+5
-5
@@ -32,11 +32,11 @@ was asked of a new, unrelated pull request.
|
||||
## Why it is a design issue
|
||||
|
||||
ADR 0239
|
||||
decision 10, *the switch*, adopts the walks open when mesh-delivery is assigned. It says nothing of the merges the
|
||||
decision 10, *the switch*, adopts the rollouts open when mesh-delivery is assigned. It says nothing of the merges the
|
||||
bus still holds from before: the forge's `pull.merged` is kept on the bus and heard again by a new consumer.
|
||||
Each was taken as a merge just made whose head was never heard, so the table's row *merged unchecked → held*
|
||||
held each for a person. The table had no state for "this already happened", and `held`'s only ways on were
|
||||
a release, which would walk it, and a stop, which says a person stopped a delivery that was never running.
|
||||
a release, which would roll it out, and a stop, which says a person stopped a delivery that was never running.
|
||||
Healer H2's `close` has no transition from `held`, rightly: a held delivery is the operator's.
|
||||
|
||||
Decision 5 says what a transition owes outside is "retried until done". An effect the forge answers can never
|
||||
@@ -45,14 +45,14 @@ be done was retried without end, every ten seconds, and nothing said so.
|
||||
## Fix
|
||||
|
||||
- **A new final state, `retired`: history.** A delivery made from the forge's word at its merge, whose merge
|
||||
was heard fifteen minutes or more after it happened and whose walk, if the controller keeps one, is not
|
||||
was heard fifteen minutes or more after it happened and whose rollout, if the controller keeps one, is not
|
||||
open, is history. At adoption it goes straight to `retired` by the observed row *history*, before any check,
|
||||
hold or walk. Its pull request is not told: the forge may no longer hold it, or holds another under its
|
||||
hold or rollout. Its pull request is not told: the forge may no longer hold it, or holds another under its
|
||||
number. Its transitions are still said on the bus and kept on the commit's note.
|
||||
- **A person's verb for the ones already held: `retire-history`**, with why. One delivery by id, refused when
|
||||
the rule does not take it, or every held one (of one repository when named), answering how many it retired
|
||||
and naming each one it left held. `dry` says what it would do and changes nothing. It is the act
|
||||
*held → retired* of the table, never a release: nothing is walked, delivered or put back.
|
||||
*held → retired* of the table, never a release: nothing is rolled out, delivered or put back.
|
||||
- **Owed work ends.** The forge's answer that what an effect is for does not exist (a 404, or "does not
|
||||
exist") is final: the effect is given up with one line, kept among the owner's refusals. Any other failure is
|
||||
tried again later each time, from ten seconds doubling to an hour, and given up after sixty tries, saying so.
|
||||
|
||||
+5
-5
@@ -5,20 +5,20 @@
|
||||
- 699 deliveries were `held`. 688 say "merged without a passing check", all made between 23:05:27 and
|
||||
23:05:29 on 2026-10-06, the owner's first start. A sample across repositories shows merges from
|
||||
2026-10-02 to two hours before that start, each with the transition "its head was not heard by this owner,
|
||||
so it is made from the forge's word", then *merged unchecked → held*. None has a walk.
|
||||
- The other 11 are held for other reasons: a merge with no walk opened within ten minutes (10), and a group
|
||||
so it is made from the forge's word", then *merged unchecked → held*. None has a rollout.
|
||||
- The other 11 are held for other reasons: a merge with no rollout opened within ten minutes (10), and a group
|
||||
whose composed check did not pass (1). They are not history, and this issue leaves them alone. One of them
|
||||
was past its bound, which makes the 689th finding.
|
||||
|
||||
**Where it comes from.** The holder subscribes to the forge's `pull.merged` on the bus after reading back its
|
||||
state and the controller's walks. The bus hands a new consumer the merges it still holds. For each, the holder
|
||||
state and the controller's rollouts. The bus hands a new consumer the merges it still holds. For each, the holder
|
||||
finds no delivery for the head, so it makes one from the forge's word. The pull request's statuses at its
|
||||
merge do not include a passing `mesh/merge-gate` (the check did not exist for most of them), so the table's
|
||||
*merged unchecked* row holds it.
|
||||
|
||||
**What would close them, and why not.**
|
||||
|
||||
- `release` walks a held delivery: wrong, these reached the machines already.
|
||||
- `release` rollouts a held delivery: wrong, these reached the machines already.
|
||||
- `stop` takes it to `stopped`, which says a person stopped a delivery and sets `mesh/delivery` to error on
|
||||
the commit. Wrong in meaning, and it would ask the forge for a view on a pull request that is gone.
|
||||
- Healer H2's `close` reads `held`'s bound, which names no H2 transition, and refuses: "the state is the
|
||||
@@ -43,6 +43,6 @@ mesh-catalog#121 (the delivery's owner retires history and ends owed work the fo
|
||||
under ADR 0250.
|
||||
|
||||
**Seen live.** `mesh-delivery.deliveries` on 2026-10-09 lists the held history as `retired`, all at
|
||||
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The walk of
|
||||
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The rollout of
|
||||
mesh-catalog#121's own delivery reads `stopped`; the holder that answers today serves the fix all the
|
||||
same, and `mesh-controller.status` names no machine behind.
|
||||
|
||||
+2
-2
@@ -53,6 +53,6 @@ lift.
|
||||
every other state in the table moves on its facts?
|
||||
3. Should `mesh/delivery-group` be required on the trunks, so a group cannot merge before its composed
|
||||
check passed? To-be 47 leaves this to the operator; this issue is the case that setting decides.
|
||||
4. The group's order put the catalogue's member first, and the node-engine's member walked at once on the
|
||||
controller's own path. Is a group's order meant to hold for a member whose walk never waits for
|
||||
4. The group's order put the catalogue's member first, and the node-engine's member rolled out at once on the
|
||||
controller's own path. Is a group's order meant to hold for a member whose rollout never waits for
|
||||
mesh-delivery?
|
||||
|
||||
+6
-6
@@ -3,7 +3,7 @@
|
||||
**2026-10-08.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`),
|
||||
from `mesh-delivery.show` of both members, and from the holder's journal on the control node.
|
||||
|
||||
**Where the hold is set.** `groupTurns` (holder.go) walks a closed group's members in order. For a member
|
||||
**Where the hold is set.** `groupTurns` (holder.go) rolls out a closed group's members in order. For a member
|
||||
in `published` it computes `composed`: the group's check exists, its verdict is `pass` or `warning`, and it
|
||||
names the members' current heads. When `composed` is false it writes the member's `HeldWhy` ("its group's
|
||||
composed check did not pass for the heads that merged") and settles it, and the table's `hold` row takes it
|
||||
@@ -20,7 +20,7 @@ to `held`. `composed` is false in three different cases, and the code does not t
|
||||
look at the members.
|
||||
- `groupTurns` skips a member in `held` (`default: return nil`), so the tick after the verdict sees the
|
||||
passing check and does nothing with it.
|
||||
- The table has two rows out of `held`: `go` (the walk started on a word that was not this owner's) and
|
||||
- The table has two rows out of `held`: `go` (the rollout started on a word that was not this owner's) and
|
||||
`release` (a person's act). There is no observed row that takes a held delivery back when the reason it
|
||||
was held is gone, and `HeldWhy` is never cleared once set.
|
||||
|
||||
@@ -41,7 +41,7 @@ check was asked. To-be 47 lists whether to require it as not decided.
|
||||
- A lost verdict. The holder logged it at 10:10:21 and the group's record holds it.
|
||||
- The order. The group's order was asked and kept before the check; it plays no part in the hold.
|
||||
|
||||
**Not this issue, noted.** The node-engine's member walked at once (`published` → `delivering` at the
|
||||
**Not this issue, noted.** The node-engine's member rolled out at once (`published` → `delivering` at the
|
||||
same instant), because the controller's own path does not wait for mesh-delivery (ADR 0239 decision 8).
|
||||
The group's order put it second. Open question 4 in the report asks whether that is intended.
|
||||
|
||||
@@ -57,9 +57,9 @@ ADR 0270
|
||||
its group's composed check is asked for the current heads and has no verdict.
|
||||
- **A hold whose reason is gone lifts, by observation.** A member held because its group's composed check had not
|
||||
passed goes back to `published` (the event *lifted*) once the check passes or warns for the heads that merged,
|
||||
while its own walk is still open. It then takes its turn in order.
|
||||
- A held member whose own walk was since taken over by a later walk is not lifted, since its walk can no longer
|
||||
deliver it. It is `covered` when that later walk was delivered and sent its module (issue 319).
|
||||
while its own rollout is still open. It then takes its turn in order.
|
||||
- A held member whose own rollout was since taken over by a later rollout is not lifted, since its rollout can no longer
|
||||
deliver it. It is `covered` when that later rollout was delivered and sent its module (issue 319).
|
||||
|
||||
Open question 3, whether `mesh/delivery-group` is required on the trunks, stays the operator's setting. Open
|
||||
question 4 is not touched.
|
||||
|
||||
+5
-5
@@ -2,7 +2,7 @@
|
||||
opened: 2026-10-08
|
||||
---
|
||||
|
||||
# 317. A held delivery's module was walked by a later merge's plan
|
||||
# 317. A held delivery's module was rolled out by a later merge's plan
|
||||
|
||||
## Symptom
|
||||
|
||||
@@ -22,7 +22,7 @@ took `openrazer` along. `mesh-controller.plans`:
|
||||
| c78b5fc9 (#124's merge) | 11:07:02 | superseded at tier 0 by the plan at 86077780; docker, gitea, openrazer planned there again |
|
||||
| 86077780 (#123's merge) | 11:07:03 | tier 0: distribution, docker, gitea and openrazer built from c78b5fc9; openrazer's gate on the laptop judging |
|
||||
|
||||
The controller's walk of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
|
||||
The controller's rollout of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
|
||||
c78b5fc9, beside `docker` and `node-tools`. From 11:10:58 UTC the laptop's condition for openrazer said
|
||||
its account was *relogin needed*. That verdict exists only in the build #125 added, so the held change was
|
||||
running on the laptop while its delivery said it waited for a person.
|
||||
@@ -41,10 +41,10 @@ gone out. A person who releases it later releases nothing; a person who stops it
|
||||
|
||||
## Open questions
|
||||
|
||||
1. When a newer merge supersedes an older plan, does a module the older plan's walk was holding for its
|
||||
1. When a newer merge supersedes an older plan, does a module the older plan's rollout was holding for its
|
||||
delivery's word come along, or stay held with that delivery?
|
||||
2. If it comes along, what does the held delivery become: superseded by the newer delivery, with the hold
|
||||
carried over to it, or something else? Today the table has no row from `held` for a superseded walk.
|
||||
carried over to it, or something else? Today the table has no row from `held` for a superseded rollout.
|
||||
3. A trunk commit holds every earlier commit, so a later build of a module carries the held change in
|
||||
any case. Is a hold on one delivery meaningful once a later merge of the same repository is walked,
|
||||
any case. Is a hold on one delivery meaningful once a later merge of the same repository is rolled out,
|
||||
or does a hold have to hold the module on that trunk until it is released?
|
||||
|
||||
+9
-9
@@ -3,7 +3,7 @@
|
||||
**2026-10-08.** Read from `mesh-controller.plans` (the list, and each plan by id), `mesh-delivery.show`
|
||||
and `deliveries`, and the code on the main branches of the controller and the catalogue.
|
||||
|
||||
**The held delivery's walk.** #125's merge made the plan at a6385479. Its walk waited for mesh-delivery's
|
||||
**The held delivery's rollout.** #125's merge made the plan at a6385479. Its rollout waited for mesh-delivery's
|
||||
word (`waited_for: mesh-delivery`, `waits: true`) and was never let go, because its delivery was held. Its
|
||||
one module, `openrazer`, was "not yet asked": nothing was built.
|
||||
|
||||
@@ -11,32 +11,32 @@ one module, `openrazer`, was "not yet asked": nothing was built.
|
||||
plan. For every open plan of the same repository and branch made before it, it takes each module the
|
||||
older plan had not built (not asked, asked and not answered, or built and not yet sent), adds it to the
|
||||
newer plan, and closes the older plan as superseded with the note "… planned there again". It does not
|
||||
ask why the older plan had not built them. A walk waiting for a held delivery's word is open, so it is
|
||||
ask why the older plan had not built them. A rollout waiting for a held delivery's word is open, so it is
|
||||
superseded like any slow plan. This happened three times over (3f552274, c78b5fc9, 86077780), each newer
|
||||
plan folding in what the one before it held.
|
||||
|
||||
The newest plan's own delivery (#123) was not held, so mesh-delivery let its walk go, and `openrazer` was
|
||||
built from c78b5fc9. The controller's walk of the builds waiting for a gate then carried it to the laptop,
|
||||
The newest plan's own delivery (#123) was not held, so mesh-delivery let its rollout go, and `openrazer` was
|
||||
built from c78b5fc9. The controller's rollout of the builds waiting for a gate then carried it to the laptop,
|
||||
since a send carries every move waiting on its machine (ADR 0236).
|
||||
|
||||
**Why the held delivery did not notice.** mesh-delivery's table has `superseded` rows only from
|
||||
`published` and `delivering` ("a newer delivery to the same trunk took over its walk"). From `held` there
|
||||
are only `go` and `release`. When the walk record said `superseded`, no row applied, so the delivery stayed
|
||||
`held` with `held_why` unchanged, and its walk shows "superseded … openrazer planned there again". Two of
|
||||
are only `go` and `release`. When the rollout record said `superseded`, no row applied, so the delivery stayed
|
||||
`held` with `held_why` unchanged, and its rollout shows "superseded … openrazer planned there again". Two of
|
||||
the three later deliveries (#122, #124) went to `superseded` correctly, through the `published` row.
|
||||
|
||||
**What the design says.** ADR 0218 decision 3 makes a newer merge take over an older plan's unfinished
|
||||
work, and was written before deliveries could be held. ADR 0239 makes a held delivery wait for a person,
|
||||
and to-be 47 says that nothing of a published delivery is registered before its walk starts, so that a
|
||||
and to-be 47 says that nothing of a published delivery is registered before its rollout starts, so that a
|
||||
send cannot carry it early. Neither says what a supersession does to a module a hold is keeping back.
|
||||
|
||||
**Ruled out.**
|
||||
|
||||
- A person's act. #125's transitions end at `hold`, with no `release`, and its own walk was superseded
|
||||
- A person's act. #125's transitions end at `hold`, with no `release`, and its own rollout was superseded
|
||||
while still waiting (`let_go` empty), so no `plans go` started it.
|
||||
- Issue 295's path, a recorded build carried by a send: here the build was asked by a plan, not left
|
||||
recorded.
|
||||
|
||||
**Located** in the controller's `supersededBy`, which folds a held walk's modules into a newer plan, and in
|
||||
**Located** in the controller's `supersededBy`, which folds a held rollout's modules into a newer plan, and in
|
||||
mesh-delivery's table, which has no way out of `held` when that happens, so the delivery's state stops
|
||||
being true.
|
||||
|
||||
+8
-8
@@ -2,11 +2,11 @@
|
||||
opened: 2026-10-08
|
||||
---
|
||||
|
||||
# 318. One module's wait for a new login held every other module's walk
|
||||
# 318. One module's wait for a new login held every other module's rollout
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-08 at 11:08 UTC the controller opened its walk of the builds waiting for a gate: the laptop
|
||||
On 2026-10-08 at 11:08 UTC the controller opened its rollout of the builds waiting for a gate: the laptop
|
||||
first, then the workstation, then the control node. On the laptop it sent three moves in one send and
|
||||
judged them together:
|
||||
|
||||
@@ -23,14 +23,14 @@ account was *relogin needed* from 11:10:58 UTC, which
|
||||
ADR 0252
|
||||
expects until the operator logs out and in again.
|
||||
|
||||
Meanwhile three other walks could not send to the rest of the mesh. `mesh-controller.plans` said of the
|
||||
Meanwhile three other rollouts could not send to the rest of the mesh. `mesh-controller.plans` said of the
|
||||
tool runner's (`novox/mesh-tools` d4845f7c, `node-tools`), the catalogue's (86077780, `docker`) and the
|
||||
controller's (950562af, `build-agent`): "a build waits there for its gate: … the workstation (…, openrazer
|
||||
88135ad0 → c78b5fc9) — a walk sends them, one machine at a time, each judged".
|
||||
88135ad0 → c78b5fc9) — a rollout sends them, one machine at a time, each judged".
|
||||
|
||||
At 11:21:06 UTC the laptop's gate failed: "not healthy within 10m0s of its apply: its unit
|
||||
openrazer.daemon … is unhealthy", and `openrazer` was put back to 88135ad0 on the laptop
|
||||
(`build.openrazer.<laptop>.rolled-back`). The walk ended `failed`. After it, all three walks still said
|
||||
(`build.openrazer.<laptop>.rolled-back`). The rollout ended `failed`. After it, all three rollouts still said
|
||||
the same: `openrazer` c78b5fc9 waits on the workstation for a gate, so nothing is sent there or to the
|
||||
machines after it.
|
||||
|
||||
@@ -40,14 +40,14 @@ The wait was known, and it was a person's: ADR 0252 says a new login holds a del
|
||||
that machine. It held every module on every machine after it instead, for changes that had nothing to do
|
||||
with it: the tool runner, the container engine and the build agent. The end was worse than the wait. The
|
||||
gate's bound turned "waiting for a person" into "failed", the build that added the account group was put
|
||||
back, and the walk that carries the other modules failed with it. A failed walk of the waiting builds
|
||||
back, and the rollout that carries the other modules failed with it. A failed rollout of the waiting builds
|
||||
does not open again on its own: a person releases it.
|
||||
|
||||
Three things meet here, each sound alone:
|
||||
|
||||
- a send carries every move waiting on its machine, and the gate judges them all at once
|
||||
(ADR 0236);
|
||||
- no other send may go where a move waits for a gate, so every walk waits for the one that judges it;
|
||||
- no other send may go where a move waits for a gate, so every rollout waits for the one that judges it;
|
||||
- a module's health that waits for a person reads as *not yet healthy*, which fails at the gate's bound.
|
||||
|
||||
## Open questions
|
||||
@@ -61,5 +61,5 @@ Three things meet here, each sound alone:
|
||||
back the account group it added, so the login the gate waited for can no longer help.
|
||||
4. The daemon's unit had failed since 10:13 UTC, almost an hour before the new build reached the laptop.
|
||||
Should a gate fail a build for a failure that was there before it was sent?
|
||||
5. While a move waits for a gate on one machine, should walks of unrelated modules still be refused there,
|
||||
5. While a move waits for a gate on one machine, should rollouts of unrelated modules still be refused there,
|
||||
or only those whose send would carry it?
|
||||
|
||||
+11
-11
@@ -1,6 +1,6 @@
|
||||
# 318 — diagnosis
|
||||
|
||||
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the walk of the waiting builds by id) between
|
||||
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the rollout of the waiting builds by id) between
|
||||
11:17 and 11:22 UTC, `mesh-controller.conditions` and `status`, and the controller's code on its main
|
||||
branch (`cmd/mesh-controller`).
|
||||
|
||||
@@ -9,15 +9,15 @@ gate all of them as carried moves. `judgeMoves` (gate.go) reads each module on e
|
||||
worst reading. Any module *not yet healthy* resets the passes to zero for the whole send: `docker` and
|
||||
`node-tools` were healthy, and the gate still said "0 of 3 passes" because `openrazer` was not. Past
|
||||
`gateBound` (ten minutes after the apply) a *not yet healthy* gate fails. `g.Failing` names only the
|
||||
modules found wanting, so only `openrazer` was put back, but the walk itself ended `failed`.
|
||||
modules found wanting, so only `openrazer` was put back, but the rollout itself ended `failed`.
|
||||
|
||||
**Everything else waits for that verdict.** `ungatedIn` (release.go) refuses any send that is not a person's
|
||||
and would carry a move no gate has seen, outside the send's own scope. A plan's sends to the rest of its
|
||||
machines carry the machine's whole declaration, so while `openrazer` c78b5fc9 waited for a gate on the
|
||||
workstation, the tool runner's, the catalogue's and the controller's walks were each refused there, and
|
||||
they say so on every retry. They do not fail; they wait for the walk of the waiting builds, which goes one
|
||||
workstation, the tool runner's, the catalogue's and the controller's rollouts were each refused there, and
|
||||
they say so on every retry. They do not fail; they wait for the rollout of the waiting builds, which goes one
|
||||
machine at a time and had stopped on the laptop. After it failed, the release code's rule applies: a
|
||||
failed walk of the waiting builds stops the next one opening on its own until a person releases it.
|
||||
failed rollout of the waiting builds stops the next one opening on its own until a person releases it.
|
||||
|
||||
**How *relogin needed* reached the gate.** ADR 0252 decision 5 says the node-engine states the account's
|
||||
health as the module's resource of kind `account`, unhealthy with *relogin needed*, and that "the
|
||||
@@ -33,8 +33,8 @@ running session lacks the group the daemon checks for. That is a guess, not chec
|
||||
readings existed before or because of the person's step, and neither is a fault the build brought.
|
||||
|
||||
**What the design says.** ADR 0252's consequences say "a new login holds a delivery of that module on that
|
||||
machine" and accept that a walk that starts on a workstation waits for the person. They do not consider the
|
||||
other modules carried in the same send, the walks refused behind it, or the gate's ten-minute bound turning
|
||||
machine" and accept that a rollout that starts on a workstation waits for the person. They do not consider the
|
||||
other modules carried in the same send, the rollouts refused behind it, or the gate's ten-minute bound turning
|
||||
the wait into a rollback. ADR 0236 makes one send judge everything waiting on a machine, to stop unjudged
|
||||
builds from moving; it does not distinguish a module waiting for a person from one that is broken.
|
||||
|
||||
@@ -50,15 +50,15 @@ Whether the account actually left the group was not checked.
|
||||
- A machine-wide fault on the laptop: its node-engine reported `applied` and current, and the gate's
|
||||
wanting named one module's unit.
|
||||
|
||||
**Located** in the controller's walk and first-node gate: one verdict per send (`judgeMoves`), sends refused
|
||||
**Located** in the controller's rollout and first-node gate: one verdict per send (`judgeMoves`), sends refused
|
||||
on any machine where a move waits (`ungatedIn`), and no reading for a health that waits for a person, so a
|
||||
known wait fails at the bound and stops every walk behind it.
|
||||
known wait fails at the bound and stops every rollout behind it.
|
||||
|
||||
**Decided and fixed, 2026-10-08**, by ADR 0254.
|
||||
Open questions 1 to 3: *relogin needed*, and a unit failing in that same account's own manager, read *waits
|
||||
for a person*, which passes with the wait carried and is said as the module's condition `relogin-needed`;
|
||||
nothing is put back for it. Question 2's other half: each module of a send counts its own passes, and, after
|
||||
the settle time, in the walk path, keeps a pass when another fails beside it. Question 5: other walks are
|
||||
the settle time, in the rollout path, keeps a pass when another fails beside it. Question 5: other rollouts are
|
||||
still refused while a move waits for a gate (ADR 0236 §4a stands). With the wait a pass, 318's case leaves
|
||||
nothing waiting for them. A fault decided before the settle time, and a plan's own tier path, still leave
|
||||
the healthy modules of a failed send without a verdict (ADR 0254, consequences). Question 4, a unit that
|
||||
@@ -71,4 +71,4 @@ between 12:41 and 13:03 UTC; the node-engine's and the controller's deliveries r
|
||||
|
||||
**Seen live.** On 2026-10-09 `mesh-controller.conditions` holds `module.openrazer.<machine>.relogin-needed`
|
||||
for the laptop and the workstation, raised at 21:02 UTC: "Its update is in place and nothing was undone",
|
||||
and no walk waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
|
||||
and no rollout waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
|
||||
|
||||
+7
-7
@@ -15,8 +15,8 @@ What they are, read with `mesh-delivery.deliveries` and `show`:
|
||||
| how many | repository | why held | what it builds |
|
||||
|---|---|---|---|
|
||||
| 4 | hq | merged without a passing check | nothing: "the change touches no module of the mesh's graph" |
|
||||
| 2 | catalogue | no walk opened within ten minutes of the merge | a new module, sent nowhere |
|
||||
| 8 | controller | no walk opened within ten minutes of the merge | the controller, the build agent, the route proxy |
|
||||
| 2 | catalogue | no rollout opened within ten minutes of the merge | a new module, sent nowhere |
|
||||
| 8 | controller | no rollout opened within ten minutes of the merge | the controller, the build agent, the route proxy |
|
||||
| 2 | catalogue | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | one module each |
|
||||
|
||||
Two examples:
|
||||
@@ -26,7 +26,7 @@ Two examples:
|
||||
`proposed` to `held`: "merged without a passing check". Its plan builds nothing.
|
||||
- `novox/mesh-controller@ec3769a8a669`, controller #105. Ready at 2026-10-07 00:26 UTC, merged at 00:27,
|
||||
held at 00:37: "merged, and the controller opened no walk for it within 10m0s". The controller's modules
|
||||
have been walked since from later commits of the same main, for example the group
|
||||
have been rolled out since from later commits of the same main, for example the group
|
||||
`feat/delivery-checks-verb`, whose controller member is `delivered`.
|
||||
|
||||
Five of them (the four hq merges and controller #105) are past the held bound of one day. The self-check's
|
||||
@@ -39,7 +39,7 @@ A held delivery asks a person "does this go on?". For these sixteen the question
|
||||
a later commit of the same trunk has already delivered what they held, or they hold nothing to deliver. The
|
||||
state table offers a person four ways out of `held`, and none fits:
|
||||
|
||||
- `release` starts a walk of the old commit: wrong for a commit a later one superseded, and nothing for one
|
||||
- `release` starts a rollout of the old commit: wrong for a commit a later one superseded, and nothing for one
|
||||
that builds nothing.
|
||||
- `retire-history` is for history made from the forge's word (ADR 0250),
|
||||
and refuses every one of these because their heads were heard.
|
||||
@@ -57,7 +57,7 @@ The issue states the need; the decision is for graduation.
|
||||
|
||||
- **An end for a held delivery that was superseded**, as `published` and `delivering` already have
|
||||
(`superseded`, "a newer delivery to the same trunk took over its walk"): either because a later commit's
|
||||
delivered walk carried the same modules on the same trunk, or because it builds nothing at all.
|
||||
delivered rollout carried the same modules on the same trunk, or because it builds nothing at all.
|
||||
- **Automatic where nothing is lost**: a held delivery whose plan builds nothing has nothing for a person to
|
||||
decide.
|
||||
- **A person's act, with why, otherwise**: a held delivery whose modules a later commit delivered is ended
|
||||
@@ -70,5 +70,5 @@ The issue states the need; the decision is for graduation.
|
||||
2. The four hq merges were heard at the owner's first start from the bus's history, as announcements rather
|
||||
than merges. Are they history in the sense of ADR 0250, missed by its rule because their head was
|
||||
replayed too?
|
||||
3. Why did the controller open no walk for eight of its own merges within ten minutes? The controller's own
|
||||
path never waits, so its walk may have been missed rather than absent. Not diagnosed here.
|
||||
3. Why did the controller open no rollout for eight of its own merges within ten minutes? The controller's own
|
||||
path never waits, so its rollout may have been missed rather than absent. Not diagnosed here.
|
||||
|
||||
+30
-30
@@ -7,14 +7,14 @@
|
||||
|
||||
| event | kind | needs |
|
||||
|---|---|---|
|
||||
| `go` | observed | its walk started on a word not this owner's (the controller's own path, or a person's `plans go`) |
|
||||
| `release` | a person's act | why and who; it walks the delivery's own commit |
|
||||
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no walk open |
|
||||
| `go` | observed | its rollout started on a word not this owner's (the controller's own path, or a person's `plans go`) |
|
||||
| `release` | a person's act | why and who; it rolls out the delivery's own commit |
|
||||
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no rollout open |
|
||||
| `stop` | a person's act | why and who; from any state not final |
|
||||
|
||||
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the walk's
|
||||
record saying `superseded`. A held delivery whose walk never opened has no walk record to read, and one
|
||||
whose walk was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
|
||||
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the rollout's
|
||||
record saying `superseded`. A held delivery whose rollout never opened has no rollout record to read, and one
|
||||
whose rollout was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
|
||||
does not match it either.
|
||||
|
||||
**Why `retire-history` refuses them.** `isHistory` requires `FromForgesWord()`: the delivery was made from
|
||||
@@ -41,7 +41,7 @@ Neither considers a held delivery whose content a later commit delivered.
|
||||
|
||||
**Ruled out.**
|
||||
|
||||
- A missing release. Releasing any of these would walk an old commit over a newer one, or walk nothing.
|
||||
- A missing release. Releasing any of these would roll out an old commit over a newer one, or roll out nothing.
|
||||
- D14 itself. It reports the state truthfully; the state has no way to change.
|
||||
|
||||
**Located** in mesh-delivery's state table (no end for a superseded or empty held delivery, and history
|
||||
@@ -52,14 +52,14 @@ read by how a delivery was made) and in to-be 47, whose state table carries the
|
||||
**Eleven held deliveries, read again.** On 2026-10-09 eleven deliveries were `held`, each raising D14's
|
||||
`delivery.<id>.stalled` and held for 24 to 48 hours. For each, `mesh-delivery.show` gave its state, why it was
|
||||
held, its merge commit and its plan. A fresh clone of the repository then showed whether the merge commit is an
|
||||
ancestor of the commit a finished walk delivered. `mesh-controller.status` listed none of the modules involved
|
||||
ancestor of the commit a finished rollout delivered. `mesh-controller.status` listed none of the modules involved
|
||||
as behind. All eleven were covered: none held a change missing where it should run.
|
||||
|
||||
**Why the controller opened no walk (open question 3).** This part comes from reading the code, and part of it
|
||||
**Why the controller opened no rollout (open question 3).** This part comes from reading the code, and part of it
|
||||
is still a guess.
|
||||
|
||||
- *A new module.* When a merge only adds a module, the controller's `SourceMoved`
|
||||
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no walk ("built,
|
||||
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no rollout ("built,
|
||||
registered when the build lands, and sent nowhere", issue 300). mesh-delivery's `MovesNothing()`
|
||||
(`model.go`) counts `Plan.New` as something to move, though. So a delivery whose plan is "builds new: …, sent
|
||||
nowhere" is not let through to `published` → `delivered`. Instead the unwalked rule in `holder.go` holds it
|
||||
@@ -67,25 +67,25 @@ is still a guess.
|
||||
read from the code, and it matches `novox/mesh-catalog@60641176abaf`.
|
||||
- *The controller's own merges (a guess, not proved).* `SourceMoved` also returns without a plan in two cases:
|
||||
when the modules the merge touches are "already built from it" (`already > 0`), and when the merge is read as
|
||||
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no walk
|
||||
of its own when another walk, or a person's push, had already built from a later commit. Which of the two hit
|
||||
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no rollout
|
||||
of its own when another rollout, or a person's push, had already built from a later commit. Which of the two hit
|
||||
these merges is not yet known. The controller's journal for the minutes after each merge would say.
|
||||
- *The link from a walk to a delivery.* `deliveryWalking()` (`holder.go`) links a walk to the delivery whose
|
||||
`merged_as` equals the walk's commit, and to no other delivery. A later walk that carried an earlier merge's
|
||||
change therefore never reaches that earlier delivery, and nothing reads "this walk's commit contains your
|
||||
- *The link from a rollout to a delivery.* `deliveryWalking()` (`holder.go`) links a rollout to the delivery whose
|
||||
`merged_as` equals the rollout's commit, and to no other delivery. A later rollout that carried an earlier merge's
|
||||
change therefore never reaches that earlier delivery, and nothing reads "this rollout's commit contains your
|
||||
merge". That is the missing input the first open question asks about. mesh-delivery could answer it from the
|
||||
forge (is the merge commit an ancestor of the delivered walk's commit) together with the walk's moved modules,
|
||||
forge (is the merge commit an ancestor of the delivered rollout's commit) together with the rollout's moved modules,
|
||||
without asking the controller what each machine runs.
|
||||
|
||||
**Four were closed with `stop`, before this issue was found.** On 2026-10-09 the operator agreed to close the
|
||||
held deliveries. Four were stopped with why "covered: its merge … is contained in … which is delivered (walk …)":
|
||||
|
||||
| delivery | merge | contained in | walk |
|
||||
| delivery | merge | contained in | rollout |
|
||||
|---|---|---|---|
|
||||
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier walk of the controller |
|
||||
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same walk |
|
||||
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same walk |
|
||||
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue walk |
|
||||
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier rollout of the controller |
|
||||
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same rollout |
|
||||
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same rollout |
|
||||
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue rollout |
|
||||
|
||||
Their stalled conditions cleared at the next D14 observation. Open D14 conditions went from 11 to 7. As this
|
||||
issue already said, `stop` is the wrong record for them: their merged pull requests now carry `mesh/delivery`
|
||||
@@ -101,12 +101,12 @@ remaining seven were left `held` for this issue's fix.
|
||||
| `novox/hq@2221be11a6ef` | merged without a passing check | merge on hq's main; builds nothing | the same |
|
||||
| `novox/hq@897bcce50252` | merged without a passing check | merge on hq's main; builds nothing | the same |
|
||||
| `novox/hq@ef6ab814b053` | merged without a passing check | merge on hq's main; builds nothing | the same |
|
||||
| `novox/mesh-controller@b98fd0f396c8` | no walk opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
|
||||
| `novox/mesh-controller@ea92af2a7d77` | no walk opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release walk that is done; the step looks stale but is not final |
|
||||
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished walk of the same module | its group is still `delivering`, and its own walk is `superseded` |
|
||||
| `novox/mesh-controller@b98fd0f396c8` | no rollout opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
|
||||
| `novox/mesh-controller@ea92af2a7d77` | no rollout opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release rollout that is done; the step looks stale but is not final |
|
||||
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished rollout of the same module | its group is still `delivering`, and its own rollout is `superseded` |
|
||||
|
||||
The seven show the two ends this issue asks for: four hold nothing to deliver, and three are covered by a
|
||||
later walk.
|
||||
later rollout.
|
||||
|
||||
## 2026-10-09 — the fix, proposed
|
||||
|
||||
@@ -115,12 +115,12 @@ ADR 0270
|
||||
mesh-delivery:
|
||||
|
||||
- **A merge that moves nothing is delivered at once.** `MovesNothing()` no longer counts `Plan.New`: a module
|
||||
the controller builds and sends nowhere opens no walk, so there is nothing to wait for. A merge whose plan moves
|
||||
the controller builds and sends nowhere opens no rollout, so there is nothing to wait for. A merge whose plan moves
|
||||
nothing goes from `proposed`, `checked`, `rejected` or `held` to `delivered`, before the row *merged unchecked
|
||||
→ held*.
|
||||
- **`covered`, a new final state: a later merge that contains it was delivered.** For every module its plan
|
||||
moves, a delivered delivery of the same repository and trunk, merged after it, delivered on its own passing
|
||||
check by the walk of its own merge commit, whose machine steps say the module reached every machine running
|
||||
check by the rollout of its own merge commit, whose machine steps say the module reached every machine running
|
||||
it. The forge then says, through a new read-only tool `gitea_contains` (its compare of the two commits),
|
||||
whether the later merge commit contains this one; merge order is only the first filter. A forge that cannot
|
||||
answer leaves it held. `mesh/delivery` says `covered: a later merge that contains it was delivered: …` as a
|
||||
@@ -128,7 +128,7 @@ mesh-delivery:
|
||||
- **Read on every tick**, so what is held now ends without an event.
|
||||
|
||||
This answers open question 1: mesh-delivery reads it from its own records and asks the forge for ancestry; the
|
||||
controller is not asked. Open question 3, why the controller's own merges opened no walk, is not answered by this fix; it no
|
||||
controller is not asked. Open question 3, why the controller's own merges opened no rollout, is not answered by this fix; it no
|
||||
longer leaves anything held.
|
||||
|
||||
**How to verify after mesh-delivery is delivered with it.** Within a minute of its start:
|
||||
@@ -137,13 +137,13 @@ longer leaves anything held.
|
||||
merges (`builds new: …, sent nowhere`), and none of the five controller merges held for "no walk opened".
|
||||
The hq and module-adding ones are `delivered`; the controller ones are `covered`.
|
||||
2. `mesh-delivery.show` of one of them, for example `novox/mesh-controller@b98fd0f396c8`, ends with a
|
||||
transition `held → covered (covered)` whose why names the later delivery and its walk.
|
||||
transition `held → covered (covered)` whose why names the later delivery and its rollout.
|
||||
3. `mesh-delivery.checks` for its pull request shows `mesh/delivery` as `success covered: …`.
|
||||
4. The forge module is delivered in the same pull request. Until its `gitea_contains` answers, nothing is
|
||||
covered, and the held controller merges stay held.
|
||||
5. At the self-check's next reading, `mesh-controller.conditions` no longer lists their
|
||||
`delivery.<id>.stalled`.
|
||||
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered walk sent
|
||||
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered rollout sent
|
||||
their module. One that stays held is a real question: its change may not run anywhere.
|
||||
|
||||
**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed.
|
||||
|
||||
@@ -52,7 +52,7 @@ not addresses, paths or secrets — so they reached the operator untouched.
|
||||
|
||||
**Nothing said whether the operator was needed, or let them answer.** No field of a condition said whether
|
||||
the operator was needed, so every warning was shown, and an explanation could imply an instruction. The
|
||||
verbs that answer the common cases existed (release or stop a delivery, start a waiting walk, restart a
|
||||
verbs that answer the common cases existed (release or stop a delivery, start a waiting rollout, restart a
|
||||
service, silence a condition), but the desk could not take an answer: `node-notifier.send` had no actions,
|
||||
and the conversation design that adds them (to-be 46) was not built. The one way to act was a session.
|
||||
|
||||
@@ -70,7 +70,7 @@ nothing is not sent. The desk offers the actions, and the operator channel's hol
|
||||
through its seat verb. Every channel says the time in the operator's zone; a clearance is one line.
|
||||
|
||||
- novox/mesh-controller: the condition's three new fields, the plain rule, a wording for every kind the
|
||||
controller raises, the richer words at the producers that know more (a walk's modules, a unit's name, a
|
||||
controller raises, the richer words at the producers that know more (a rollout's modules, a unit's name, a
|
||||
hand-act's cause, a delivery's repository), and a lint over every condition the suite raises.
|
||||
- novox/mesh-catalog: the messenger shows the words, keeps quiet what needs nothing, offers and performs
|
||||
the answers, says since when in the operator's zone (the new setting `time-zone`), says a clearance in
|
||||
|
||||
@@ -88,5 +88,5 @@ To-be 45 §2 is amended: what `status` lists, and the new condition.
|
||||
## Resolved, 2026-10-09
|
||||
|
||||
mesh-controller#150 and mesh-lab#68 (R325) merged on 2026-10-08 at 14:58 UTC. Every later controller
|
||||
delivery carries #150; the last, of mesh-controller#170, finished its walk on 2026-10-09 at 00:11 UTC, and
|
||||
delivery carries #150; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at 00:11 UTC, and
|
||||
`mesh-controller.status` named no machine behind.
|
||||
|
||||
+3
-3
@@ -20,7 +20,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
|
||||
1.4 a second, or about 43 every 30 seconds.
|
||||
- Over the server's whole uptime (39 hours 49 minutes at 14:13 UTC) the average is 39.7 writes a second.
|
||||
|
||||
In that window one walk was in progress, so a few of those writes were real changes. The diagnosis
|
||||
In that window one rollout was in progress, so a few of those writes were real changes. The diagnosis
|
||||
shows why most were not.
|
||||
|
||||
## Why it is an issue
|
||||
@@ -40,7 +40,7 @@ watcher, and the write rate cannot be used to see that deliveries are moving.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **In mesh-delivery's tests:** a reconcile pass over walks that did not change performs no write. The
|
||||
- **In mesh-delivery's tests:** a reconcile pass over rollouts that did not change performs no write. The
|
||||
test's fake store counts writes, and the test fails on any write.
|
||||
- **Live:** over ten minutes in which no delivery changes state, the write count on
|
||||
`$KV.mesh-delivery_deliveries.>` (`nats.nats_subscriptions`) rises by no more than the number of
|
||||
@@ -56,6 +56,6 @@ build agent. So it resolves without a replay. The test above is still the proof.
|
||||
|
||||
- `keep` writes only when the delivery differs from the value it last wrote. Either compare the encoded
|
||||
value with the last one written, or set a changed flag wherever the delivery is changed.
|
||||
- `Reconcile` skips a walk whose revision the delivery already holds.
|
||||
- `Reconcile` skips a rollout whose revision the delivery already holds.
|
||||
- A retry of an owed effect that fails again changes only its try count and next time. Whether that
|
||||
needs a write at every try is a choice for the fix: the count may stay in memory between writes.
|
||||
|
||||
+8
-8
@@ -12,16 +12,16 @@ runtime. `keep` does not compare the delivery with what it wrote before. Every c
|
||||
of transitions is empty.
|
||||
- `settleAndKeep` calls `settle` and passes the result to `moved`. A delivery with nothing to move is
|
||||
still written.
|
||||
- `walkMoved` takes one walk from the controller, records it on its delivery (`linkWalk`) and calls
|
||||
`settleAndKeep`. A walk that has not changed since the last pass leaves the delivery as it was, and
|
||||
- `walkMoved` takes one rollout from the controller, records it on its delivery (`linkWalk`) and calls
|
||||
`settleAndKeep`. A rollout that has not changed since the last pass leaves the delivery as it was, and
|
||||
the delivery is written anyway.
|
||||
- `Reconcile` asks the controller's `delivery-walks` for the walks it keeps and calls `walkMoved` for each.
|
||||
The answer holds every open walk and the last 50 that ended. The ticker in main.go runs `Reconcile`
|
||||
- `Reconcile` asks the controller's `delivery-walks` for the rollouts it keeps and calls `walkMoved` for each.
|
||||
The answer holds every open rollout and the last 50 that ended. The ticker in main.go runs `Reconcile`
|
||||
every 30 seconds.
|
||||
|
||||
So each pass writes about one delivery per walk the controller returns. On 2026-10-08 that was 50 walks
|
||||
(32 done, 14 superseded, 3 failed, 1 walking), against about 43 writes per pass measured. *I think* the
|
||||
difference is walks that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
|
||||
So each pass writes about one delivery per rollout the controller returns. On 2026-10-08 that was 50 rollouts
|
||||
(32 done, 14 superseded, 3 failed, 1 `walking`), against about 43 writes per pass measured. *I think* the
|
||||
difference is rollouts that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
|
||||
|
||||
**What made it 40 a second before today.** Before mesh-catalog #121 (the fix of
|
||||
[issue 313](../313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md), merged
|
||||
@@ -39,5 +39,5 @@ doubling to an hour) and ends them after 60 tries. A retry still writes its deli
|
||||
- *The groups bucket.* `mesh-delivery_groups` has 26 keys, and its subscription counted 719 writes in the
|
||||
same 39 hours.
|
||||
|
||||
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every walk the
|
||||
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every rollout the
|
||||
controller keeps through `settleAndKeep` every 30 seconds.
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
|
||||
|
||||
The long-lived connections were young too. At 14:15 UTC the controller's two were 4 minutes old, the
|
||||
build agents' 2 to 19 minutes, the route proxies' 13 to 17 minutes and the tool runners' 10 minutes to 3
|
||||
hours. Two walks were sending new builds of the controller, the build agent and catalogue modules
|
||||
hours. Two rollouts were sending new builds of the controller, the build agent and catalogue modules
|
||||
between 14:00 and 14:20 UTC, and the build agent's container on the control node started at 14:19 UTC.
|
||||
*I think* those young connections are restarts by deliveries; they are tens, not thousands.
|
||||
|
||||
|
||||
+2
-2
@@ -17,7 +17,7 @@ controller's, so each place that needs the bus opens one, uses it and closes it:
|
||||
command.
|
||||
- `sayingOnTheBus` (delivery.go): a delivery's `check`, `go` and `stop`, which mesh-delivery asks for.
|
||||
|
||||
The serving controller dials too. Its watchdogs run every 30 seconds, and while a walk waits for a
|
||||
The serving controller dials too. Its watchdogs run every 30 seconds, and while a rollout waits for a
|
||||
build, `buildSeatPause` (plan_retry.go) opens a fresh connection to read whether the build seat's
|
||||
holders paused, and closes it. `status` does the same.
|
||||
|
||||
@@ -41,7 +41,7 @@ its long-lived connections are named `mesh-controller` too and are told apart on
|
||||
minute. The research's guess came from their age alone.
|
||||
|
||||
**Located** in the controller: one connection per command process that touches the bus, and one per
|
||||
watchdog tick while a walk waits for a build.
|
||||
watchdog tick while a rollout waits for a build.
|
||||
|
||||
## 2026-10-09, resolved
|
||||
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ the fix is the attribution tool; with it, this diagnosis continues.
|
||||
- **A holder that never said it paused.** `PausedSaid` (controller, internal/link/queue.go) asks for the
|
||||
last message on each build holder's pause subject. A holder that never said one gets a not-found
|
||||
answer, which the server counts as an API error. The serving controller's watchdog reads it every 30
|
||||
seconds while a walk waits for a build (issue 327).
|
||||
seconds while a rollout waits for a build (issue 327).
|
||||
- **Short-lived consumers.** The controller heard 495 "consumer deleted" advisories in the first 8
|
||||
minutes of its current connection, about one a second. They are short-lived consumers: the controller's
|
||||
advisory reader (internal/link/advisories.go) ignores them and says why, "every watch of a bucket and
|
||||
|
||||
+7
-7
@@ -7,12 +7,12 @@ opened: 2026-10-08
|
||||
## What was observed
|
||||
|
||||
On 2026-10-08 a person released a delivery of the catalogue that the group check (issue 316) had held. Its
|
||||
walk had one tier with one module, `dunst`, which runs on the laptop and the workstation. The walk sent the
|
||||
rollout had one tier with one module, `dunst`, which runs on the laptop and the workstation. The rollout sent the
|
||||
new build to the laptop first, and the gate passed it there:
|
||||
|
||||
> dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent
|
||||
|
||||
Twenty-eight minutes later the same walk said the opposite, about the same gate:
|
||||
Twenty-eight minutes later the same rollout said the opposite, about the same gate:
|
||||
|
||||
> dunst failed its gate on the laptop in tier 0: healthy 3 times over 2m5s; put back dunst to ea462666 on
|
||||
> the laptop, in one send; the workstation left as it was
|
||||
@@ -31,22 +31,22 @@ build that worked, recorded it as failed, and so blocked its delivery until a ne
|
||||
|
||||
## Fix
|
||||
|
||||
The controller judged the first machine's report again on every step of the walk, also after the gate had
|
||||
The controller judged the first machine's report again on every step of the rollout, also after the gate had
|
||||
passed (see `01-diagnosis.md`). The fix merged in mesh-controller pull request 165 (8170fc5), with its
|
||||
replay in mesh-lab pull request 72.
|
||||
|
||||
- Once a module's gate has passed, the first machine's later reports no longer decide anything about it:
|
||||
the walk sends the rest.
|
||||
the rollout sends the rest.
|
||||
- A module carried by another module's send takes over that pass before its step is read.
|
||||
- A guard, not reached with the two changes above: should a later path still stop a walk at a module whose
|
||||
gate passed, that module is not put back and its build is not marked failed. The walk stops with the
|
||||
- A guard, not reached with the two changes above: should a later path still stop a rollout at a module whose
|
||||
gate passed, that module is not put back and its build is not marked failed. The rollout stops with the
|
||||
reason, and the module is left sent to its first node only, which `plans retry` takes up again.
|
||||
|
||||
## How it is checked
|
||||
|
||||
`TestReplay335` in the controller replays the live case with only what the controller had before the fix:
|
||||
a passed gate on the laptop, the laptop's later report not about this build (or a failure of another
|
||||
send), and the walk 30 minutes and 9 seconds after the first send. A second test drives the walk over a
|
||||
send), and the rollout 30 minutes and 9 seconds after the first send. A second test drives the rollout over a
|
||||
tier of two modules, the second carried by the first's send, and fails if the carried module's step is read
|
||||
before it takes over the pass. On the commit before the fix it fails
|
||||
with "laptop did not report it applied within 30m0s", and the passed build reaches the put-back. It is R335
|
||||
|
||||
+8
-8
@@ -6,15 +6,15 @@
|
||||
found ADR 0236 (what a rollback is), issue 318 (a wait for a login failing a gate) and issue 279, none of
|
||||
them about a gate that passed and was then failed.
|
||||
|
||||
**The walk's own lines.** The controller's journal on the control-node said, for this walk (times UTC):
|
||||
**The rollout's own lines.** The controller's journal on the control-node said, for this rollout (times UTC):
|
||||
|
||||
| time | what the controller said |
|
||||
|---|---|
|
||||
| 16:44:45 | tier 0 built; sent dunst to the laptop first, the rest once its gate passes |
|
||||
| 16:44:55 | the laptop applied what it was sent |
|
||||
| 16:46:52 | dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent |
|
||||
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another walk sends |
|
||||
| 16:49:12 | the other walk (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
|
||||
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another rollout sends |
|
||||
| 16:49:12 | the other rollout (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
|
||||
| 17:14:54 | dunst failed its gate on the laptop: healthy 3 times over 2m5s; put back on the laptop; the workstation left as it was |
|
||||
|
||||
The laptop's node-engine logged `applied` at 16:44:55 and nothing more until 18:01:58. It did not report
|
||||
@@ -25,14 +25,14 @@ on the 16:49:12 send for over an hour. Why it did not is not yet known; the lapt
|
||||
waits 30 minutes (`planWaitBound`) for a first machine to report what it was sent. The failure came at that
|
||||
bound and not at the gate's.
|
||||
|
||||
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a walk reads
|
||||
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a rollout reads
|
||||
`nextRollout(state, …)` before anything else, for every module in the tier not yet sent to the rest. For a
|
||||
module sent first, `nextRollout` looks at the first machine's last report. If that report is not about the
|
||||
declaration the machine was last sent (`Current` false), it waits; past the bound it answers
|
||||
`failed: <machine> did not report it applied within 30m0s`. Nothing in it looked at the gate's verdict.
|
||||
|
||||
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the walk tries again on
|
||||
the next step). At 16:49:12 the other walk's send made the laptop's last report no longer current. From
|
||||
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the rollout tries again on
|
||||
the next step). At 16:49:12 the other rollout's send made the laptop's last report no longer current. From
|
||||
then on `nextRollout` waited for a report it would never get about this build. At the bound it failed, and
|
||||
`advanceOnce` called `failFirstSend`.
|
||||
|
||||
@@ -47,7 +47,7 @@ laptop. That is the line seen at 17:14:54.
|
||||
- The person-wait of issue 318 (ADR 0254), `putBackBroken` and the carried-module paths: the plan had one
|
||||
module, nothing was carried, and nothing was found broken.
|
||||
- Sleep: the laptop did not sleep until 18:01, and the failure was at 17:14:54.
|
||||
- The release itself (`mesh-delivery.release`): it let the walk go at 16:44, and the walk ran as any other.
|
||||
- The release itself (`mesh-delivery.release`): it let the rollout go at 16:44, and the rollout ran as any other.
|
||||
|
||||
**Located.** mesh-controller, `cmd/mesh-controller/release_plan.go`: `nextRollout` judged a passed gate's
|
||||
first machine again, and `failFirstSend` put back a module whose gate had passed. The path has been there
|
||||
@@ -57,4 +57,4 @@ step as the pass and another send reaches the first machine before the 30-minute
|
||||
**What it needs to deliver dunst.** The build is recorded as failed at its gate, and the controller does
|
||||
not send a build marked that way again on its own. The registered build is back at the one before. Once the
|
||||
fix is delivered, a new build of dunst (`rebuild dunst`, or a newer merge of it) is judged at the gate
|
||||
again and walked to both machines.
|
||||
again and rolled out to both machines.
|
||||
|
||||
@@ -20,10 +20,10 @@ From the merge on, every send to the control-node was held. `mesh-controller.pla
|
||||
> planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned
|
||||
> step ... — tried again for 27m45s
|
||||
|
||||
The same refusal held the controller's own walk and the release of builds kept back for a gate.
|
||||
The same refusal held the controller's own rollout and the release of builds kept back for a gate.
|
||||
|
||||
For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it
|
||||
waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and
|
||||
waited for a person. The operator was not told. The hold was found only because a rollout looked stuck, and
|
||||
the operator asked: "where would you be waiting then? Where is my approve button?"
|
||||
|
||||
The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54.
|
||||
@@ -33,7 +33,7 @@ The held sends then went through.
|
||||
|
||||
The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a
|
||||
wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each
|
||||
held walk, which is read only by somebody already asking `plans`. Rule 4 of
|
||||
held rollout, which is read only by somebody already asking `plans`. Rule 4 of
|
||||
ADR 0227
|
||||
(no failure is silent) is met in the letter and missed in effect: the mesh held every send to its
|
||||
most important machine and said nothing until asked.
|
||||
@@ -47,10 +47,10 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
|
||||
|
||||
## Impact
|
||||
|
||||
- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk
|
||||
- Every send to the control-node stops: the controller's own rollout, the node-engine's, every module's rollout
|
||||
that includes the control-node, and gated builds waiting for release. Other machines in the same tier
|
||||
wait with it.
|
||||
- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching.
|
||||
- The operator learns of it only by noticing a stuck rollout, which can take hours when nobody is watching.
|
||||
- The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no
|
||||
sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name
|
||||
neither the bus nor the verb a person runs. It never fired in this case, because the step was taken
|
||||
@@ -62,9 +62,9 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
|
||||
|
||||
| Rule | Check |
|
||||
|---|---|
|
||||
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound |
|
||||
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
|
||||
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step |
|
||||
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a rollout wants to send raises a condition on the next watchdog tick, before any tier bound |
|
||||
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held rollouts or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
|
||||
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a rollout whose only refusal is the bus's planned step |
|
||||
| it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens |
|
||||
| the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then |
|
||||
|
||||
@@ -81,8 +81,8 @@ why none is possible.
|
||||
with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the
|
||||
control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since
|
||||
when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It
|
||||
is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a
|
||||
walk held by a paused build seat is.
|
||||
is raised on the first refusal, not after a bound. A rollout held only by it is excluded from S3, as a
|
||||
rollout held by a paused build seat is.
|
||||
2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any
|
||||
channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask
|
||||
to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word
|
||||
|
||||
@@ -13,13 +13,13 @@ The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X
|
||||
(nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but
|
||||
only in the error.
|
||||
|
||||
**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the
|
||||
**Where the error goes.** In `release_plan.go`, the rollout's step loop catches the error, keeps it as the
|
||||
plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the
|
||||
controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why
|
||||
`plans` showed it and `conditions` did not.
|
||||
|
||||
**Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises
|
||||
`plan.<id>.stalled` when a walk has been in one tier past its bound. The bound is
|
||||
`plan.<id>.stalled` when a rollout has been in one tier past its bound. The bound is
|
||||
`max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`,
|
||||
`watchdogs.go` `boundsOf`). So:
|
||||
|
||||
@@ -29,14 +29,14 @@ controller's log once, and tries again on the next tick. Nothing raises a condit
|
||||
mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have
|
||||
appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is
|
||||
not the operator;
|
||||
- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its
|
||||
delivery's word has its own watchdog (S16). A walk held for the bus step has neither.
|
||||
- a rollout held by a paused build seat is excluded from S3 (`pausedWaiting`), and a rollout waiting for its
|
||||
delivery's word has its own watchdog (S16). A rollout held for the bus step has neither.
|
||||
|
||||
**What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only
|
||||
while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step.
|
||||
Before a step is started there is nothing for it to read, so a pending bus build raises nothing.
|
||||
|
||||
**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge
|
||||
**Before the merge.** `delivery.go` lists the bus among the modules whose rollout starts at the merge
|
||||
(`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither
|
||||
reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a
|
||||
merge that moves the bus's build passes green and blocks every later send to the control-node.
|
||||
@@ -64,12 +64,12 @@ person, and the person must be told.
|
||||
ADR 0271
|
||||
(proposed) takes suggested fixes 1 and 3. mesh-controller #173 builds them:
|
||||
|
||||
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a walk's send was refused
|
||||
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a rollout's send was refused
|
||||
for the bus's planned step, while the new bus build would still replace the bus on its machine. It is a
|
||||
warning, for the operator. It names the machines, each waiting walk and the modules of its tier, the bus build
|
||||
warning, for the operator. It names the machines, each waiting rollout and the modules of its tier, the bus build
|
||||
from and to, since when, and the `mesh-controller.bus` call with `upgrade`. It clears once the bus's machine has
|
||||
been sent the build the mesh holds.
|
||||
- **S3 leaves out a walk held only by it.**
|
||||
- **S3 leaves out a rollout held only by it.**
|
||||
- **The delivery plan says it before the merge**: "merging this needs a person's bus upgrade: … nothing else is
|
||||
sent to <machine> either", and its summary, which `mesh/merge-gate` carries, says the same.
|
||||
|
||||
|
||||
+3
-3
@@ -20,7 +20,7 @@ The plan rebuilt and moved them whatever the merge changed. Two examples:
|
||||
|
||||
- Mesh-controller PR #159, merged as `ca09a07f`, changed files under `cmd/mesh-controller`,
|
||||
`internal/broker`, `internal/link`, `internal/catalogue/verbs.go` and the root `module.json`. Nothing
|
||||
under `examples/route-proxy` changed. Its walk said, on the home server, "recreates 1 of route-proxy's
|
||||
under `examples/route-proxy` changed. Its rollout said, on the home server, "recreates 1 of route-proxy's
|
||||
3 container(s) (with a new image: server)" and "recreates 1 of build-agent's 1 container(s)".
|
||||
- Mesh-controller PR #163, merged as `9b028b42`, did the same. The route proxy's program at that commit
|
||||
is byte for byte the program at the commit before it (measured in the diagnosis), and it was still
|
||||
@@ -43,8 +43,8 @@ The operator stated the rule the delivery plan should follow:
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
- **Deliveries are slow.** A controller merge walked for about fifteen minutes instead of about five,
|
||||
and the walks of one repository must not overlap (issue 335), so every merge after it waits too. The
|
||||
- **Deliveries are slow.** A controller merge rolled out for about fifteen minutes instead of about five,
|
||||
and the rollouts of one repository must not overlap (issue 335), so every merge after it waits too. The
|
||||
operator considers this a major bug and a main reason deliveries are slow.
|
||||
- **A public service restarts for nothing.** The route proxy on the home server was recreated at
|
||||
merges that did not change its program.
|
||||
|
||||
+11
-11
@@ -31,7 +31,7 @@ Read in mesh-controller at `8170fc58`.
|
||||
4. `mesh-controller` itself is built from the repository's root (its manifest's bundle compiles
|
||||
`cmd/mesh-controller`; its source path is empty), so every changed file in the repository touches
|
||||
it as well.
|
||||
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the walk serial here:
|
||||
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the rollout serial here:
|
||||
- a `packages` edge, from a module whose build read a repository to the module whose source that
|
||||
repository is. So `route-proxy` and `build-agent` both depend on `mesh-controller`, because they
|
||||
share its repository, not because either is built from the controller's artifacts;
|
||||
@@ -108,9 +108,9 @@ change. The build source of the route proxy is `examples/route-proxy`, `internal
|
||||
## Measured: what the extra tiers cost
|
||||
|
||||
From `mesh-controller.plans` (read only), the first-node gate of each tier of tonight's controller
|
||||
walks:
|
||||
rollouts:
|
||||
|
||||
| walk | mesh-controller | build-agent | route-proxy | three gates |
|
||||
| rollout | mesh-controller | build-agent | route-proxy | three gates |
|
||||
|---|---|---|---|---|
|
||||
| `3f68a495` | 3m40s | 3m43s | 3m46s | 11m09s |
|
||||
| `0cf5a3a5` | 3m39s | 3m26s | 3m34s | 10m39s |
|
||||
@@ -120,12 +120,12 @@ walks:
|
||||
| `ca09a07f` | 3m43s | 3m46s | 3m40s | 11m09s |
|
||||
| `efcdd5dd` | 3m38s | 3m40s | 3m37s | 10m55s |
|
||||
|
||||
Two more walks (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still walking at
|
||||
21:50. The gates alone are 10 to 11 minutes per walk, of which 7 minutes are the two packaging tiers.
|
||||
Two more rollouts (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still rolling out at
|
||||
21:50. The gates alone are 10 to 11 minutes per rollout, of which 7 minutes are the two packaging tiers.
|
||||
Each tier also builds before it sends: the build seat's image took 53 seconds at 21:36. Estimated from
|
||||
the plan's creation time and the build records, `ca09a07f` walked from 21:07 to about 21:23 and
|
||||
the plan's creation time and the build records, `ca09a07f` rolled out from 21:07 to about 21:23 and
|
||||
`efcdd5dd` from 21:32 to about 21:47, about 15 minutes each. The controller's own tier, alone, is about
|
||||
4 to 5 minutes. Of the seven completed walks, five recreated `route-proxy` on the home server
|
||||
4 to 5 minutes. Of the seven completed rollouts, five recreated `route-proxy` on the home server
|
||||
(`3f68a495`, `e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`), and its program had changed in one of
|
||||
them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`).
|
||||
|
||||
@@ -133,7 +133,7 @@ them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07
|
||||
|
||||
The deploy plan names a build of a packaging module by the module's own source commit, which is the
|
||||
catalogue's. `moveFacts.identical` (cmd/mesh-controller/release.go) calls two builds identical when
|
||||
their commits are equal. In `efcdd5dd`'s walk both modules went `7b307a06 → 7b307a06`, nothing was
|
||||
their commits are equal. In `efcdd5dd`'s rollout both modules went `7b307a06 → 7b307a06`, nothing was
|
||||
said to be recreated, and the gate still took about 3.5 minutes for each. Yet the route proxy's
|
||||
program did change at `efcdd5dd` (table above). **I do not know whether that program reached the home
|
||||
server.** Either it was sent and the plan said nothing, or it was not sent and the route proxy runs the
|
||||
@@ -146,7 +146,7 @@ commit would also make issue 280's "repeat" rule register a different program as
|
||||
- **The changed paths not reaching the planner.** PR #159's delivery record carries its paths and
|
||||
`module_dirs_said: true`. They are read, and `touchedBy` uses them for the controller's own module.
|
||||
They are never consulted for a packaging module.
|
||||
- **The catalogue moving at the same time.** In some walks the catalogue's commit also changed
|
||||
- **The catalogue moving at the same time.** In some rollouts the catalogue's commit also changed
|
||||
(`32307bd1 → 7b307a06`), but the README-only what-if shows the controller's repository alone is enough
|
||||
to plan both modules.
|
||||
|
||||
@@ -181,7 +181,7 @@ the commit into the program.
|
||||
with the build; that is exact for Go and needs a reader per language.
|
||||
2. **Reproducible builds**: `-buildvcs=false` (or a `.dockerignore` that leaves `.git` out of the
|
||||
context) in both recipes, so an unchanged source gives an identical program. This alone does not
|
||||
stop the walk: the image digest still differs (issue 280), and the planner would still plan and gate
|
||||
stop the rollout: the image digest still differs (issue 280), and the planner would still plan and gate
|
||||
three tiers. It is cheap and worth doing beside 1, so the program's identity no longer depends on the
|
||||
commit.
|
||||
3. **Split the route proxy and the build seat into repositories of their own.** Then the existing
|
||||
@@ -191,5 +191,5 @@ the commit into the program.
|
||||
and a change to that library then moves all three again, through a real build edge.
|
||||
|
||||
Recommendation: 1 with its check, and 2 beside it. A further question for the fix: whether `built-by`
|
||||
should order a walk at all. It is a running-order edge for the build seat's worker (issue 206); the
|
||||
should order a rollout at all. It is a running-order edge for the build seat's worker (issue 206); the
|
||||
operator's rule 2 counts only modules built *from* a changed module.
|
||||
|
||||
@@ -23,7 +23,7 @@ What happened, in order (UTC):
|
||||
**under the same JSON key `declared`**. The two edits touched different lines, so git merged them
|
||||
without a conflict. The result does not compile:
|
||||
`internal/store/store.go:145:2: Declared redeclared`.
|
||||
4. **00:55.** The walk the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
|
||||
4. **00:55.** The rollout the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
|
||||
in tier 0". `mesh/delivery` on the head reads `failure`. Nothing was sent to a node, and every node kept
|
||||
`d28ca95`.
|
||||
5. **01:09.** Node-engine #67 repaired the trunk. It renamed #66's field to `DeclaredUnit`, key
|
||||
@@ -33,7 +33,7 @@ The facts behind the "different lines" claim, read from the node-engine's reposi
|
||||
contain `d28ca95`. Their common ancestor is `a2c2c35` (#63's merge). `dfb1415` holds one `Declared`
|
||||
(store.go line 107), `d28ca95` holds the other (line 139), and `690966d` holds both (lines 107 and 145).
|
||||
|
||||
No node was harmed, because a walk builds before it sends. But the merge check said "this may merge" of
|
||||
No node was harmed, because a rollout builds before it sends. But the merge check said "this may merge" of
|
||||
a merge that did not build, and both required statuses were green when it merged.
|
||||
|
||||
## The cost before this: checks that went stale
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ merging made, with two, was never built before the merge.
|
||||
or ready delivery back to proposed. Its comment says "a person asked for it again, or its group
|
||||
changed".
|
||||
- `holder.go`, `PullMerged` (line 347): a merge records the merge commit on that pull request's own
|
||||
delivery and links its walk. It does not touch the repository's other open deliveries into the same
|
||||
delivery and links its rollout. It does not touch the repository's other open deliveries into the same
|
||||
trunk.
|
||||
- The forge marks a head pending only when it announces a new head (issue 305). A merge into the trunk is
|
||||
not a new head of any other pull request.
|
||||
|
||||
@@ -60,14 +60,14 @@ note. Each gap says what is open, why it matters, and what would close it.
|
||||
|
||||
9. **On a large data pool the root search may never finish.** The node-engine's search for programs that run as
|
||||
root now has one bound of fifteen minutes (`mesh-host` #61; the inner two-minute limit is gone). On a machine with
|
||||
a very large data pool the walk can still run past it, and the verdict then stays "not judged". That is loud,
|
||||
a very large data pool the traversal can still run past it, and the verdict then stays "not judged". That is loud,
|
||||
not silent.
|
||||
*Closes with:* setuid turned off on the data datasets, so the walk skips them.
|
||||
*Closes with:* setuid turned off on the data datasets, so the traversal skips them.
|
||||
|
||||
10. **An agent can merge its own pull request.** No mesh repository's main requires an approval on the forge
|
||||
(`required_approvals: 0`), and the forge tool merges as the forge's administrator once the required checks are
|
||||
green. An agent can therefore change a module that runs on every node, open the pull request, wait for the
|
||||
checks and merge it, and the controller walks the change out. The checks judge whether every machine still
|
||||
checks and merge it, and the controller rolls the change out. The checks judge whether every machine still
|
||||
composes, not what the change is for. The operator's yes in a session is a convention, and nothing checks it.
|
||||
*Closes with:* a merge that needs the operator's own approval: a required approval on the forge from the
|
||||
operator's personal account only, or an approval through ADR 0259's asks. That holds only when the forge's
|
||||
@@ -98,7 +98,7 @@ note. Each gap says what is open, why it matters, and what would close it.
|
||||
|
||||
14. **Ways to root the node-engine's judge does not see.** ADR 0266 §3 lists them under *not judged, so not seen*:
|
||||
among them a root-run unit or script the agent account can write, a directory on root's `PATH` it can write,
|
||||
root's ssh keys readable by it, and a setuid program on a filesystem the search does not walk. None is known
|
||||
root's ssh keys readable by it, and a setuid program on a filesystem the search does not traverse. None is known
|
||||
on the control-node, but a healthy verdict says nothing about them. The record left them "for a later record"
|
||||
with nothing tracking it; added here on 2026-10-10.
|
||||
*Closes with:* each way judged by the node-engine beside the others, or a record that decides against judging
|
||||
|
||||
+6
-6
@@ -16,14 +16,14 @@ stops answering while it runs.
|
||||
| 10:57:57 | the route proxy's first "lookup … on <private address>:53: … connection refused" | the route proxy's log |
|
||||
| 10:58:36 | `machine.<home server>.network` raised: one of its two resolvers does not answer | conditions |
|
||||
| 10:58:45 | `machine.<control node>.network` raised; the home server's cleared, held under it | conditions |
|
||||
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the walks |
|
||||
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the rollouts |
|
||||
| 10:59:37 | `module.step-ca.<control node>.unhealthy` raised | conditions |
|
||||
| 10:59:42 | `machine.<control node>.network` **cleared**, its newest evidence (10:59:37) still "connection refused" | conditions |
|
||||
| 11:00:23 | `machine.<control node>.network` reopened, raised time 11:00:23 | conditions |
|
||||
| 11:06:00 | `seat.mesh-dns-resolver.<control node>.wrong` raised by D2 | conditions |
|
||||
| 11:10:29 | both builds failed their gate: "<control node> as a whole: raised since it was sent: machine.<control node>.network" — put back | conditions |
|
||||
| 11:18:12 | `dnsmasq.service` restarted by hand; every question answered again | the operator |
|
||||
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release walked the node-engine | conditions |
|
||||
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release rolled out the node-engine | conditions |
|
||||
|
||||
### What is measured, and what is guessed
|
||||
|
||||
@@ -54,12 +54,12 @@ answered none of its questions for the next run (`Confirm`, issue 277), because
|
||||
question can be lost. Its first run after 10:57:57 held it; the next, at 11:06:00, raised it. Every one
|
||||
of those questions had been refused, which no load explains.
|
||||
|
||||
**Measured: why the second walk of the node-engine waited on its own first machine.** A release keeps no
|
||||
**Measured: why the second rollout of the node-engine waited on its own first machine.** A release keeps no
|
||||
record per module — its modules are in its tier only — and D10 read only the records to know what a plan
|
||||
is rolling out. So while a release walked the node-engine at 11:21, D10 raised `core-behind` on every
|
||||
is rolling out. So while a release rolled out the node-engine at 11:21, D10 raised `core-behind` on every
|
||||
machine, "and no plan is rolling them out", and the gate on the home server, its first machine, waited
|
||||
on it (issue 281's settle window) until D10's next run five minutes later found the new engine there.
|
||||
The walk then passed. It did not fail, but it waited five minutes for a condition its own send caused.
|
||||
The rollout then passed. It did not fail, but it waited five minutes for a condition its own send caused.
|
||||
|
||||
**Not explained: why the second resolver did not save the lookups.** Every machine's resolver file lists
|
||||
both resolvers (`options timeout:1 attempts:2`), and the control node's own file lists its private address
|
||||
@@ -153,7 +153,7 @@ registry, is the operator's decision.
|
||||
- An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says
|
||||
— holds the open network conditions whose newest evidence names that part, and only those. A statement
|
||||
undecided as a whole holds every part. The next statement decides.
|
||||
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
|
||||
- D10 counts what a release's tier names as rolling out, so a rolled out node-engine is not `core-behind` on
|
||||
its own first machine.
|
||||
- D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because
|
||||
a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it
|
||||
|
||||
+3
-3
@@ -61,8 +61,8 @@ A replay in mesh-lab's register is still owed before this core issue resolves (A
|
||||
|
||||
## A sequel (2026-10-10)
|
||||
|
||||
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a walk:
|
||||
mesh-delivery matches a walk to the delivery merged as its commit, so two catalogue merges 18 seconds apart
|
||||
left the earlier one held for a person and the walk carrying it waiting for a word nothing gave, for 58
|
||||
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a rollout:
|
||||
mesh-delivery matches a rollout to the delivery merged as its commit, so two catalogue merges 18 seconds apart
|
||||
left the earlier one held for a person and the rollout carrying it waiting for a word nothing gave, for 58
|
||||
minutes, with no condition raised. See
|
||||
[issue 362](../362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md).
|
||||
|
||||
+14
-14
@@ -2,7 +2,7 @@
|
||||
opened: 2026-10-09
|
||||
---
|
||||
|
||||
# 351. A held delivery whose module a backlog walk carried to every machine is never covered
|
||||
# 351. A held delivery whose module a backlog rollout carried to every machine is never covered
|
||||
|
||||
## Symptom
|
||||
|
||||
@@ -19,12 +19,12 @@ What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `ha
|
||||
|
||||
| time | what happened |
|
||||
|---|---|
|
||||
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
|
||||
| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
|
||||
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
|
||||
| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
|
||||
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
|
||||
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
|
||||
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its rollout waited for its word |
|
||||
| 10-08 11:06 | three later catalogue merges took its rollout over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
|
||||
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the rollout ended `failed`, then superseded |
|
||||
| 10-08 14:27 | #127 merged as bf542128; its rollout built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
|
||||
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog rollout sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
|
||||
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog rollout, as it had for 24 hours |
|
||||
|
||||
No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
|
||||
move it again (ADR 0267):
|
||||
@@ -32,22 +32,22 @@ left as it is, the delivery stays held for ever.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
|
||||
ADR 0270 covers a held merge when a later delivered merge's rollout "sent the module to every machine running
|
||||
it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
|
||||
commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
|
||||
answered by the operator's own release of the build. Yet the record cannot end:
|
||||
|
||||
- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
|
||||
- **A backlog rollout's end is read by nothing.** While it judges, the controller's record of it names what
|
||||
its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
|
||||
machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
|
||||
from that: the steps stay `judging` for ever.
|
||||
- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
|
||||
it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
|
||||
- **`covered` reads only a rollout's own aggregate steps.** A rollout's own record says "every machine running
|
||||
it" or "the rest running it"; a step per machine, which is all a backlog rollout gives, never counts as the
|
||||
module sent everywhere.
|
||||
|
||||
ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
|
||||
whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
|
||||
walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
|
||||
rollout of the merge that built it. A held delivery in that case has no end but a person's wrong act
|
||||
([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
|
||||
the condition it raises every hour is one nobody can clear correctly.
|
||||
|
||||
@@ -56,7 +56,7 @@ the condition it raises every hour is one nobody can clear correctly.
|
||||
- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
|
||||
`gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
|
||||
- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
|
||||
- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
|
||||
- The rollout that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
|
||||
rightly no coverer.
|
||||
- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
|
||||
- A person's `release` of #125 (the verb): it would roll out a6385479, an older commit than the one the module
|
||||
was built from, and is what ADR 0270 ruled out.
|
||||
|
||||
+21
-21
@@ -2,56 +2,56 @@
|
||||
|
||||
**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
|
||||
at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
|
||||
`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
|
||||
`mesh-controller.delivery-walks` for #127's rollout and for the backlog rollout, `plans`, `hand-acts`, `module list`,
|
||||
the forge's `gitea_contains`, and the holder's journal on the control node.
|
||||
|
||||
**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
|
||||
**Where a backlog rollout's steps come from.** `walkMoved` (holder.go) hands a rollout whose record carries a
|
||||
`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
|
||||
the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
|
||||
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
|
||||
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog rollout
|
||||
holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
|
||||
`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
|
||||
and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
|
||||
module was not in that walk's tiers, and the backlog walk is a walk of its own.)
|
||||
and 14:46 UTC were never moved on. (The controller's own record of #127's rollout never named the module: the
|
||||
module was not in that rollout's tiers, and the backlog rollout is a rollout of its own.)
|
||||
|
||||
**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
|
||||
later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
|
||||
on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
|
||||
on a `sent` step whose machine is the rollout's own aggregate, "every machine running it" or "the rest running
|
||||
it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
|
||||
found, and the forge is never asked — which is why the journal has no line.
|
||||
|
||||
**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
|
||||
backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
|
||||
**What "done on a machine" means for a backlog rollout.** Read from the controller's records, not its code: a
|
||||
backlog rollout that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
|
||||
(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
|
||||
with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
|
||||
the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
|
||||
is wrong there.
|
||||
|
||||
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
|
||||
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed rollout, and the `release` verb —
|
||||
see the report.
|
||||
|
||||
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
|
||||
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog rollout's `done` machines for
|
||||
nothing, and `carries`, which counts no per-machine step.
|
||||
|
||||
## The fix
|
||||
|
||||
In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
|
||||
|
||||
- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
|
||||
walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
|
||||
step it failed on, and a step another walk carried, are left as they are.
|
||||
- **A finished backlog rollout passes the steps it carried.** `creditReleased`: every step credited to that
|
||||
rollout (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
|
||||
step it failed on, and a step another rollout carried, are left as they are.
|
||||
- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
|
||||
the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
|
||||
the rollout's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
|
||||
delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
|
||||
judging, one failed or one never sent leave the delivery held.
|
||||
- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
|
||||
open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
|
||||
- **A rollout that carried a step still judging is asked for by its id.** The controller lists back only its
|
||||
open rollouts and the last fifty ended at a start; the backlog rollout of this case, ended the day before, was
|
||||
no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
|
||||
such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
|
||||
such a rollout by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
|
||||
its record as said.
|
||||
|
||||
Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
|
||||
walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
|
||||
rollout of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
|
||||
The record's consequence that named the one-machine gap gains a progressive insight naming this one.
|
||||
|
||||
**How it is checked**, in `covered_test.go`:
|
||||
@@ -59,15 +59,15 @@ The record's consequence that named the one-machine gap gains a progressive insi
|
||||
- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
|
||||
still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
|
||||
- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
|
||||
and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
|
||||
and leaves the other judging; a failed step stays failed; a step another rollout carried is not touched.
|
||||
- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
|
||||
while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
|
||||
while the backlog rollout judges, covered once it is done and read by its id, naming the later merge and the
|
||||
module.
|
||||
|
||||
All three fail on the catalogue's main before the change and pass after it.
|
||||
|
||||
**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
|
||||
asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
|
||||
asked for by the backlog rollout's id and pass; #127 then carries the module for #125; the forge is asked once
|
||||
whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
|
||||
delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
|
||||
acceptance of this issue, read with `mesh-delivery.show` of #125.
|
||||
|
||||
+3
-3
@@ -21,7 +21,7 @@ On 2026-10-09, local times on the control node (its controller's and node-engine
|
||||
migrations than the store had applied, and served what it could.
|
||||
- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3
|
||||
times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent,
|
||||
which the put-back had removed from the machine. The plan walked on, believing the controller delivered.
|
||||
which the put-back had removed from the machine. The plan rolled out on, believing the controller delivered.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
@@ -35,8 +35,8 @@ Four rules did not hold.
|
||||
gate of the build it had replaced.
|
||||
- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than
|
||||
itself"). A controller older than the store's schema starts behind its own records. Nothing refused it.
|
||||
- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to
|
||||
the merge plan's send: a release keeps no module records, and the check for a walk in progress read
|
||||
- **One rollout of a module on a machine at a time** (issue 281, ADR 0236). A release's rollout was invisible to
|
||||
the merge plan's send: a release keeps no module records, and the check for a rollout in progress read
|
||||
only those.
|
||||
|
||||
The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user